If you want to drill them out, you need to try to get a good drill bit (made for drilling into metal) about the same size as the rivet shank. (The compressed head (flat, not rounded) of the rivet is supposed to be 1.5 times the diameter of the shank.) I find that it goes much easier if you take a center punch and put an indention dead center of the (rounded, not flat) rivet head. Put the tip of the drill bit in the indention and drill through the head until you are very close to flush with the surface of the material being held together by the rivet. If your drill bit is the right size and your drilling angle was correct, you can wiggle the drill while the bit is still in the hole and the head will separate from the shank. At that point, you can take a punch and tap the rest of the rivet out. Another good way to do this is use a punch about the same size as the drill bit, put it in the drilled hole, and wiggle it. Once again, the head should pop off and you can tap the rivet out with punch. If the head won't pop off, don't keep trying! Aluminum will tear pretty easily and you don't want a hole to turn into a crack. You may need to drill further, or maybe even go up a drill bit size. You shouldn't have to pound hard to drive out a rivet shank either.
I hope I made some kind of sense. I can picture it clearly in my head. lol. If you have to remove several (hundred) rivets, they do make a rivet shaver. It chucks up into a drill and would cost around 75$ Take all of this with a grain of salt. This is how it works for rivets on aircraft. I am sure that much of the same still applies to boats, but we usually don't see huge rivets on aircraft. Most of ours are 1/8 or 3/16 in diameter.