It looks like the top layer is
paint, not gelcoat. Looks like possibly an old repair that has disintegrated. The original fabrication would have been chopped
fiberglass embedded in polyester resin, with a top layer of gelcoat (pigmented polyester resin).
Again it looks like an old repair that's coming apart, with an outer
paint layer. Also, and this is important, the repair might have been epoxy, not polyester. Because polyester resin does not adhere well to epoxy resin, its best to do your repair with epoxy resin and hardener. Just grind what you want removed down to
fiberglass OR solid epoxy base, sand, clean with acetone or denatured alcohol, then build it back up to establish the original shape.
You'll need to use thickened epoxy. A small quantity of thickened epoxy such as
sold by Marine Tex, PC-7, or JB Weld will work.
EDIT: ... then apply matching
paint.