It looks like the top layer is paint
, not gelcoat. Looks like possibly an old repair that has disintegrated. The original fabrication would have been chopped fiberglass embedded in polyester resin, with a top layer of gelcoat (pigmented polyester resin).
Again it looks like an old repair that's coming apart, with an outer paint
layer. Also, and this is important, the repair might have been epoxy, not polyester. Because polyester resin does not adhere well to epoxy resin, its best to do your repair with epoxy resin and hardener. Just grind what you want removed down to fiberglass OR solid epoxy base, sand, clean with acetone or denatured alcohol, then build it back up to establish the original shape.
You'll need to use thickened epoxy. A small quantity of thickened epoxy such as sold
by Marine Tex, PC-7, or JB Weld will work.
EDIT: ... then apply matching paint