I scraped as much as I could using a plastic scraper ("spudger"), then I used some silicone remover I found at my local hardware store called Silicone-Be-Gone, made by DAP.
Home Hardware - 90mL Silicone Seal Remover
It works slowly, you have to give it at least an hour to react (2 hours preferable) and it will turn the silicone into some goo that is much easier to remove, still with a plastic scraper. You may have to put another coat or two, and when the leftover is very thin you can simply wipe it off with a rag. The stuff will not affect the gel coat, and it saved me A LOT of work when I resealed my trailer windows