Toluene works great at removing glue residue.
I have used it lots for cleaning off of plastic and metal. When gluing accessories onto an inflatable boat it is used to clean the surface well. I would do a small test spot on
paint though, as toluene is often used as a solvent in
paint and may take it off too. I have never used it on paint myself. Maybe I will run out to the shop and give it a test.
Also, provide good ventilation and use with rubber gloves.