Lots of ways to try. Clay will usually work, even on rubber. Isopropol alcohol will work. Or step up to denatured alcohol if that doesn't. Finally, aerosol brake cleaner spray will work. These are all progressively stronger.
You ALWAYS need to be careful when using solvents near paint. Apply to a swab or towel first then rub trim. Wipe residue off. If it doesn't sit and soak onto paint it will not harm it. I have done this hundreds of times.
Re-apply a rubber dresing when done.