Disclaimer- I have *not* used the SSR line of polishes or the orange pad in question.
But I have a fair amount of experience with Audis

so some random observations follow.
CaliPhotog- Don't beat yourself up too much about this. Doing an Audi with just a PC is tough. I've *never* been able to get an Audi 100% without using a rotary. Note how many people are talking about using a rotary (or at least a Cyclo) on Audis.
I find the Auid clear to be rather particular about what you use on it, especially when working by PC. I dunno how much longer I'd keep struggling with the SSR 2.5. Just because something works fine for somebody else that doesn't mean it'll work fine for you. The 3M PI-III stuff I use for most Audi correction is no longer available, so I'm not sure what to recommend. I'd probably consider Optimum, Menzerna, and the Hi-Temp line from TOL. On this paint, I'd rather use an aggressive product (but still something that breaks down and is suitable for PC use) than a really aggressive pad. You might want to switch to 4" pads for the worst of the correction. When the PC "bogs down" and "just jiggles" it won't do much to fix things.
On some rare occasions you simply can't fix Audis with a PC, you just have to have a rotary (a member from the old board had this happen to his black A4, its dealer-installed holograms were simply beyond the abilities of the PC). Yes, on this paint you can have damage that a PC simply will not remove, at least without spending more than five or six hours per panel and maybe not even then.
Good news- I'm absolutely certain that you haven't killed your clear unless there was something *really* drastic done to the car before you got it. You can polish an Audi with a PC/wool pad/fine cut RC for hours without hurting it (without even hurting the swirls much

). You probably just have some hazing from your attempts, mixed with the marring you wanted to get rid of in the first place.
As Bill D said, this job is gonna take a *LOT* of time, so I'd work on one area until you get it right.
And make sure you *fully* break down your products.