It's a white car, the unaided human visual system has only so much resolution. Once you drop below that resolving ability you are finished.
I'd hate to tell you how messy the car's paint is going to look under an electron microscope. Yet we don't goto all the effort involved in making silicon wafers perfectly flat.
Since we are talking about a matter of opinion "it's no big deal", there is no way for you to declare it to be correct or incorrect. :cooleek:
I think this comes down to a difference in philosophy. My attitude is that I want the car to look as good as possible, as is consistent with it looking good for as long as possible.
Grinding away the paint (thereby making it thinner and causing it to fail sooner) to achive transient perfection is not consistent with that idea. Unless you keep the car in a garage 24/7 and never drive it, you will never be able to prevent all maring and swirling. OTH you can easily get it to the point where surface is defect free to given level of visual inspection. (e.g the 5 foot test).
White cars are nice, since they don't show paint defects as much, which means they tend to need less work to get them to the same level of visual quality. I'd spend more time doing the trim/tires because the visual contrast between the dark trim and white body will make the white paint seem that much brighter.