I have a car duster in every car and use them regularly. Have for over a decade. Now, I use it as the name implies. I don't try to remove 'dirt', water spots, etc with it and I apply zero pressure... just the weight of the duster.
I had one car that while it was mostly a garage queen, I did put about five thousand a year on that car. Since it was generally only driven in nice weather it was not unusual for it to only get washed once or twice a year. The rest of the year was duster followed by a QD. So, it only cleaning, which was regular, was with a duster followed by QD. Once a year I would strip all finish product from it for a really good cleaning. It never needed more then a last step type of polish and generally didn't need that... more quality garage time for me and the car. So, I think dusters can be used safely. Just remember, it is a duster and nothing more.
Some where around here in another thread I posted a pic or two of it up on jack stands where you can see close up the paint. That is a 26 year old single stage lacquer paint job that saw a duster every week.