Rubbing your car with any towel that even has a few specks of dirt can cause this.
Using a applicator pad with not enough product on this can cause it.
Just running your hand ovr a dirty car can cause those swirl marks.
Anything that comes in contact with your paint and grinds in dirt can easily damage your finish. You need to make sure your washing with clean water, clean towels/mitts, if you can use lots of suds in the soap, rinse your washing towel/mitt often, don't over rub your surface when drying ... see where I'm going with this. Just try to keep everything away from touching your paint, if it must touch your paint try to make sure the paint and the item are clean and contact is kept to a minimum.