Yes. I have a pearl white, so scratches/swirls are not noticeable, so I can't say for sure that I don't have any due to the towels.
I wash my car with terry cloths and a few buckets in the garage when it's really cold out. Last January I washed/dryed the car with terry cloths when it was 6 degrees out.
I could only do about 1/2 of a door at a time before it would freeze. (I used water that was pretty warm to do this)