Not at all, that is if the surface is prepared properly. You may be getting some particles or hard dust onto your surface. That could happen if your car is driven daily and parked in a sunny place for prolonged periods. My daily driver, going against what I said earlier, used to get that way. With the daily driver, it did not get detailed frequently, so I attribute the infrequent waxings to it collecting dust and particles. Car #1, on the other hand, doesn't suffer those things happening to its paint.
My previous new car that I drove daily did not suffer from roughness in the paint, because that car was detailed regularly.
Edit: If you think its the wax's fault anyway, change your brand. IMO the better waxes "should" not act in that manner. Mothers is a good behaving, over the counter wax for reference.