You likely won't be able to keep it swirl free. Black shows off defects quite readily. If you have buffer holograms like you posted, You'll be in for some long hours to fix it by hand. You may want to get the car "fixed" by a pro first, then work on maintaining it afterwards. A "quick" fix would be to use a glaze to hide the marring until you can get around to fixing it "properly."
As to the scratching by rubbing your finger through the dust, it doesn't surprise me that you induced marring. I wouldn't recommend anyone doing that to any car, regardless of color of paint/quality of clearcoat. You are dragging dirt across the paint. It's like a little bit of sandpaper.
When you are washing a car, you *should* have a layer of lubrication between the mitt and the paint that should help *reduce* the abrasion incurred.
When washing, try to have a plush mitt, and don't keep using it "to the last drop." After it loses its "fluffiness" you run a greater risk of marring the paint when using it, as it isn't able to "capture and hold" the dirt as well as it used to, and it may well have an accumulation of dirt that hasn't rinsed out over the many washes it has been used on. I've seen people using mitts that looked like they had them for years. Threadbare and tattered. Not good for paint.
I have a black car, and I would say that it is possible to keep a black daily driver looking very nice. Obviously not "showroom new" all the time, but better than a vast majority of cars out there. But it takes some time and patience. Washing is an area that is relatively simple to learn "correct" techniques, and once you have that mastered, the rest will be able to add to the overall appearance. However, if you don't wash your car "well" you can undo all your previous work lickety split.
I don't put any pressure on my mitt. I just glide it over the paint so as to get an even film of shampoo over everything (lots of shampoo/water). Then I'll go back to the bucket, rinse out the dirt from that panel, then glide over the same panel again. You might want to consider having a separate mitt for upper and lower areas of the car, so you don't bring the mud from the lower parts to the upper parts next time you wash.
As mentioned by many others, try reading the links offered, then use the "search" button for specific questions you may have. I'm confident there are many threads on DC that will help educate you far better than a couple quick reply posts will.
Good luck!