You can remove the polish with the terry towel but, not the oils that are possibly hiding in the swirls that you have not removed with the buffer. With a terry towel you will just glaze over the finish without getting rid of the oils that might be hiding the swirls or scratches. You either need to wash the car or use alcohol. That is why when you polish a car sometimes and then you wash it all the swirls come back, because you removed the oils that were hiding them.
:up