It partly depends on the type of clay being used because some clays are a lot more agressive than others, but most consumer grade clays are pretty safe if 'properly' used. Clay can be made more or less aggressive by increasing or decreasing the type of abrasives, the physical size of the abrasives, the concentration of abrasives and the firmness of the resin being used. There's a very nice article about clay on the home page of DavidB's Guide to Detailing which can be quickly accessed by clicking on the "Learn" button at the top of this page.
A comment was made about wet sanding, but using clay is in fact wet sanding in my opinion except that its happening on a much more microscopic level.
Clay uses microscopic abrasives to remove above surface contaminants, however it shouldn't be damaging the paint if it's hydroplaning on a film of fluid! Once the clay hits something above the surface, then it abrades it away or shears it off and the contaminants are embedded into the clay instead of being pushed into the paint. Without proper lubrication, the clay won't hydroplane and you greatly increase the risk of working those contaminants into and or across the paint and that will cause scratching, marring, etc.
Not all clay products are created equal and if you're going to clay often, then choose the appropraite clay for the job.