great question, I have clayed many times with no problems, except once when I clayed early in the day on a cold day. After claying I noticed scratches that I never got from claying before, and im fairly gentle and use good lube.
Feel the bar at room temp and compare with a bar at 40F, there is quite a difference in how hard the clay is. You could keep some inside and use a little at a time bit I try not to clay when its too cold out.
Others have not reported this but I can honestly say it was an observation that I attribute to temp since everything else was the same as my previous clay experiences.