I'm having a hard time understanding your question exactly.
Answer me this:
When you got the car, was the area that is now exposed gray? In that it didn't look like it currently does?
When you were rubbing, did it all of the sudden remove the coloring/coating in that area?
Or are you asking was this likely like this when you started, or not? (That is impossible if you are asking hehe).
Basically, regardless of if the spot was just sprayed with a light dye/paint and the leather was gone -- Or not.. The owner is going to see this, and not be happy. It is ultimately your responsbility.
On the other hand, it is quite obvious by the products you used that you did not cause the damage, it is a VERY worn part of the leather , you either gave it its final stroke before hitting the under -- Or you wiped off the 'cover up dye'.
If you did do it, and aren't wondering if it was like this when you got it -- Then I would just point it out to the owner. Don't act defensive, just be honest with him. Tell him what you did and what occurred, and that the product is used every day on all cars, etc -- Doesn't damage leather, blah blah. (No need to go into detail).
The guy will either understand that its beat up leather, and the rest of the car looks great and get on with his life (getting it fixed or replaced) -- Or get angry and demand you replace his seat.
I'm getting ahead of things here - But,.. if it comes down to him demanding a new seat or some such outrageous request. -- The most *I* would do in this case is offer to either pay for a leather specialist to fix the specific spot (they can fill and dye it to match) -- Or, offer the job to the customer for free.
I just had a slit in my leather and several spots sprayed by a specialist, and it was $85 for the repair in the seat, and 25$ extra for the dye spray on discoloration spots. He was good, and the prices seemed reasonable.
If they hold true/similar in your area, you very well could end up making money off the guy if he demands you pay for the fix and not take a free detail job (Or discounted job).
If you go this route, tell him to bring you a quote of the repair. For him not to have anything done without you first seeing the quote, and so forth.
But enough of what to do with the customer

-- (I'm just going into this a little because I was the customer on TWO things just like this) -- One was a rip in my leather by an audio shop, a few weeks later.. a Body shop buffed off some paint) -- The first resulted in the aforementioned repair and the bodyshop and I agreed to split the respray of the panel needing repainted 50/50. -- As I am reasonable and we agreed it wasn't exactly in pristine condition to start with. Which is exactly what you are working with here.
Option C) -- You can always go grab some dye of similar color and fill it in. --A bit shady and dishonest, but it might look fine -- If that was basically what was on it to begin with. --I'm not sure if you removed leather or just dye/paint here.
Again, if you are curious whether it came into your posession looking like this -- only the owner would know.
Good luck with the situation, and I'm am positive the professionals around here have dealt with things like this a time or two and can give you much more superb advice.
Take care,