I'm getting really tired of having to continually explain the report to folks, especially when they may or may not have even read it...but, here it goes:
1) "The product rankings" - I HATE that particular chart!!!! I begged the publisher to not post such a thing, because I knew it would be misinterpretted, and boy, was I right.
Example: I got a genuinely ANGRY phone call from someone who said that "you are a bunch of idiots and I want my money back! How can you rate Mothers Cleaner Wax higher than their California Gold product?!"
I calmly explained to the gentleman that there was less than 0.01 points difference between the two products, and that a simple shift from a + to a neutral on a single grade could have swayed the results in either direction.
He said, "Oh, I didn't look at the actual numbers! How about that? Well, I guess I'll keep using my Calfiornia Gold, then. Thanks!"
THE CHART IS NOTHING MORE THAN A LIST OF ALL OF THE PRODUCTS TESTED AND THEIR "FINAL" GRADES. THERE'S
MUCH MORE INFO IN THE DETAILS OF EACH PRODUCT - DO NOT RELY ON THE CHART FOR YOUR ONE-STOP INFORMATION!!!!!!!!!!!!!
Again, I begged the publisher to remove the chart, but since he footed the $150k for printing/production, he has final say in what happens.
2) Grades were weighted as follows:
Overall Performance: 40% (averaged from 80 individual grades over 8 weeks)
Durability: 30%
Ease of Use: 20% (10% for app, 10% for removal)
Availability: 10%
When you see each product's grades, you'll notice there are 5 total grades - let's look at one, for example...
Armor All Protective Barrier Car Wax (paste): Final Grade: B- 2.73
Durability = C+ (30% of the grade)
Performance = C+ (40% of the grade)
Ease of Use = A- (20% of the grade)
Availability = A (10%) of the grade)
Summary: 70% of the grade = C+, 30% of the grade = a strong A-. Add all of them, average them any you wind up with 2.73 and a B-
This does not mean that it's a "better" product than a product like Souveran. Souveran kicked AA's *** in performance.
Again, the "chart" burns me, and I have to explain this constantly, b/c people don't want to read the "how's" the "why's" and the "what's" of the report. They want to be like Bret and look at the pictures, then write or call me to tell me what idiot's me and my staff are.
3) DO NOT LET THE REPORT DETERMINE WHAT PRODUCTS YOU USE!!!
I've said this a thousand times - literally! The report is a subjective/objective guide to product performance in nearly identical situations. It should serve as a relative comparison, with the caveat that if you're happy with your current product(s), keep using them.
In every test, one product comes in first place, and another comes in last place. That's the nature of a test! Think about Car&Driver, and when they test a Ferrari, a Porsche, a Viper and an AMG Mercedes. All of them are fantastic cars in their own right, but one hast to come in 4th, one in 3rd, one in 2nd and one in 1st.
Unlike Car&Driver, we didn't have millions of dollars in ad revenue "helping" our decisions - we used the best testing methods available to us, and left it at that.
4) The Products and the Audience were not designed with Autopian's in mind!
We had to leave out several products due to space constraints, and we had to take a serious look at who would actually be interested in the report.
In reality, Autopia is a very small website - when we published the report, Autopia had about 3,000 members, of which about 5 actually bought reports from us. Had we written/designed the test with Autopia in mind, we'd have bored and/or scared away the rest of the world!
We're all considered "nuts" here by outsiders - think about all of the comments that you get from your neighbors, etc. about how "obsessed" we are with car care. How many people of the general public would want to read a 54 page report that was Autopian in nature?
I love getting calls from people who say something like, "I really appreciate the report and the info in it - some of it was above my head, but I understood most of it, and I learned about microfiber towels, and tricks for removing defects! Thanks!"
Realize, that of the 8,500 - 9,000 reports we've sold, less than 100 of them have been sold to "pro" detailers and/or Autopian's. 99.9% of the reports are sold to average Joe's and people who are enthusiast detailers. That was our audience.
Finally, as I mentioned earlier, we had space constraints to deal with, so a few products didn't make the cut. Products like S100, Platinum, etc. missed the boat because they were similar (if not identical) to other products in performance.
I hate to brag, but I have nearly 1,000 e-mails from people regarding P21S wax and how they had never heard of it before our report came out...I wonder, how many of you heard about it through our report?
Ok, enough ranting....sorry for the sermon.
BTW, Lemon Pledge finished dead last.
) I never heard it said better.