Occasionally we get asked – almost always from someone in the vendor community – why nobody audits the analyst research. For instance, here is an email from when I was the Director of AR for a major vendor: “Does anyone in the AR community ever take analysts to task for the accuracy of their predictions? There would seem to be a huge ‘debunking’ potential here – auditing the accuracy or otherwise of their claims.”
When the response by one of my AR colleagues was “Answer to your question is Naaah, we don’t take ’em to task, though it would be enjoyable,” the original correspondent asked:
“I’m surprised that no-one in the industry takes them to task. Given the large fees charged for reports and prognostications it would be a real service for an independent agency to review the material 12, 24 and 36 months down the road and measure accuracy. This seems like serious low-hanging watermelon for someone with knowledge of the AR field to do as an independent and charge $$$ for their reports (hint: could even be done evenings and weekends and bring in some nice fees in addition to the day job :-)”
Because I just had a similar conversation with someone from Europe a couple of weeks ago, I thought I would go ahead and put my email answer to my then colleague into this post.
“If it was that easy and somebody (e.g., vendors or end users) would be willing to pay good dollars, don’t you think that there would be many firms offering such a service?
The task is actually incredibly difficult, expensive to do, fraught with copyright issues, and nobody is willing to pay for such a service.
Sure the vendors would all love to have someone ‘expose’ the analysts, but they don’t Continue reading