Why nobody tries to audit the accuracy of analyst predictions and recommendations

question-mark-graphic.jpgOccasionally we get asked – almost always from someone in the vendor community – why nobody audits the analyst research. For instance, here is an email from when I was the Director of AR for a major vendor: “Does anyone in the AR community ever take analysts to task for the accuracy of their predictions? There would seem to be a huge ‘debunking’ potential here – auditing the accuracy or otherwise of their claims.” 

When the response by one of my AR colleagues was “Answer to your question is Naaah, we don’t take ‘em to task, though it would be enjoyable,” the original correspondent asked:

“I’m surprised that no-one in the industry takes them to task. Given the large fees charged for reports and prognostications it would be a real service for an independent agency to review the material 12, 24 and 36 months down the road and measure accuracy. This seems like serious low-hanging watermelon for someone with knowledge of the AR field to do as an independent and charge $$$ for their reports (hint: could even be done evenings and weekends and bring in some nice fees in addition to the day job :-)”

Because I just had a similar conversation with someone from Europe a couple of weeks ago, I thought I would go ahead and put my email answer to my then colleague into this post.

“If it was that easy and somebody (e.g., vendors or end users) would be willing to pay good dollars, don’t you think that there would be many firms offering such a service?

The task is actually incredibly difficult, expensive to do, fraught with copyright issues, and nobody is willing to pay for such a service.

Sure the vendors would all love to have someone ‘expose’ the analysts, but they don’t Continue reading

Bursts of analyst departures in a hot research area is not unusual

The clump of departures of social media analysts – Brian Haven, Peter Kim and Charlene Li (from Forrester), and Rachel Happe (from IDC) – is not at all unusual and follows typical patterns.

There are several reasons why analysts leave a firm: just want a change or new professional challenge, recruited by another company, desire to start own firm, the current employer has grown too large and its culture has changed and a few others. In this current sitaution, there are two primary reasons why the analysts are leaving: lured by startups and hanging out their own shingle.

From late 1997 to early 2000 a number of analysts covering ecommerce/ebusiness got lured away from the firms by Dot Com startups. For example, in one week Gartner lost four of five analysts covering ecommerce. Yes, they were lured away by various startups dangling stock options, but these analysts were also annoyed at the money Gartner was investing in Jupiter Communications (ancestor of JupiterResearch) rather than beefing up Gartner’s own ecommerce/ebusiness research team.

Another common reason for analysts in a hot research area to leave a firm is to Continue reading

Forrester analyst comments on Gartner research

icon-social-media-blue.jpgWhile single practitioner analysts or analysts at boutiques will occasionally comment on the research done at the major firms (often in a snide tone), it is rare for an analyst at a major firm to acknowledge something from a major competitor. Forrester’s Jeremiah Owyang (Twitter handle, poster child for tech analysts using social media) does just that in Understanding Gartner’s “Generation Virtual”. Jeremiah – fairly, in a professional tone – highlights the research and points out where he agrees or disagrees with Gartner’s conclusions. My only quibble is that rather than focusing on just the Gartner research, Jeremiah might have also contrasted the Gartnerian model with Forrester’s Social Technographics Ladder.

Bottom Line: One way for major analyst firms to cut through the clutter of the competing voices on the Web could be to Continue reading

Forrester acquires JupiterResearch – will the analysts stay or walk?

logo-forrester.gifForrester Research acquired JupiterResearch for $23 million in cash plus assumed liabilities. JupiterResearch joins Forrester’s Marketing & Strategy Client Group. Click here to read the press release and click here to read a blog post by analyst Josh Bernoff.

The key question for any analyst firm merger & acquisition (M&A) activity is whether the acquired analysts – the core intellectual property value – stay with their new employer or leave. For example, in the case of Gartner’s acquisition of META more than 50% of the analysts left voluntarily or through buyouts within a few months.

Our initial impression is that the JupiterResearch acquisition is more of an expansion of Forrester’s services than a consolidation move to eliminate a competitor. This is similar to Forrester’s Giga acquisition, but different from Gartner’s grab of META which was clearly a strategic move to Continue reading

Analyst firms should notify vendors about staff changes

Especially vendors with scheduled briefings, consulting days, or key projects by analysts  who submit their resignations

Analyst relations (AR) professionals are sometimes blindsided in the final preparations for a long scheduled briefing, analyst summit, or analyst consulting day (aka SAS) to discover that the analyst had submitted his or her resignation several weeks before. Worse yet are situations where the vendor has just conducted a briefing only to learn days later that the analyst has just left the firm. Either way it is bad for AR who now has to scramble to change plans and could experience the wrath of executives who perceive that AR just wasted their time by being uninformed.

For a variety of reasons, analyst firms are reluctant to admit that an analyst is leaving the firm. However, these reasons are insufficient for withholding critical information from AR teams who work hard to facilitate the flow of information from the vendor to the analyst firms. It is not appropriate for the firm to arrange a last minute substitution without Continue reading

Sorry ‘Net searchers, you will not find a Forrester Magic Quadrant

We track the search terms people use that leads them to the SageCircle website and blog for SEO purposes. It is interesting the number of times that people search for some variation of Forrester Magic Quadrant. This reinforces our point that “Magic Quadrant” is a brand name that is become a generic description. However, this is very dangerous for vendors as we pointed out in Kleenex, Frisbee, and Magic Quadrant – what do they have in common? AR teams should always be on the lookout for colleagues that are using Magic Quadrant inappropriately and eliminate that usage before it causes you embarrasment in front of an analyst.

Whoa, I think people are going a little overboard on the IIAR “Analyst of the Year” survey

Don’t get me wrong, I think the IIAR “Analyst of the Year” survey was quite fun. We promoted it right here on the blog. But, good grief, folks are going a little overboard in reading into the survey that it signals some major shifts in the analyst industry.

The latest item about the IIAR survey that caused me to chuckle was the press release by Enterprise Strategy Group (ESG) touting “ESG Named One of World’s Top Ten Global Analyst Firms,” because it was ninth in the firm standing. Hurrah, we’re #9, we’re #9! Remember, only 116 AR professionals participated in the survey. So how many votes did ESG get to make it to ninth? Six or seven? If I saw this in a vendor analyst briefing presentation I would tell them to delete it, because it is the type of silly hype that would be red meat to analysts, like at ESG.

Follow

Get every new post delivered to your Inbox.