Michael Rasmussen (blog, bio, Twitter) is a former Forrester analyst now with his own boutique firm, Corporate Integrity. In his recent blog post The Forrester GRC ‘Ripple’ (OOOPS . . . I Mean, ‘Wave’) Michael calmly dissects the Wave methodology and makes several suggestions for improving it. It is well worth the read.
However, this SageCircle blog post is actually in response to a Twitter direct message Carter received about Rasmussen’s post: “while part of me admires Rasmussen for offering critique, why didn’t he do so while AT Forrester? Hints at sour grapes.”
Probable Answer: It was only after he left the firm could he see the problem in the methodology
Major firms are constantly tweaking their methodologies with input from the analysts. But that is typically done around the edges with the goal of increasing the efficiency, fixing minor problems, or silencing vendor complaints. Frankly, it is the rare analyst at a major firm who takes the time to do an in-depth analysis of her firm’s research methodology to see where it is really broken. This lack of observations occurs for a variety of reasons:
- Analysts are too busy with day-to-day activities
- They drank the kool-aid that what the firm does is perfect
- “If it ain’t broken, don’t fix it” attitude toward things that appear to be working
So Rasmussen should not be criticized for not criticizing the Wave methodology when he was an employee of Forrester. Rather than think of this post as “sour grapes,” it is much more likely that he did not have an “ah, ha!” moment until he was on the outside looking in. We all know that hindsight is 20-20.
SageCircle Technique:
- Research consumers and AR professionals should develop a deep understanding of the methodologies used by their key analysts
- Research consumers should press analysts for detail on data and methodology underlying the analysts’ recommendations
- AR should understand the weaknesses of the analysts methodologies so as to take reasonable countermeasures to limit the damage of analysts working with incomplete information or insights
Bottom Line: It is likely futile for research clients or vendor AR to try to convince analysts about the weaknesses of their methodologies. What is more important is to understand the methodologies so as to ask the right questions and provide the right ideas.
Question: Research consumers – Have you tried to ask questions of your analysts about their methodology and data only to be stonewalled?
Building a Community 
Carter, it’s worth noting that Michael left Forrester years ago, and it is the publishing of a new Wave by someone he affirms his respect for that triggered his post. As a part of management during his tenure, I can tell you that he was – and left as – a very well-regarded analyst. And he still is.
His critique is only partially about the process; in fact I think he’s more concerned about the lack of evolution in the criteria used than anything else, which really makes the conversation more about a disagreement between two analysts looking at the same problem.
All of us at Forrester, and no doubt our counterparts at Gartner and other firms, bemoaned the length of time the process took, and how new releases sometimes came in the middle of the lengthy workflow that happened after the initial data collection and review. But that’s inevitable; every similar product does and will suffer from it until these things get updated in real time, and neither the vendors nor the analysts are likely to make that happen – it’s too labor intensive for both.
I shouldn’t put words in Michael’s mouth, though. If you are going to cover this, I think your best approach would be to talk to him directly about it. Anything the rest of us say is speculative at best.
I am not sure if I would say “futile”… I think analysts are open to new ideas and new insights. Nothing changes overnight, but every now & then, when there are improvements, you can see a vendor’s input here & there. Also, analysts are usually pretty open to admitting that some of the data may not be as tight as we would like. Once you know that, and you can give the appropriate discount, then all’s fair.
Carter respectfully may I humbly suggest that in the case of criticizing an analyst firms methodology that it would be in poor taste on several levels to openly criticize your employer. The expression: “Don’t bite the hand that feeds you” comes to mind. In many instances employees of a firm must hold their tongue ,even though the have issues with a methodology. It comes down to the culture of the firm if it welcomes constructive dialogue.
Carter,
Thank you for the interest in my posting and I respect your comments.
In defense of Forrester – the Wave process is more transparent than Gartner’s Magic Quadrant. I feel that Forrester truly has the better but not perfect process.
Where can it improve? For one thing – the process takes way to long. As in my post, this Wave is out of date the moment it was published. That is unfortunate for technology vendors as well as buyers.
My other critiques were in response to the measurement and lack of evolving criteria to match the market in this particular Wave.
Per Alex’s comment – yes, as an employee it is not in his/her best interest to publicly speak out against a process of the employer. While at Forrester I was an advocate for change. In at least two Wave’s that I did during my tenure at Forrester I went to management (of the Wave process) with technology vendors to argue for more integrity and fact checking of a company’s performance. Right now there is too much reliance on what a vendor states which can be nothing more than smoke and mirrors. This is not an issue for functionality as that is demoed against the criteria. It is an issue for company performance figures. Two vendors that I worked with to change the Wave process (which never happened) asked alongside me to require vendors to submit audited financial statements for validation (under NDAs of course) of their answers. This was never given serious attention.
Having left Forrester nineteen months back this has been my first opportunity within my expertise (GRC) to comment on the process. I did not leave Forrester on a vendetta with guns firing. This GRC Wave was released and I gave my perspective. So far the only negative reaction (which has been limited) has been focused on personal attacks against myself and no one has discredited my opinions on their merit.
As I mentioned in my posting – analysts wield a lot of power and need to be kept on their toes. Iron sharpens iron – I hope the Forrester’s process (which is the best of this sort) improves. If I am wrong on any of my points – I want to improve as well.