Gartner’s Analyst Relations team holds a quarterly conference call for the analyst relations (AR) community. SageCircle occasionally will post about the call, but for this particular call there was so much information that we have a seven-part series to highlight details and provide commentary. See below for links to all seven posts.
One of the announcements at the Gartner Q3 AR Calls was the rollout of the “Evidence Side-bar” for written research. During the presentation the Evidence Side-bar was described as “a description of the evidence behind the written research” and it will be “positioned on the front page of each document.” This is a welcomed development as increased transparency can only enhance the credibility and usefulness of Gartner’s research. Additional detail about the Evidence Side-bar taken from the AR Call presentation includes:
- A high level view of the methodology …or…
- A link to the Methodology Document …or…
- A pointer to the Methodology Statement
- Primary research, e.g., “Gartner Survey”
- Secondary research
- Reference to another Gartner note, etc – with appropriate details
- Additional information or commentary
- A description of models used
- Criteria or inclusion of technologies or technology
- Include forecast assumptions.
This is all well and good. However, there was a glaring omission in the discussion of sources so SageCircle submitted the following question:
Frankly, we did not think that the Gartnerians would respond to the question. Much to our surprise, they did. Here is VP Mike Anderson’s reply:
“Inquiry is a great source of the evidence that a lot of analysts use for those results. I do not believe that we have standardized on the content of how inquiry be presented or how inquiry was used.
Instances where there were substantial numbers and quantities or where particular demographics have become important in the analysis that is being presented, those will be the things that analysts will be putting into the Side-bar. We’ll see feedback to them to ensure the inclusion of that information.
I think it will be dependent on the kind of analysis being published and what kind of source information is available from those inquiries. I would envision that Side-bars, where it’s relevant to analysis and conclusions, that level of detail will be provided.”
Kudos to Gartner for indicating that end-user inquiry will be discussed in the new Evidence Side-bar. However, we think that including end-user inquiry statistics and insights into Evidence Side-bars should be the #1 priority of management and this includes developing formal standards. Vendors engaged in vigorous debate on a disputed analysis with Gartner analysts are often told that the analysis is right because of what the analysts are hearing from end users. End of story, nothing the vendor can say after a statement like that matters because they can’t examine the “evidence” and counter it. A transparent disclosure of evidence and standards should help both parties resolve these conflicts.
- Research clients and AR should be aggressive in challenging analyst positions and asking for background information on how they developed their recommendations
- AR especially should require details whenever an analyst plays the “that is what I’m hearing from end-user clients” card, e.g., number of inquiries relevant to the current discussion, demographics of the end users, etc.
Bottom Line: Gartner deserves kudos for proposing the new Evidence Side-bar concept. However, these kudos should be tempered until research consumers see how they are implemented. Until there are standards and policies developed concerning the disclosure of the role of end-user inquiry in Evidence Side-bars then research consumers and AR will need to demand background information on any data points put forward by an analyst.
Question: AR – Have you ever been told by analyst that the data for their research position came from client inquiry? Were you given details about the source of that data?
- Prepping for Gartner Symposium
- Don’t bring your CEO to Symposium and expect to brief the analysts
- Cost optimization at Symposium will be a critical thread to follow for vendors
- Make sure to attend relevant Magic Quadrant presentations at Symposium ITxpo Marketplace Theater
- The CIO Panel is reason enough to attend the AR Forum at Symposium
- Evidence Sidebar – Gartner needs to discuss end user inquiry data points
- Gartner’s updated Vendor Research Escalation Process