Thursday, February 7, 2008

Aberdeen Group Under Fire from WSJ Reporter

As a former Aberdeen Group employee and an industry analyst, I feel compelled to comment on the recent WSJ article by Lee Gomes criticizing Aberdeen’s integrity. The reporter challenged Aberdeen’s purported reformation from “pay-for-praise” to “sponsored research”, as relatively unchanged. While I don’t entirely disagree with Gomes’ claims, I take issue with the fact that his shortsighted article has the blogosphere questioning the merits of the entire technology research industry.

Aberdeen’s “fact-based” research is a thinly veiled ruse to provide vendors with third-party marketing collateral that supports their technology road maps and hidden sales agendas. This is no big secret, it's their business model! No vendor sponsors have direct input into the final documents and their names are not mentioned in the benchmark reports. Vendors do have an opportunity to weigh in on survey creation to potentially shape the direction of the findings, albeit subtle. Vendors are also encouraged to distribute the surveys to their customers to glean insight from their use of vendor technologies, which in turn serves to validate use of that technology. Yet, Aberdeen makes it abundantly clear that their research is distributed free of charge to all interested parties. The nothing-comes-for-free catch is fulfilled in the form of captured email address and a “lead” passed onto each vendor sponsor. While Aberdeen typically delivers nearly 1000 leads to each sponsor, the quality of these lists is dubious. Students, researchers, analysts and reporters consume research all day and would never be considered qualified prospects for any vendor sponsor. Numerous other consumers of this research may find value in the educational aspects, yet have no interest in a technology purchase. Aberdeen subjects recipients of its free research to email messages with near spam-like frequency. This is a fair-value trade for some, and minor annoyance for others. However, the lead generation aspects of the Aberdeen sponsorship programs are clearly and emphatically articulated to all prospects. This distinction separates Aberdeen’s model from all other subscription-based research firms.

So why did I leave the Aberdeen Group? I was conflicted by the sales emphasis placed on the research, which the company did not attempt to hide. An inter-office mantra was “What are you selling today?”, which applied equally to sales associates and analysts. This is one area where I took offense and felt compromised as an industry analyst. So much so, that I expressed my displeasure in the form of a resignation. As any analyst worth their salt will tell you, there is generally a separation between sales and research that is akin to church and state. This distinction did not exist at the Aberdeen Group during my tenure. In fact, analysts were encouraged to “sell” sponsorships and were accountable for sales numbers. Sales pressure was spread throughout the office on a constant basis extending beyond quarterly milestones, to monthly publication deadlines placed on the benchmark reports. If sponsors were not secured for reports they did not publish and quotas were missed, which held financial repercussions for sales and analysts alike. The sales tactics at Aberdeen were strong to say the least. As an analyst I was often embarrassed by the high pressure sales techniques. While not all sales executives practiced questionable tactics, my experience soured me.

To Aberdeen’s credit, they do have a robust methodology founded on strict Best-in-Class qualification criteria that was grilled into analysts by Aberdeen’s former CEO, Jamie Bedard. Unfortunately for them, 2006 and 2007 produced a revolving door of analyst departures due to sales pressures and strong-arm tactics. Since his appointment to President of the Aberdeen Group in June 2007, Steve Gold has made great strides to change the internal structure within Aberdeen. His retort to Lee Gomes was poignant and accurate. While I maintain a great deal of respect for the leaders and analysts at the Aberdeen Group, they do walk the fine line between white paper for hire company and industry analyst firm. Lee Gomes, however, appears to have discrediting Aberdeen on his agenda. While his own reputation as an instigator was established by criticizing the Long Tail theory, I won't hold my breath for an Aberdeen retraction.

This brings me to the degradation of industry analysts as a whole. While I do not purport to know more than many experts in their respective technology fields, I am diligent about the research I conduct and publish. This was true during my time at the Aberdeen Group and even more so now at JupiterResearch. From my vantage point as an industry analyst I have the good fortune to preview and evaluate many vendor technologies that enable me to identify industry trends. I take this responsibility very seriously. I also conduct formal consumer and executive surveys and talk to a wide variety of technology experts, users and developers. The research, both qualitative and quantitative, shapes my opinions and informs my writing. My internal publishing procedures are subject to critical data review, editorial fact-checking and rigorous peer review. Some may perceive industry analyst research as subjective. While I do not claim to produce empirical evidence of technology merits, I do help companies make informed decisions based on my deep understanding of technology producers and consumers. Research is created to inform and educate and these goals align with my own. So before damning the value of technology research consider the exertion required to collect, synthesize and articulately enlighten you with facts about technology that you seek to understand.


Anonymous said...

Hi John- Clearly you were at Aberdeen in the Jamie Bedard years and I bet you're pulling your punches when it comes to his "strong-arm" tactics.

I was also an Aberdeen employee during those years, was there when he arrived and cleaned house, firing the majority of the long-time analysts and support staff, then turning around and hiring a team of cronies and yes-men. Somehow I weathered the storm and was able to make my own decision to leave.

My time at Aberdeen was before the Harte Hanks acquisition, so things may have changed with the influx of HH's money and lead lists. But while I was there, one of the many outright deceitful acts I saw at Aberdeen was the manner in which they generated leads for each report.

Frequently, when downloads were not up to snuff on a report, names were pilfered from downloads of other reports and added to the lists as if these individuals had downloaded the content of their own volition, effectively providing hundreds of irrelevant or uninterested names to companies that had paid for qualified leads.

This is unfair to both the companies who are footing the bill and those individuals who downloaded reports who were contacted by sales reps trying to sell them technologies in which they had no interest.

You have to also question the quality of Aberdeen's surveys themselves. While I was there, we were receiving mayby 70-100 responses to surveys that were sent out via e-mail to anybody who had shown interest in the topic, report downloaders, low-level titles, etc. Heck, I still receive Aberdeen surveys to a test account I used while there...these surveys are for all sorts of technologies for which I have no responsibility. What's top stop me from filling out these surveys and skewing Aberdeen's research?

John Lovett said...

Thanks for sharing your perspective. While the leads may have been dubious during your time, the lead padding largely stopped as Aberdeen reached critical mass of subscribers after the HH acquisition.

But you are correct in that there is no qualification process (at least there wasn't when I was there) for who could populate surveys.