Death of IT research firms greatly exaggerated

Last month I wrote about the threat that IT research firms face from what I called "open source research." But Mark McManus, VP at Computer Economics, an IT research firm, wrote to me with a different view.

Frank, I have to comment on your post regarding paid research versus "free" Internet research.

I agree with some of your points, but there is a whole research world regarding IT that cannot be easily found on the Internet. For instance, take our annual Information System Spending study. This is a four-month project and involves interviewing hundreds of senior IT and business executives. It is very expensive to produce and involves a significant amount of analysis. How could that be duplicated for free? I've seen statistics that claim to represent this data, but much of the time this is a self-serving process by the individual or organization that has produced the data. The buyer should beware.

While there are many good sources of free research data (such as The Enterprise System Spectator), I do not expect that senior IT and business executives will ever consider information retrieved via an Internet search as an alternative to paid, targeted research performed by a reputable (and independent) IT research firm-- especially when major competitive and financial decisions are on the line.

Another issue your commentary does not clearly address is the analysis capability of major research organizations. More often than not the main reason that end user organizations will go to Gartner or Forrester is not to get statistics, but rather to get their analysis and recommendations. IT managers performing their own research will find many sources to choose from--and many varying points of view. Do you think these extremely busy individuals will really have the time to wade through the page after page of information and do the required analysis they need to make hard technical and business decisions? Maybe in some cases, but I think that is unlikely in most.

Additionally, our own experience is that companies are using our research capabilities more in the past several months than in the past two to three years. So, even though the Globe article may be correct that "major" research firms are having a difficult time, I think that there other business issues that may be creating their hardship, such as over-extending their research capabilities and areas of expertise. In fact, our research on spending trends indicates that in 2004 more IT organizations increased their research budgets than in 2003 or 2002. The "Googlization of America" may be a minor contributing factor to IT research firm woes, but it is probably not the leading factor.

The value of aggregating and filtering information
I've known Mark for several years and appreciate his feedback. In follow up correspondence, I pointed out that in my original post I indicated that in-depth research, such as Computer Economics IT spending studies, or IDC's market share studies, that require significant resources and expense, are impossible to reproduce by individual experts contributing to message boards or writing technology blogs. I also recognize the value that paid research firms provide in filtering and aggregating information so that business and IT executives don't have to wade through it themselves. I pointed out, however, that technology blogs and group blogs (blogs authored by multiple experts) might be able to provide this aggregation/filtering service in a richer way, much like blogs such as Instapundit or Command Post do for news and current events.

The question of bias
In follow up correspondence, Mark replied,

I agree that tech and group blogs can do some filtering, but there is still an issue with bias. How can you (as the recipient) trust the underlying motives of someone who is posting free research and advice? There are certainly unbiased blogs out there, but many are built as a sounding board to support a particular position. Not that research firms haven't shown bias, but generally speaking you can usually get recommendations from an independent research firm that are not heavily tilted to a particular bias.

Again, Mark brings up a good point, which is the issue of bias. Individual technology experts do have their biases, based on what they've experienced or what technologies and vendors they are familiar with. But I would point out that the paid research firms have their own problems with bias as well. An obvious example is research firms that accept money from technology vendors to produce specific research. I wrote about one such example two years ago: IDC's report on Windows 2000, which was sponsored by Microsoft, which concluded that Windows 2000 had a lower cost of ownership than Linux. Although the report was not totally one-sided, the conclusions of the report, and Microsoft's press release on it, were clearly slanted in Microsoft's favor.

But a more subtle form of bias is seen in the fact that many of the research firms have technology vendors as major subscribers. This limits how directly critical the research firms can be in their assessment of those vendors. It also influences how effusive the praise is for certain vendors. A blatant example is an Aberdeen report in 2001 that concluded that Linux is less secure than Microsoft Windows. I tore apart Aberdeen's report in a post two years ago, called, Aberdeen: new poster child for sloppy research. Unfortunately, Aberdeen' original report is no longer online, but follow the link in the preceding sentence and see if you agree with me.

Now, on the whole, I have a great deal of respect for individual research analysts at the major research firms. I've met some of them and find them to be smart, insightful, and hard working. I do not mean to imply that they would deliberately slant their findings in favor of those vendors that are subscribers. But as a whole, because vendors are their clients, you are more likely to find positive or neutral assessments of vendors rather than hard-hitting criticism.

I heard one vendor refer to the subscription fees that he was paying to the major research firms as "protection." Think about that for a minute. If that's true, it's hard to avoid the conclusion that the coverage of the IT research firms is influenced by which vendors are subscribers to their research services.

It's similar to the situation with the Wall Street analyst firms. Because many of the investment analyst firms derive revenue from the companies that they research, it's rare to see a sell recommendation from them. In the same way, it's easier for IT research firms to write positive reviews of technology vendors, and to soften critical research to the point that it doesn't offend anyone.

The gravy train is over
But back to the big picture. I think that the research firms have to face the fact that there is a great deal of free information available on the Web today that was not as readily available 3-5 years ago. Furthermore, it's a lot easier for business executives today to locate individual technology experts when they need them. Yes, it is still more convenient to call up Gartner or Forrester as a one-stop-shop for all your IT research questions. But it's also expensive. Few companies, except the largest, will continue to shell out tens or hundreds of thousands of dollars every year just to have access to research and analysts.

The trend to open source research is just beginning, but I think it will develop. Technology blogs and Web-based message forums are not going to put IT research firms out of business. There will always be a need for firms like IDC, Gartner, and Forrester that have the resources and the scale to do difficult original research. But the gravy train is over.

Related posts
Research firms face threat of "open source research"
Microsoft-sponsored study on Win2K vs Linux is NOT all good news for Microsoft
Aberdeen: new poster child for sloppy research
Memo to Forrester