Enterprise System Spectator blog: ERP and enterprise system vendor evaluation, selection, and implementation.

The Enterprise System Spectator

Saturday, December 11, 2004

Death of IT research firms greatly exaggerated

Last month I wrote about the threat that IT research firms face from what I called "open source research." But Mark McManus, VP at Computer Economics, an IT research firm, wrote to me with a different view.

Frank, I have to comment on your post regarding paid research versus "free" Internet research.

I agree with some of your points, but there is a whole research world regarding IT that cannot be easily found on the Internet. For instance, take our annual Information System Spending study. This is a four-month project and involves interviewing hundreds of senior IT and business executives. It is very expensive to produce and involves a significant amount of analysis. How could that be duplicated for free? I've seen statistics that claim to represent this data, but much of the time this is a self-serving process by the individual or organization that has produced the data. The buyer should beware.

While there are many good sources of free research data (such as The Enterprise System Spectator), I do not expect that senior IT and business executives will ever consider information retrieved via an Internet search as an alternative to paid, targeted research performed by a reputable (and independent) IT research firm-- especially when major competitive and financial decisions are on the line.

Another issue your commentary does not clearly address is the analysis capability of major research organizations. More often than not the main reason that end user organizations will go to Gartner or Forrester is not to get statistics, but rather to get their analysis and recommendations. IT managers performing their own research will find many sources to choose from--and many varying points of view. Do you think these extremely busy individuals will really have the time to wade through the page after page of information and do the required analysis they need to make hard technical and business decisions? Maybe in some cases, but I think that is unlikely in most.

Additionally, our own experience is that companies are using our research capabilities more in the past several months than in the past two to three years. So, even though the Globe article may be correct that "major" research firms are having a difficult time, I think that there other business issues that may be creating their hardship, such as over-extending their research capabilities and areas of expertise. In fact, our research on spending trends indicates that in 2004 more IT organizations increased their research budgets than in 2003 or 2002. The "Googlization of America" may be a minor contributing factor to IT research firm woes, but it is probably not the leading factor.
The value of aggregating and filtering information
I've known Mark for several years and appreciate his feedback. In follow up correspondence, I pointed out that in my original post I indicated that in-depth research, such as Computer Economics IT spending studies, or IDC's market share studies, that require significant resources and expense, are impossible to reproduce by individual experts contributing to message boards or writing technology blogs. I also recognize the value that paid research firms provide in filtering and aggregating information so that business and IT executives don't have to wade through it themselves. I pointed out, however, that technology blogs and group blogs (blogs authored by multiple experts) might be able to provide this aggregation/filtering service in a richer way, much like blogs such as Instapundit or Command Post do for news and current events.

The question of bias
In follow up correspondence, Mark replied,
I agree that tech and group blogs can do some filtering, but there is still an issue with bias. How can you (as the recipient) trust the underlying motives of someone who is posting free research and advice? There are certainly unbiased blogs out there, but many are built as a sounding board to support a particular position. Not that research firms haven't shown bias, but generally speaking you can usually get recommendations from an independent research firm that are not heavily tilted to a particular bias.
Again, Mark brings up a good point, which is the issue of bias. Individual technology experts do have their biases, based on what they've experienced or what technologies and vendors they are familiar with. But I would point out that the paid research firms have their own problems with bias as well. An obvious example is research firms that accept money from technology vendors to produce specific research. I wrote about one such example two years ago: IDC's report on Windows 2000, which was sponsored by Microsoft, which concluded that Windows 2000 had a lower cost of ownership than Linux. Although the report was not totally one-sided, the conclusions of the report, and Microsoft's press release on it, were clearly slanted in Microsoft's favor.

But a more subtle form of bias is seen in the fact that many of the research firms have technology vendors as major subscribers. This limits how directly critical the research firms can be in their assessment of those vendors. It also influences how effusive the praise is for certain vendors. A blatant example is an Aberdeen report in 2001 that concluded that Linux is less secure than Microsoft Windows. I tore apart Aberdeen's report in a post two years ago, called, Aberdeen: new poster child for sloppy research. Unfortunately, Aberdeen' original report is no longer online, but follow the link in the preceding sentence and see if you agree with me.

Now, on the whole, I have a great deal of respect for individual research analysts at the major research firms. I've met some of them and find them to be smart, insightful, and hard working. I do not mean to imply that they would deliberately slant their findings in favor of those vendors that are subscribers. But as a whole, because vendors are their clients, you are more likely to find positive or neutral assessments of vendors rather than hard-hitting criticism.

I heard one vendor refer to the subscription fees that he was paying to the major research firms as "protection." Think about that for a minute. If that's true, it's hard to avoid the conclusion that the coverage of the IT research firms is influenced by which vendors are subscribers to their research services.

It's similar to the situation with the Wall Street analyst firms. Because many of the investment analyst firms derive revenue from the companies that they research, it's rare to see a sell recommendation from them. In the same way, it's easier for IT research firms to write positive reviews of technology vendors, and to soften critical research to the point that it doesn't offend anyone.

The gravy train is over
But back to the big picture. I think that the research firms have to face the fact that there is a great deal of free information available on the Web today that was not as readily available 3-5 years ago. Furthermore, it's a lot easier for business executives today to locate individual technology experts when they need them. Yes, it is still more convenient to call up Gartner or Forrester as a one-stop-shop for all your IT research questions. But it's also expensive. Few companies, except the largest, will continue to shell out tens or hundreds of thousands of dollars every year just to have access to research and analysts.

The trend to open source research is just beginning, but I think it will develop. Technology blogs and Web-based message forums are not going to put IT research firms out of business. There will always be a need for firms like IDC, Gartner, and Forrester that have the resources and the scale to do difficult original research. But the gravy train is over.

Related posts
Research firms face threat of "open source research"
Microsoft-sponsored study on Win2K vs Linux is NOT all good news for Microsoft
Aberdeen: new poster child for sloppy research
Memo to Forrester

by Frank Scavo, 12/11/2004 05:07:00 PM | permalink | e-mail this!

 Reader Comments:

Post a Comment

Links to this post:


Powered by Blogger

(c) 2002-2018, Frank Scavo.

Independent analysis of issues and trends in enterprise applications software and the strengths, weaknesses, advantages, and disadvantages of the vendors that provide them.

About the Enterprise System Spectator.

Frank Scavo Send tips, rumors, gossip, and feedback to Frank Scavo, at .

I'm interested in hearing about best practices, lessons learned, horror stories, and case studies of success or failure.

Selecting a new enterprise system can be a difficult decision. My consulting firm, Strativa, offers assistance that is independent and unbiased. For information on how we can help your organization make and carry out these decisions, write to me.

My IT research firm, Computer Economics provides metrics for IT management, such as IT spending and staffing benchmarks, technology adoption and investment trends, IT management best practices, IT salaries, outsourcing statistics, and more.

Go to latest postings

Search the Spectator!
Join over 1,700 subscribers on the Spectator email list!
Max. 1-2 times/month.
Easy one-click to unsubscribe anytime.

Follow me on Twitter
My RSS feed RSS News Feed

Computer Economics
Outsourcing Statistics
IT Spending and Staffing Benchmarks
IT Staffing Ratios
IT Management Best Practices
Worldwide Technology Trends
IT Salary Report


2014 Best Independent ERP Blog - Winner 2013 Best ERP Writer - Winner Constant Contact 2010 All Star Technobabble Top 100 Analyst Blogs

Key References
Strativa: Business strategy consulting, strategic planning
Strativa: IT strategy consulting
Strativa: Business process improvement, process mapping, consultants
Strativa: IT due diligence
Strativa: ERP software selection consulting and vendor evaluation
Strativa: CRM software selection consulting and vendor evaluation
Strativa: Project management consulting, change management
StreetWolf: Digital creative studio specializing in web, mobile and social applications
Enterprise IT News: diginomica

Spectator Archives
May 2002
June 2002
July 2002
August 2002
September 2002
October 2002
November 2002
December 2002
January 2003
February 2003
March 2003
April 2003
May 2003
June 2003
July 2003
August 2003
September 2003
October 2003
November 2003
December 2003
January 2004
February 2004
March 2004
April 2004
May 2004
June 2004
July 2004
August 2004
September 2004
October 2004
November 2004
December 2004
January 2005
February 2005
March 2005
April 2005
May 2005
June 2005
July 2005
August 2005
September 2005
October 2005
November 2005
December 2005
January 2006
February 2006
March 2006
April 2006
May 2006
June 2006
July 2006
August 2006
September 2006
October 2006
November 2006
December 2006
January 2007
February 2007
March 2007
April 2007
May 2007
June 2007
July 2007
August 2007
September 2007
October 2007
November 2007
December 2007
January 2008
February 2008
March 2008
April 2008
May 2008
June 2008
July 2008
August 2008
September 2008
October 2008
November 2008
December 2008
January 2009
February 2009
March 2009
April 2009
May 2009
June 2009
July 2009
August 2009
September 2009
October 2009
November 2009
December 2009
January 2010
February 2010
March 2010
April 2010
June 2010
July 2010
August 2010
September 2010
October 2010
November 2010
December 2010
January 2011
February 2011
March 2011
April 2011
May 2011
July 2011
August 2011
September 2011
October 2011
November 2011
December 2011
January 2012
February 2012
March 2012
April 2012
May 2012
June 2012
July 2012
September 2012
October 2012
December 2012
January 2013
February 2013
March 2013
May 2013
June 2013
July 2013
September 2013
October 2013
December 2013
January 2014
February 2014
March 2014
April 2014
May 2014
June 2014
July 2014
August 2014
September 2014
October 2014
November 2014
December 2014
February 2015
March 2015
April 2015
May 2015
June 2015
July 2015
September 2015
October 2015
November 2015
February 2016
May 2016
June 2016
July 2016
August 2016
September 2016
October 2016
January 2017
February 2017
May 2017
June 2017
October 2017
January 2018
April 2018
May 2018
Latest postings