Back to top

Can Enterprise Search Experiences Guide “Big Data” Strategy?

Companies who use enterprise search effectively may be best placed to exploit big data

Since the 1990s, enterprise search vendors, supported by industry analysts, have proposed that time and money can be saved by making information more easily available across the organisation. We agree. 

Numerous surveys have supported this notion over the years, for example:

  • In 2001, Outsell Inc, in a survey of 6,300 US companies, concluded that time spent seeking information represented about $10,000 per employee per year
  • In 2006, the Butler Group concluded in a report entitled “Enterprise Search & Retrieval”, that as much as 10% of corporate salary costs were frittered away as employees scramble to find the information they need

Yet surveys continue to show dissatisfaction. For example:

  • In 2001, an IDC study entitled “Quantifying Enterprise Search” concluded that “Searchers are successful in finding what they seek 50% of the time or less”
  • In 2011, a MindMetre/Smartlogic report called “Mind the Enterprise Search Gap”, concluded that “less than half enterprise search applications meet expectations”

Should we be surprised by such ongoing negativity?  Given the amount of investment that has been made in search engine technology during the past 10 years by some of the world’s largest software companies, perhaps we should be surprised. But anecdotally, we know that these surveys reflect the realities of life in many large organizations. For one reason or another, most users still struggle to find the information they need.

A 2009 Aberdeen Group study of 118 companies showed significant differences between companies who used enterprise search effectively and those who did not.

 "Executives at top performing companies spent six hours a week less, personally looking for information, compared to just one hour at other companies."

That’s a big difference.

Search Technologies, a specialist IT service firm focused on search engine implementation and with a roster of 300+ corporate and government customers, believes that a major factor is “data neglect”.

Corporate IT departments and decision makers are used to dealing with structured data:

  • It comes in neat, tidy rows and columns
  • It is largely produced by automated processes which are consistent and predictable

In contrast, enterprise search primarily deals with unstructured information, produced almost entirely by humans – sometimes lazy, careless, enthusiastic, long-winded, emotional, biased, thoughtful, diligent, unpredictable, but always unique, human beings. 

Human-created, unstructured content cannot be treated the same as structured data.

Search Technologies’ has built an extensive customer billing database over the past four years, which details more than 50,000 consultant-days of services delivered in support of search engine implementations.

The majority of the work we deliver in helping companies to improve existing search implementations that have so far disappointed, has a data quality focus.  

Rubbish in, rubbish out

A key issue for enterprise search is ensuring that data is appropriately processed prior to indexing, to enable important search features (which in turn drive productivity) and prevent misleading data from polluting indexes.

Dissatisfaction with enterprise search can often be addressed by massaging unstructured data into shape prior to indexing.  For example, capturing or generating metadata to drive search navigation and results sorting options, or removing misleading menu structures, or headers and footers. 

Maybe the current enthusiasm for big data will bring these issues into focus. 

During the past 15 or so years, the inefficiencies caused by the lack of an effective enterprise search capability have been largely invisible to senior executives.  They exist, are probably known about, but are not easily measurable.

But “big data” is not about saving a few minutes here and there for every employee. Big data is about insight, which is only of business value if it is actionable. 

Senior executives will be responsible for decisions taken, based on big data insights

Providence and transparency ought be important to decision makers. Would you want to take a meaningful business decision based on a data analysis, unless you fully understood the origin of the data and how it came to contribute to the analysis?  

The expertise and processes needed to prepare unstructured data appropriately for enterprise search and for big data insight, are much the same. Companies like Search Technologies have a lot of experience dealing with unstructured data, and building transparent content processing systems to support key applications.

So, the key difference between big data and enterprise search may prove to be that in the big data world, executives have a compelling reason to care.