Back to top

Gartner Magic Quadrant for Insight Engines – Our Take on the Emergence of Natural Language Processing and Proactive Search

The new Gartner Magic Quadrant for Insight Engines* was recently released, effectively replacing Gartner’s Magic Quadrant for Enterprise Search. In this blog, we'll give our take on what we’ve seen in our projects that confirms much of what Gartner notes as emerging trends. You can access the summary or download the full report from Gartner’s website. You can also get the report from Sinequa or Coveo, both of whom were listed as “Leaders” in this MQ.

Enterprise Insights beyond the Search Box

With enterprise search becoming intelligent enterprise assistants – a growing trend we’ve seen in recent projects – the launch of the Insight Engines MQ is a logical replacement for Gartner’s Enterprise Search MQ. Gartner defined "insight engines" as:

“Insight engines provide more-natural access to information for knowledge workers and other constituents in ways that enterprise search has not. Application leaders will benefit from comparing insight engine and search vendors with their own current and future needs…

… They extend beyond enterprise search by providing the capability to engage with content and extract insights without touching the source of that content.”

Last year, when Gartner published its first article on this topic, titled “Insight Engines Will Power Enterprise Search That Is Natural, Total and Proactive,” we asked our Chief Architect, Paul Nelson, for his thoughts. In his blog, he commented:

“For years now, customers have been asking for question answering systems like Siri. With Google Now and Cortana, these systems are starting to become ever more ubiquitous and therefore more in demand. 

This just tells me that there’s really a groundswell for question answering systems [or insight engines].”

Recently, in one of our guest blogs, Laurent Fanichet of Sinequa – a Leader in the MQ and a Search Technologies’ partner – discussed how this type of intelligent search is impacting modern business use cases:

“Cognitive search brings to data-driven organizations a new generation of search enabling them to go far beyond the traditional search box, empowering its users to get immediate and relevant knowledge at the right time on the right device.” 

Moving Towards Natural Language

Understanding the query in natural language is the key to the highest possible quality search. This demand is so significant that Google Search has moved towards making search more natural. 

Try asking Google a question – “What’s natural language processing?” You’ll notice that the top relevant answer is displayed prominently in what Google calls Featured Snippets rather than just a list of results.

natural language processing featured snippet.jpg

With Google, Microsoft, and Apple leading this initiative, the enterprise search and business analytics world followed, as our Chief Architect remarked: 

“Since Google is doing question-and-answer, our customers are coming to us to ask us how to do question answering within their search box.”

But the key challenge in making natural search work in the enterprise is domain understanding. Generic digital assistants (e.g. Siri, Google Now) only understand a very broad, generic domain – things like places, recipes, biographies, etc. But that’s not what business users look for in their daily work.

After all, each of our customers wants to create an insight engine of their own world, whether it is in their intranet, e-commerce catalogs, employee database, publishing database, or public sector content. They have their own languages, acronyms, metrics, and processes – and they expect their insight engines to understand their unique domain in order to answer questions or execute actions like: 

  • “What languages are our employees fluent in?”
  • “What’s the revenue of women’s shoes sold in EMEA last quarter?”
  • “Reserving conference room A.”

So, although insight engines have the capabilities to provide metadata management, natural language interfaces, and knowledge discovery beyond search, these systems would need to be heavily tuned to be able to handle domain-specific questions.

Building the Insight Engine of Your World

insight-engines.jpgAt Search Technologies, our Natural Language Processing (NPL) Toolkit, including text processing, language processing, entity extraction, and advanced pattern/relationship recognition based on machine learning, is leveraged for a multitude of search and analytics use cases.

When asked about their latest work in developing a question-answering system or insight engine, our experts shared this:

“If you combine Cloudera Search (based on open source Solr) with Spark, Hadoop, and an NLP toolkit, you have the rudimentary building blocks for a custom DIY stack for Insight Engines.”

We also do a lot of work with Microsoft, including search application deployments on the Azure ecosystem – Azure Search, Azure Machine Learning, Azure HDInsight, and Stream Analytics. We recently partnered with Coveo to deliver consulting and implementation to customers looking to implement next-generation search applications built on insight engines.

"Innovative companies are already seeing the value of insight engines, through automatically suggesting and recommending the most relevant information to anyone that interacts their business. By harnessing the data of every customer, partner or employee’s digital journey, these Coveo enabled companies are able to recommend content that has already proven to improve business results and outcomes, at scale, one-to-one, in context," Coveo CEO, Louis Tetu, wrote in his blog after the MQ announcement. 

To learn more about some approaches and techniques for building an insight engine, read our Chief Architect’s white paper here.

What about Elasticsearch and Open Source?

Anyone following the search / insight engine market will notice that Gartner’s Insight Engines MQ makes only minor mentions of open source search solutions, of which Elasticsearch and Solr are the most well-known. Gartner’s MQ criteria may have disqualified these pure open source solutions; however, they are prominent in the market.

In the case of Solr: 

  • It is represented in Lucidworks Fusion in the MQ.
  • It is also at the core of Cloudera CDH (Cloudera Distribution of Hadoop), in the form of Cloudera Search. Cloudera Search has had a lot of traction in this market (and in our projects) but is not a candidate for the MQ because it is not a standalone offering.

In the case of Elasticsearch:

  • The absence may be surprising considering that it has surpassed Solr in terms of popularity and momentum among open source solutions, especially when paired with Kibana for analytics and visualization.
  • In our own discussions with our partner, Elastic - the commercial distributor of Elasticsearch and Kibana, the absence of Elasticsearch on the MQ is partly due to the fact that Elasticsearch is not an end-to-end solution that fits Gartner’s criteria for the Insight Engine MQ.   
  • Elasticsearch is actually embedded in many other solutions for search and analytics. An example is Search Technologies’ customizable Google Search Appliance replacement solution built around Elasticsearch.

Let’s Not Forget the Cloud

Last but not least, we noted that there is a trend towards cloud-based search / insight engine solutions. Many vendors are offering cloud-based and hybrid (cloud plus on-premises) solutions. Just this week, Google announced an Elasticsearch-based solution for Google Cloud. This is another player besides Amazon’s CloudSearch (based on Solr) and Microsoft’s Azure Search (an abstraction of Elasticsearch).

As organizations collect more data, cloud solutions have emerged as cost-effective, highly scalable alternatives for data storage and processing. So we expect to see increasing interest and adoption of these cloud-based search / insight engines in the enterprise world. 

*Gartner's Magic Quadrant for Insight Engines, authored by Whit Andrews, Guido De Simoni, Jim Murphy, and Stephen Emmott, was published in March 2017.