Topics

“Insight Engines” win praise from Gartner1 every year. You have read the reports. You might have even started implementation. But what exactly is an insight engine, and what do you need to leverage one effectively?

We cut through the jargon and give you the skinny on the 12 capabilities Insight Engines must have for scalable enterprise deployments.

What an Insight Engine Does

An insight engine combines keyword search, data connectors, UI components, and Machine Learning to provide customers and employees alike with better information. Where a database tries to structure data to answer a few select questions, an insight engine uses structured and unstructured data to answer an almost infinite number of questions.

An insight engine is not just “search” but search, data, and intelligence. The difference between a search engine and an insight engine is the difference between a wheelbarrow and a self-driving electric pickup truck.

What an Enterprise Insight Engine Must Have

#1 Connectors and text-preprocessing

Most organizations have data everywhere in many different systems and many different formats. Loading and transforming data might not be the most attractive feature of an insight engine, but it is a critical foundation. If you cannot get data from your systems, you cannot get the answers you need from your insight engine.

 

Coveo connects to hundreds of different types of systems: learn more>>

#2 One Unified Index

Think about the last time you spoke with a large healthcare or service provider. Frequently they put you on hold while they look something up or transfer you to a different person. Often, their websites are not navigable or searchable because the information you need is in multiple places.

Better answers require unifying data in one central index. Entropy is the nature of data. Data disperses and becomes less coherent. While most search is contextual, answers often require looking in multiple places, including answers from data in multiple formats.

Humana transformed its user experience using Coveo

#3 Instant Results: Customers and Employees Can’t Wait!

When was the last time that you went to a website, used the search bar or really any interface component, saw it spin, and you just waited? Oh, you left? Customers do not wait either. Moreover, while challenging to measure, the cost of employee wait time continually drains dollars and cents out of a company. In a modern business, answers should be both instant and correct.

An insight engine combines results into a single index and suggests queries to users as they type. It does not require going back to the source system to provide instant gratification.

#4 Facets and Filters

Facets and filters may not be as new or shiny as Artificial Intelligence, but any insight engine should allow searching, sorting, and filtering on categorical data called “facets.”

Filtering on a facet is how a worker limits their search to a particular domain (i.e., Human Resources Documents), or a customer searches on a specific category (i.e., Shoes). Most searches are pretty straightforward, but your most dedicated employees and customers likely dig deeper, and these tools make their life simpler.

 

#5 ‘Know what I mean?’ NLP and Intent Detection

For a person or an insight engine to answer a question, they need to know what the question is and what the person means. Natural Language Processing goes beyond mere keyword matching and derives actual meaning from the question. NLP finds synonyms and matches phrases.

People do not always say what they mean or ask complete questions. According to Coveo customers, most searches are two words or less. A modern insight engine needs to not only understand the language or just match keywords but understand user intent based on user behavior.

Learn more about NLP, NLG, and NLU >>

#6 Automatic Relevance Tuning: ART of Relevance

Perfect results may not come from matching what users asked for with the data at hand. An insight engine is smart enough to notice when users are not getting the best result. Tuning a query is not even a one time fix. The meaning of a search may change over time. For example, consider a general ecommerce retailer. In October 2019, the search “mask” should return a different result than it did by March 2020 at the onset of the COVID-19 pandemic.

Modeling, interpreting, and tuning to user behavior must be automatic in the modern world. The days of pouring over logs and getting user feedback to tune results are over. In most cases, too many users look at too much data on too massive of a scale to make this possible. An insight engine should see this behavior and modify itself.

Learn more about Automatic Relevance Tuning (ART)>>

#7 Rules

 

“No one knows how to spell HIPAA” said Eric Immerman, practice director for Perficient, while working with Humana. While an insight engine might eventually notice this and tune itself appropriately — sometimes users are really creative with spelling.

Moreover, even the best AI algorithm requires a lot of data before it learns. Sometimes you just need a simple rule to tell the insight engine “when they say “HIPPA” or “HIPPO,” they mean “HIPAA.” Rules have a lot of usages beyond spelling.

Sometimes you want a higher margin product (yes insight engines are even for ecommerce) or this year’s new employee benefits manual near the top even though something else had more clicks.

Learn more about Rules and Ranking Expressions>>

#8 Question Answering and Chat Bots

In the early days of the net, every question was answered with a search engine result page (or SERP). These days if you ask Google, Siri, or Alexa any number of questions, you get a direct answer.

If you type “What is the weather” or “Who was Milton Freedman,” you get a direct answer. Insight engines use a combination of NLP and other techniques to not just provide search results but actual answers.

Question Answering technology is useful for everything from giving employees answers in an Enterprise Search application to handling support requests.

However, not all questions are just a question and an answer. These days Chatbots can answer questions that have “branches” or need more context.

For instance, if you ask, “Where can I download a driver for my webcam,” it might ask your model number and which computer you have. If you have questions about what your health insurance covers, maybe it asks which policy you have or where you live. Chatbots guide a user to answer beyond what search can provide.

Learn more about ChatBots>>

#9 UX Components and Tools

Nearly every search application has a search bar, query suggestions (aka type-ahead), and a slew of user interface and user experience traits. Redeveloping them for every application is absurdly wasteful, which is why an insight engine provides prewritten components that can be composed using JavaScript or TypeScript.

Learn more about the Coveo Javascript Search Framework>>

#10 Deployment Flexibility Across Clouds

The world is moving to the public cloud quickly, but not everything lives there yet, and some information requires more careful handling. An insight engine is only truly useful if it can handle data from both public and private clouds.

A cloud-based insight engine that adheres to robust enterprise security protocols, provides the flexibility to connect and index content both within and beyond the organization’s internal data sources. Needing to index and unify content that lives on an arcane internal file server, Sharepoint, and Google Drive, is typical for most enterprises.

Learn more about how Coveo handles private cloud content>>

#11 Security

Most businesses have a team of infosec professionals trying to keep data safe and systems secure. However, your security is only as good as your weakest link. An insight engine must preserve ownership and permission information. It must encrypt data in transit and at rest.

To truly maximize the value from an insight engine, you need the confidence that comes from knowing a third party has audited and assessed vendor practices and procedures as well as the software itself.

Learn more about security at Coveo>>

#12 Contextual personalization

Being responsive to user requests and giving them what they want, is the base level of search. Insight engines go much further: suggesting what other content will be helpful to them – despite not being asked. We call those “recommendations” – and anticipate what visitors will need next,

Understanding the context of what people are searching for, what they have already done, and what others like them have already found useful, provides the basis for personalizing relevant results and recommendations that meet their immediate needs, and guide them to what they are going to need next – all delivered automatically using specific machine learning models for each purpose.

Beyond an Insight Engine

However, those are really the minimum for a modern insight engine. You can get somewhere on a bus, but if you can take a high-speed electric train and get there faster and cheaper, why would you go the more challenging route?

Modern software is necessarily complex. There are multiple options, topologies, connection points, and more. However, the configuration and administration should not be complex. Ease of configuration is not something your users will notice. Simple configuration might not even be something employees see daily, but it is more than added “chrome.”

Ease of configuration and administration allows you to better use, tune, and get better results from your insight engine. It also saves time and money. If DevOps professionals have to jump hoops, there is a cost.

However, simple configuration and administration is also a security feature. While bugs can happen in any software, a quick perusal of the news reveals that many if not most major security breaches have had something to do with poor configuration and how hard it is to administer the software.

Simple configuration is a great feature, and all of these insight engine tools are helpful, but life, search, and AI are not standing still. Ideally, the team developing your insight engine always thinks about making you and your customers’ life more straightforward. Ideally, there is ever more R&D going into AI, chatbots, and implementation practices.

Learn more about the latest developments from Coveo’s AI Labs>>

Insight Engines Are Where Intelligence Meets Search

While full-text-search is at the core, as you can tell, a lot more than a search engine goes into an Insight Engine. While these 12 features are essential aspects of an insight engine, Coveo goes much further. For Coveo, developing an insight engine is about doing everything necessary to make people’s lives easier, whether competing in the marketplace, searching for employee benefits, trying to shop or the security or DevOps professionals responsible for tuning and keeping it all working. No matter what your interests, Coveo is here to help make your digital experience a great one.

Learn more about how intelligent search and enterprise search are finding the answers customers and employees need.

 

1-Gartner, Magic Quadrant for Insight Engines, Stephen Emmott, Anthony Mullen, 17 March 2021.

All statements in this report attributable to Gartner represent Coveo’s interpretation of data, research opinion or viewpoints published as part of a syndicated subscription service by Gartner, Inc., and have not been reviewed by Gartner. Each Gartner publication speaks as of its original publication date (and not as of the date of this report). The opinions expressed in Gartner publications are not representations of fact and are subject to change without notice.

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. or internationally, and is used herein with permission. All rights reserved.

Share this story:

About Andrew Oliver

Andrew C. Oliver is a developer, technologist, and writer. He's worked on data, search, and software architecture for some of the world's largest companies. His regular column in InfoWorld covers AI, cloud, and data architecture along with other developer topics.

Read more from this author