Artificial intelligence (AI) and Natural Language Processing have the potential to propel organizations through digital transformation. And in the midst of what are challenging economic times, machine learning is becoming more relevant, as they enable organizations to jump-start and accelerate change in order to remain competitive in an environment where customers’ experiential expectations have skyrocketed. 

Find out exactly where your organization currently stands with our Coveo Relevance Maturity Model white paper.

Cognitive Framework
The Coveo Relevance Maturity Model

AI Transforms Enterprise Search into Intelligent Search it has transformed enterprise search into intelligent search, and it’s having a significant cross-industry impact on customer experience, and subsequently ROI.

With simple search solutions, manual coding has been relied upon to address the inconsistency in content, the ambiguity in language, and the uncertainty in intent. And so the application of data and AI represents an opportunity to address these challenges efficiently and at scale, as it enables the collection of rich metadata, automated tuning of relevance scoring, and so much more.  

What is machine learning?

Machine learning improves from and makes predictions on data. Applied to search, every time a user performs an action on your website or support portal, he or she provides data about what’s useful.

Did they submit a support ticket? That means the articles they just read did not help. Do most people spend only one minute with a document that would normally take 10 minutes to read? That’s a sign that the content isn’t useful, or perhaps that it’s too difficult to understand.

With machine learning, all of that information and more can be used to make data-driven predictions and decisions without manual intervention, that will meet your customers’ expectations.

How will Artificial Intelligence and Natural Language processing improve machine learning and make search intelligent?

When someone submits a search query or clicks on the third search result, they are implicitly telling you what is most relevant. As your website visitors and online community members download content, visit various web and product pages, watch videos, start an online chat with your support agents, buy products, or submit support tickets, their behavior provides information on the relevance of the content they come across. This behavioral data as well as search behavior – which signals intent – can be captured by search usage analytics and acted upon in order to optimize user experience. This is a crucial step as AI Transforms Enterprise Search into Intelligent Search.

While humans are capable of undertaking this process manually, machine learning algorithms are better suited to respond to the challenges of relevance – as they can detect patterns and learn how to make predictions and recommendations by processing data and experiences, rather than by receiving explicit programming instruction.  

In a previous post, we broke down enterprise search into a process comprised of different steps – Gathering, Indexing, Processing Queries, and Returning Results – in order to show exactly where Machine Learning could potentially be applied to capture and activate that data in an intelligent manner. In this post, we’ll dive into the details regarding how machine learning can actually be applied at the stages that matter – Index and Query – in order to truly make search intelligent.

Indexing 

In an enterprise search system, once content is either pulled or pushed from its source system via a connector, its data must be processed before it can be stored in the index.

We call this Pre-index processing, and machine learning can be applied at this stage in a number of ways:

  • Summarization: Summarizes text extracting the most important sentences from documents and uses the output to augment the index.  
  • Concept Extraction: Identifies the most important concepts of a document. 
  • Terminology Extraction: Automatically extracts relevant terms from a long query that can be used to find related content.
  • Collocation Extraction: Identifies words that frequently occur together. 
  • Hyperparameter Optimization: Consisting of a mix of local search, simulated annealing and genetic algorithms, this technique is used to optimize/personalize the model parameters for any use case. 

Processing Queries

After content is collected, classified, and further enriched through the processes highlighted in the section above, queries are processed in order to extract attributes that can be used to match a given query to any relevant content.

Once again, there are a variety of machine learning techniques that are applicable at this stage: 

  • Word sense disambiguation: The user’s context is used to make sense of the query words and understand what the user looking for. 
  • Statistical spelling and vocabulary correction: A statistical machine learning algorithm is used to correct erroneous keywords and find good documents based on all users’ search sessions. 
  • Automatic language detection: Automatically detects the language of each query. 

Returning Results

Once queries have been processed, content is then returned to a user in ranked order based on its degree of relevance to a user’s query.

Machine learning can be applied at this point to draw deeper connections between queries and content so as to boost the relevance of search results that are returned to a user, and this application takes on various different forms, such as: 

  • Question Answering: Automatically identifies queries that are questions and answers them directly by using part-of-speech tagging, GloVE embeddings, vector distances, string-based distances, principal component analysis, classification, etc. to extract relevant sub-sections of documents and return them to the top of the results. 
  • Semantic Search: Word embeddings-based search that calculates the semantic similarity between queries and content. This is done in order to determine which content is the most relevant in a given situation and have that be reflected in the ranking of search results. 
  • Query Suggestions: Based on a user’s actions, content, and one or more letters typed, it suggests query expansions.
Query Suggestions in action

Without usage analytics data, many of the above techniques and capabilities can still be developed. However, it would require administrators to do so manually by creating boosting rules, adding synonyms, promoting documents, etc.

Unfortunately, due to the fact that relevance is an ever-evolving process – the document that was the most relevant last week may no longer be relevant today – it is almost impossible for administrators, especially at large organizations or those with multiple product lines, to keep pace with the rate of change.

With machine learning, highly manual and complex enterprise search can be transformed into intelligent, self-learning, and self-tuning search where all of the above techniques and capabilities are automated in a manner that is geared toward boosting search relevance and optimizing the individual user experience as a result. 

With cloud-based, self-learning search, all the required components are hosted and managed by a vendor, such as Coveo. Due to its scalability, it has the potential to change the digital journeys we take, whether as customers or employees.

In the past, the high cost of using and managing machine learning systems meant that machine learning was rarely used for traditional enterprise searches or websites. The cloud makes that affordable for all customers and departments, especially when deploying self-learning search on self-service support sites, communities, and across digital properties that contain vast amounts of data.

Why now?

Machine learning has been around for a long time. It used to be very complex to deploy and manage. Collecting usage data, managing databases, provisioning servers, developing and maintaining machine learning algorithms and using machine learning predictions in the search system has traditionally been complicated and hard to achieve. This required data scientists, database experts and developers.

But the fast adoption of cloud solutions has made the use of machine learning much easier, cheaper and more attainable. Intelligent search technology is no longer a nice-to-have. It is the new normal and without it, you will not be able to meet the growing demands of your customers – especially as many organizations have already found ways to further elevate its capabilities. 

Don’t wait. Effectively leveraging your company knowledge and making it accessible to your customers and employees alike is the only way to keep your customers from jumping ship to your competitors – who are likely already delivering self-service experiences though intelligent search and looking to do even more in a post-pandemic future.