Search Algorithm Optimisation: Behind-the-Scenes of Our Project at Elsevier
Over the past couple of years, search engine scoring has become one of the key processes that we use in our projects (for a deep-dive into engine scoring techniques, watch our on-demand webinar). It acts as a continuous improvement cycle in which the accuracy of the search engine is measured against a set of queries for which we have obtained or inferred the relevant results. Within each cycle, the query set is executed against the current search engine configuration and metrics are generated, including an overall accuracy score (the engine score). The metrics and results are then analysed in order to identify improvements to the search engine. Those improvements are implemented and the next cycle starts.
Too Many Options in Your Search Algorithm?
My current project is to improve the relevance for Elsevier DataSearch - a public-facing search engine that allows researchers and scientists to search for data over many domain-specific and cross-domain content repositories. The project makes use of engine scoring and it has been incredibly valuable to be able to see, very quickly, whether a change to the algorithm has improved accuracy or not (a change doesn’t always lead to an improvement but we can often learn more from those cases).
Over time, I had implemented several new features that had led to some significant improvements in search accuracy. Each of these features had multiple parameters that could be used to tune the algorithm and those parameters had, in some cases, up to one hundred potential values.
I had ended up with an extremely flexible search algorithm but one that provided too much choice! How was I going to decide which values to set all of these parameters to? I had a good feel for many of the values in isolation but how would they interact? I did the following steps:
- Started by using engine scoring cycles to change parameter values
- Measured the resulting search accuracy
- Recorded all of the values for each cycle in a spreadsheet
- Analysed the changes in the accuracy of the search engine as a whole and of individual queries
- Used all of this data to decide on what the next set of parameters should be
It was taking me at least 30 minutes to go through each cycle and it was a toss-up as to whether I would run out of time or patience first.
As luck would have it, around that time I happened to see a video of a Lucene/Solr Revolution presentation by Simon Hughes of Dice.com in which he discusses using optimisation algorithms to tune standard Solr search parameters, such as query field boosts. He also provided some sample code on GitHub.
This approach was exactly what I needed in order to tune all of the parameters that I now had available in my search algorithm. It would enable me to automatically run hundreds or thousands of engine scoring cycles using an algorithm that could intelligently explore the parameter space.
After tailoring the code to reflect my parameters and my engine scoring data, I was able to perform my first optimisation and the results were impressive. I optimised 13 parameters with a total of 1027 possible combinations, and within four hours, I had run a few thousand iterations to find a set of parameter values which resulted in:
- A 22% increase in Precision for the top 10 results
- A 37% increase in Mean Average Precision for the top 10 results
- An 8% increase in NDCG (Normalized Discounted Cumulative Gain) for the top 20 results
An Ongoing Process
I am currently running my optimisations by constructing Solr query URLs based on the parameter values and sending them directly to Solr. The same approach would work for any search engine that can receive queries by HTTP. However, the optimisation becomes even more powerful if the search requests can be sent to an API endpoint that can perform complex logic to construct the final queries. This could be our Query Processing Language (QPL) embedded in Solr or Elastic or an external client such as SolrJ.
Search algorithm optimisation should not be a one-off process. The optimisation should be re-run whenever there is new data on which to optimise and/or new parameters are added to the search algorithm. For my project, I re-run the optimisation every three weeks. And although the improvements are not as significant as the first time, each optimisation cycle results in an increase in accuracy. The optimisation has become an ongoing process which has evolved to reflect the changing search algorithm and this has also led to further improvements in the tool. For example, I have added multi-threading to enable queries to be sent in parallel and additional optimisation algorithms to explore the parameter space in alternative ways.
The graph below is an example of how quickly the optimum can increase even over a relatively small number of iterations. Here, two different optimisation algorithms are compared for a specific case.
Search Algorithm Optimisation – the Next Step from Engine Scoring
Once you have implemented engine scoring, you will already have most of what you need to start optimising your search algorithm. Engine scoring provides the ability to measure the resulting accuracy of a specific configuration and this is one of the two key components of optimisation. The other component that you will require is the optimisation algorithm to explore the parameter space and to call engine scoring on each iteration.
Optimisation is not a replacement for engine scoring; instead, I see it as a complementary tool in the ongoing process of improving search relevance. Engine scoring should be used to identify and test new search features which optimisation can then tune.
I have found search algorithm optimisation to be so powerful that I will be using it as part of future search engine scoring projects. If you would like to find out how it could help improve your search, please get in touch.