A Methodology for Improving Search Results Relevancy and Overall Quality
Understand the Complete Search Picture
A Search Quality Analyst understands the complete picture of the user’s search experience and all of the “touch points” which can be adjusted to improve end-to-end user satisfaction and conversion rates.
In addition, the Search Quality Analyst can help business and IT stakeholders evaluate and understand the current state of the architecture and where problem areas may lie that ultimately affect search results and user satisfaction.
The Search Improvement Process and the Role of a Search Quality Analyst
As previously stated, business analysts on ecommerce search teams often don’t have the time or experience to do the analysis necessary to recommend how to “fix” what is broken with search. Search Technologies offers the services of a full time Search Quality Analyst with deep search expertise to fulfill this role. The Search Quality Analyst:
- Is dedicated to search quality
- Communicates between business & search development to ensure business needs are met
- Understands how to analyze query logs
- Understands how to “fix” search by identifying necessary changes to:
- Content processing rules
- Query processing rules / query structures
- Index & Query dictionaries, ontologies, synonyms, etc.
- How exception cases are handled
- Content text and metadata
- Document improved results and compute ROI
Knowing What to Fix
As important as it is to know how to fix search quality issues, it’s equally important to know how to prioritize what search query issues to address. A Search Quality Analyst accomplishes this by:
- Reviewing Query Logs
- Analyzing Random Samplings for Use Cases / Query Types
- Suggesting improvements per user case / query types
- Maintaining a list/database of queries & categories (helpful in understanding the “long tail” or large population of important but less frequently run user queries)
- Identifying specific queries that, if sufficiently popular, should be handled with specific rules to generate the best results
This analysis also helps the Search Quality Analyst make a determination as to what query problems are most important to fix (prioritization) and how to fix them.
As a further improvement to this methodology, Search Technologies recommends including search engine scoring as a key metric produced by automated tools for log file analysis. While not a prerequisite for Search Quality Analysis, Search Engine scoring provides key insights into how the search engine is performing, focused on the user’s experience.
Finally, the Search Quality Analyst can design and lead the implementation of A/B testing of search improvements and the analysis of the resulting logs.
In addition to verifying the accuracy and quality of implemented search engine improvements, A/B testing can tie these gains to standard KPIs such as abandonment rate and conversion rates. This further helps to quantify bottom-line improvements and ROI calculations.
In our experience customers of Search Quality Analyst services have seen the following benefits:
- Improved search quality of 20% to 55% in queries returning the best possible results
- Improved customer satisfaction results for search users
- Better communications between business and development stakeholders on the search team, resulting in more agile responses to changes / fixes when problems arise
- Substantive increase in conversion rates and revenue
While our Search Quality Analysts are engaged on a monthly basis, it is likely that the best results and most effective continuous improvement processes are established in a multi-month engagement.
For more information about engaging a Search Quality Analyst or other Search Technologies services, please CONTACT US.