Not too long ago, shoppers used to rely on salespeople to help them find exactly what they were looking for. Salespeople were also trained well enough to understand the customer’s needs and pick out the perfect product for them, or even suggest similar options that the customer might not have been aware of.
But today’s customers increasingly shop online, and it is up to search engines to be smart enough to predict a user’s intent and provide them with an experience as close as possible to the personal touch of a salesperson.
Why Synonyms?
Imagine the simplest search engine: it would use a matching process against a simple string to provide the user with the search results. The disadvantage of this system is that even a small spelling mistake in a search query would prevent a match with the document being searched for. [1]
Building a SuperApp like Careem’s presents further complexities. The wide diversity of our user base means that users of different backgrounds, languages, or regions may search for the same item using different names or spellings. For example, one person might search for “shawarma” while another person could search “wrap”. So relying on a simple string comparison would not be sufficient.
This is where leveraging the power of synonyms in ElasticSearch helps us define a list of words that are similar to each other, and hence improve the recall of our search engine.
Gaining User Trust With Better Search Results
In Careem’s search engine for the “Food” mini-app, there were two distinct categories of problems that we tackled to create better search results:
- The customer searches for an item/store that is available in our SuperApp, but under a different name, or spelling, hence cannot find any search results. (E.g. “bread” vs. “toast”)
- The customer searches for an item/store that is not available in our SuperApp, hence cannot find any search results. However, Careem has similar options within the same category that it can suggest instead. (E.g. “Carrefour” vs. “Lulu Hypermarket”)
For both of the problems above, our goal was to provide customers with results that not only match their search query but also intelligently suggest other options that are similar to their query. This was achieved by adding the capability of synonym search within the Careem SuperApp.
The Synonyms Journey
Data Analysis
Our data science and analytics team helped analyze popular search queries that were returning empty search results – displayed in the bar chart below.
With this information, as well as data from our merchant catalogs, we manually compiled a list of synonyms for each of these queries. In the future, we hope to build out the list further by leveraging machine learning algorithms to automatically detect and store synonyms. [2]
Index-Time vs. Query-Time Synonyms
Once the list was compiled, we created a synonym token filter and a custom analyzer in Elasticsearch which utilized the synonyms dictionary that was compiled. The index mappings for grocery shops names and item names were then configured to use this custom analyzer. [3]
Elasticsearch offers the option to configure synonyms either at index-time, or query-time. Both options carry advantages and disadvantages when it comes to performance and storage.
With query-time synonyms, the synonym dictionary is looked up at every single search query, which can increase the latency of searches, and hence offer a poor user experience. On the flip side, query-time synonyms do not affect the size of the index.
With index-time synonyms, all synonyms get indexed into ElasticSearch, hence the size of the index increases. Our current dictionary in this synonyms experiment contained ~375 synonyms for items and ~1800 synonyms for shops. In production, we had about ~15.3 million documents currently indexed. So by introducing index-time synonyms, we observed an increase of about 14GB in the index size.
Looking overall at the pros and cons, the biggest advantage that index-time synonyms offer is definitely performance – all synonyms from the list need to be indexed just once and for all. By choosing performance over storage, we opted to go with index-time synonyms in the SuperApp.
Observing our current metrics for the ElasticSearch cluster, our search latency is currently between 4-13 ms per query. And the time it takes to index a document on ES varies between 0.1-1ms.
Reindexing Our Clusters
To introduce index-time synonyms, we needed to update the Elasticsearch indexes. This meant that we had to carefully re-index all of the documents without disrupting any of Careem’s services in production.
To achieve this, we decided to spin a separate Elasticsearch cluster managed via cloud, where we created the new index with synonyms. Then, we migrated all the Elasticsearch documents over to the new index, while continuing to accommodate all user search traffic on the original Elasticsearch cluster.
This upgrade activity took about 1.5 hours, during which time our customers were still able to execute search queries on the SuperApp (against the original Elasticsearch cluster).
Examples of Synonyms in Action!
The image on the left shows synonyms added to grocery items. We can see that if a user searches by the query “tissue”, then we get search results for synonyms like “kitchen roll” and “toilet paper” as well as the original query of “tissue”.
The image on the right shows a list of grocery shops that are defined as synonyms of each other. For example, a user searching for “Grandiose” will also see similar shops like “Al Maya”, “Aswaaq” and “Al Adil”.
Moreover, our system today also offers multi-language support for synonyms. For example, a user can search an Arabic word like “خبز”, and the Careem SuperApp can return a list of synonyms in English, like “bread” and “toast”.
Impact of Synonyms Experiment
Gains in Conversion
The synonyms experiment was launched for a period of one month for users in Dubai and Amman. Thirty percent of our user base was randomly selected as the treatment group. With this experiment, the primary metric of concern was the number of queries that resulted in null search results. In addition, we also monitored any impacts on user conversion.
The table below shows the results of the synonyms experiment for a user session in which a user searched for a query that included at least one word from our synonyms dictionary.
| Conversion | Null Searches | |
| Difference – Absolute (pp) | 1.5% | -0.03 |
| Difference – Relative (%) | 3.6% | -14.3% |
The results were really exciting to see – the null search results dropped by 14.3%, and conversion gains were reported as high as 3.6% for the month!
Taking a closer look at the data, we found that synonyms had a much larger impact on merchants/stores than they did on grocery items.
We concluded that this was because of the way the current implementation of synonyms performs a full-phrase match of search terms against the words in the synonyms dictionary. For example, merchant/store names are generally shorter (e.g. “Carrefour”, “Al Maya”), and so they are easier to match against search terms. On the other hand, some merchant/store item names may be longer (e.g. “Cottonelle Ultra ComfortCare Soft Toilet Paper, 12 Mega Rolls”), so matching them against synonyms like “Kitchen Roll” or “Tissue” does not yield a match.
To solve this issue, we completed a POC where the tokenizer on Elasticsearch was modified in such a way that it broke up multi-word synonyms into individual words. This allowed us to achieve higher match rates on synonyms for items as well. We plan to work on this more in the future as a second experiment.
Cutting down API Response Time
It is a well-known phenomenon that web pages that take longer to load can potentially damage the user experience. Similarly, in search engines, speed is of the essence. Not only does the user want to find exactly what they are looking for, but they also want it fast!
With this project, we also analyzed the time it took for a search query to respond with appropriate search results. We experimented with doubling the number of shards on the Elasticsearch cluster, by leveraging Elasticsearch’s essence of being a distributed search engine. The higher number of shards allowed each search query to be run against more shards in parallel, hence reducing the overall API latency. It was observed that with this optimization, our API response time reduced from ~500ms to ~250ms!
Future Work
With the great results that this synonyms experiment has delivered, we are excited to take the next steps that will help improve the SuperApp further.
- As discussed earlier, a second experiment would be launched with a modified tokenizer on Elasticsearch. This will allow us to achieve more synonym matches on grocery items and help improve conversion.
- Our synonyms dictionary for items is currently very small. This can be further enlarged by analyzing more user data for commonly-searched phrases. Theoretically, this will also allow us to get more search hits and higher conversion.
- In terms of results analysis, we would also like to measure a few more metrics to understand the impact of synonyms:
- Precision and recall: How precise are the new synonym items and shops that we are suggesting to the user?
- CTR: How often are users actually clicking on the synonym search results, vs the exact match search results?
References
[1] https://www.elastic.co/blog/boosting-the-power-of-elasticsearch-with-synonyms
[2] https://lucidworks.com/post/search-automatic-synonym-detection/
[3] https://lucasmagnum.medium.com/elasticsearch-setting-up-a-synonyms-search-facea907ef92
Written by Mariam Zaim, a Software Engineer in Careem’s Server-Side Engineering team.
