Elasticsearch ICU now understands emoji!
And that simple change in Elastic 6.4 may have a bigger impact on your indices that you might think.
Elasticsearch 6.4 is shipped with Lucene 7.4 – this is a one-liner in the official Release Notes but if you look closer, this new version ships updated ICU data and real support for emoji. And that’s a game changer 😎 (for some!).
International Components for Unicode (ICU) is an open source project of mature C/C++ and Java libraries for Unicode support, software internationalization and software globalization, it’s used everywhere (on your computer, your phone, and probably even your connected fridge).
All usages of icu_tokenizer
are impacted, that means that everyone using the must-need icu_tokenizer
should probably reindex everything, because “🍕” is now a token!
Section intitulée the-new-behavior-of-icu-tokenizerThe new behavior of icu_tokenizer
With this simple query we are testing Elasticsearch ICU Tokenizer:
GET /_analyze
{
"tokenizer": "icu_tokenizer",
"text": "I live in 🇫🇷 and I'm 👩🚀"
}
The 👩🚀 emoji is very peculiar as it’s a combination of the more classic 👩 and 🚀 emoji. The flag of France is also a special one, it’s the combination of 🇫 and 🇷. So we are not just talking about splitting Unicode code points properly but really understanding emoji here.
Let’s compare the resulting tokens of this _analyze
call with both Elasticsearch 6.3 and Elasticsearch 6.4:
Section intitulée elasticsearch-6–3Elasticsearch 6.3
- I
- live
- in
- and
- I’m
Emoji are just dropped like punctuation.
Section intitulée elasticsearch-6–4Elasticsearch 6.4
- I
- live
- in
- 🇫🇷
- and
- I’m
- 👩🚀
Emoji are kept and understood!
More tokens means more relevant search results! Here is how to take advantage of this new capability.
Section intitulée using-the-power-of-emoji-for-searchUsing the power of emoji for search
Now that you have the tokens, you can search for meaning and relevance inside the emoji world. Thousand of new words, new meanings and ways to communicate are available to you.
Your users will be able to search for a pizza place by typing “pizza”, or “🍕”, or both.
In order to add this capability to your indices, you must use the CLDR annotation for each emoji and add them as synonyms via a custom Token Filter. Here is an example:
PUT /emoji-capable
{
"settings": {
"analysis": {
"filter": {
"english_emoji": {
"type": "synonym",
"synonyms_path": "analysis/cldr-emoji-annotation-synonyms-en.txt"
}
},
"analyzer": {
"english_with_emoji": {
"tokenizer": "icu_tokenizer",
"filter": [
"english_emoji"
]
}
}
}
}
}
You may wonder how to populate the cldr-emoji-annotation-synonyms-en.txt
file? Wonder no more! Everything is already done and it looks like this:
🤧 => 🤧, face, gesundheit, sneeze, sneezing face
🧞 => 🧞, djinn, genie
🕺 => 🕺, dance, man, man dancing
👂 => 👂, body, ear
🐅 => 🐅, tiger
🍺 => 🍺, bar, beer, drink, mug
🆘 => 🆘, help, sos, SOS button
👩🚒 => 👩🚒, firefighter, firetruck, woman
🇮🇪 => 🇮🇪, Ireland
I took the time to generate properly formatted and Elasticsearch-compatible synonym files for all the languages and emoji supported by CLDR, and you can find this on Github.
That way when you search for 🐅, “tiger” is also searched, and the other way around.
Section intitulée a-complete-emoji-search-exampleA complete Emoji Search example
Let’s build a simple index, add some documents and search for them (gosh, I wish found.no/play was still working, it was like a JS Fiddle but for Elasticsearch!):
Section intitulée the-index-with-english-based-analysisThe index with English based analysis
PUT /tweets
{
"settings": {
"number_of_replicas": 0,
"number_of_shards": 1,
"analysis": {
"filter": {
"english_emoji": {
"type": "synonym",
"synonyms": [
"🐅 => 🐅, tiger",
"🍺 => 🍺, bar, beer, drink, mug",
"🍍 => 🍍, fruit, pineapple",
"🍕 => 🍕, cheese, pizza, slice"
]
},
"english_stop": {
"type": "stop",
"stopwords": "_english_"
},
"english_keywords": {
"type": "keyword_marker",
"keywords": ["example"]
},
"english_stemmer": {
"type": "stemmer",
"language": "english"
},
"english_possessive_stemmer": {
"type": "stemmer",
"language": "possessive_english"
}
},
"analyzer": {
"english_with_emoji": {
"tokenizer": "icu_tokenizer",
"filter": [
"english_possessive_stemmer",
"lowercase",
"english_emoji",
"english_stop",
"english_keywords",
"english_stemmer"
]
}
}
}
},
"mappings": {
"_doc": {
"properties": {
"content": {
"type": "text",
"analyzer": "english_with_emoji"
}
}
}
}
}
Section intitulée add-some-documentsAdd some documents
POST /tweets/_doc
{
"author": "NotFunny",
"content": "Pineapple on pizza, are you kidding me? #peopleAreCrazy"
}
POST /tweets/_doc
{
"author": "JulFactor93",
"content": "🍍🍕 is the best #food"
}
Section intitulée and-searchAnd search!
GET /tweets/_search
{
"query": {
"match": {
"content": "pineapple pizza"
}
}
}
GET /tweets/_search
{
"query": {
"match": {
"content": "🍍🍕"
}
}
}
Both those searches will return our two documents, because they both search with the emoji and the words.
{
"took": 16,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0.653994,
"hits": [
{
"_index": "tweets",
"_type": "_doc",
"_id": "-PJ172YBCoXstbpG75QM",
"_score": 0.653994,
"_source": {
"author": "JulFactor93",
"content": "🍍🍕 is the best #food"
}
},
{
"_index": "tweets",
"_type": "_doc",
"_id": "-fJ172YBCoXstbpG9pRv",
"_score": 0.3971361,
"_source": {
"author": "NotFunny",
"content": "Pineapple on pizza, are you kidding me? #peopleAreCrazy"
}
}
]
}
}
Section intitulée final-wordsFinal words
This change is great news for me as my Elasticsearch plugin Emoji Search is no longer needed! It’s a relief since building and shipping it is no fun at all. The documentation is not really helping and you have to compile your plugin for each Elasticsearch release.
Supporting emoji is now easier than ever, if you take a look at this technical article from 2016 your eyes will bleed a little, we had to use a whitespace tokenizer, some char filters and hacks everywhere. Elasticsearch 6.4 improves the search engine Unicode support and we will enjoy it.
Happy emoji searching 🔎!
Commentaires et discussions
Search for Emoji with Elasticsearch
Edit on November 2018: please have a look at our new article on emoji search because Elastic 6.4 now has a better support for Emoji! Edit on December 2016: there is now an Elasticsearch plugin for that! Since 2011 and Unicode 6.0, emoji is an integral and standardized part of…
Lire la suite de l’article Search for Emoji with Elasticsearch
Nos articles sur le même sujet
Nos formations sur ce sujet
Notre expertise est aussi disponible sous forme de formations professionnelles !
Elasticsearch
Indexation et recherche avancée, scalable et rapide avec Elasticsearch
Ces clients ont profité de notre expertise
À la recherche du meilleur moyen de réaliser une fonctionnalité, Weglot à fait appel à nos équipes pour de l’expertise Elasticsearch. Fort de notre grande expérience avec Elasticsearch, nous avons pu itérer sur différentes implémentations qui répondaient au besoin, et choisir la meilleure. En plus de donner des axes d’amélioration concrets et argumentés, …
Afin de soutenir le développement de son trafic, Qobuz a fait appel à JoliCode afin d’optimiser l’infrastructure technique du site et les échanges d’informations entre les composants de la plateforme. Suite à la mise en place de solution favorisant l’asynchronicité et la performance Web côté serveur, nous avons outillé la recherche de performance et…
Ouibus a pour ambition de devenir la référence du transport en bus longue distance. Dans cette optique, les enjeux à venir de la compagnie sont nombreux (vente multi-produit, agrandissement du réseau, diminution du time-to-market, amélioration de l’expérience et de la satisfaction client) et ont des conséquences sur la structuration de la nouvelle…