Leveraging Scripted Fields In Elasticsearch

Summary

Ok, some times you really want your results to come back in a specific order based on a specific field. E.g. Hot jobs, Normal jobs and old jobs (on a jobs board). Within each category you want the results to be ordered by each documents score/relevance. This post provides a simple example on how to achieve this result.

Setup

If you want to test on your machine - I have a docker-compose in the following github repo with kibana and elasticsearch https://github.com/swarmee/projects/tree/master/elastic-stack-6.3 This docker-compose file includes a short lived container that loads up a nested template and some sample data. So you should just be able to run docker-compose up to bring everything up.

Remember on linux machines you need to have the folowing variable set. sudo sysctl -w vm.max_map_count=262144

Sorting Results on Specific Field

Often there is a field within your dataset that should be used to prioritised the results returned. In this example 'Deceased Estates' should be returned first followed by 'Bank Initiated' sales, then all normal sales. This is a snap with painless scripting, as illustrated below. Of particular note the track_scores parameter has been set to true. What this means is that within the groups the results will be sorted by their overal relevance.

GET /real-estate-sales/_search
{
  "track_scores": true,
  "sort": [
    {
      "_script": {
        "type": "number",
        "script": {
          "lang": "painless",
          "source": "int sortOrder = 0;       if (doc['transactionType.keyword'].value == 'Deceased Estate') {sortOrder =  1}     else if (doc['transactionType.keyword'].value == 'Bank Initiated') {sortOrder =  2}       else if (doc['transactionType.keyword'].value == 'Normal') {sortOrder =  3} sortOrder; "
        },
        "order": "asc"
      }
    }
  ],
  "query": {
    "nested": {
      "path": "role.party.name",
      "query": {
        "match": {
          "role.party.name.fullName": "John"
        }
      }
    }
  }, "_source": "transactionType", "size": 200
}

The results look like this. As you can see the the required order has been enforced.

{
  "took": 33,
  "timed_out": false,
  "_shards": {
    "total": 3,
    "successful": 3,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 26,
    "max_score": 4.5627613,
    "hits": [
      {
        "_index": "real-estate-sales",
        "_type": "sales",
        "_id": "169",
        "_score": 4.347633,
        "_source": {
          "transactionType": "Deceased Estate"
        },
        "sort": [
          1
        ]
      },
      {
        "_index": "real-estate-sales",
        "_type": "sales",
        "_id": "291",
        "_score": 4.347633,
        "_source": {
          "transactionType": "Deceased Estate"
        },
        "sort": [
          1
        ]
      },
      {
        "_index": "real-estate-sales",
        "_type": "sales",
        "_id": "387",
        "_score": 4.0260463,
        "_source": {
          "transactionType": "Deceased Estate"
        },
        "sort": [
          1
        ]
      },
      {
        "_index": "real-estate-sales",
        "_type": "sales",
        "_id": "443",
        "_score": 4.0260463,
        "_source": {
          "transactionType": "Deceased Estate"
        },
        "sort": [
          1
        ]
      },
      {
        "_index": "real-estate-sales",
        "_type": "sales",
        "_id": "123",
        "_score": 4.5627613,
        "_source": {
          "transactionType": "Deceased Estate"
        },
        "sort": [
          1
        ]
      },
      {
        "_index": "real-estate-sales",
        "_type": "sales",
        "_id": "11",
        "_score": 4.347633,
        "_source": {
          "transactionType": "Bank Initiated"
        },
        "sort": [
          2
        ]
      },
      {
        "_index": "real-estate-sales",
        "_type": "sales",
        "_id": "129",
        "_score": 4.0260463,
        "_source": {
          "transactionType": "Bank Initiated"
        },
        "sort": [
          2
        ]
      },
      {
        "_index": "real-estate-sales",
        "_type": "sales",
        "_id": "231",
        "_score": 4.0260463,
        "_source": {
          "transactionType": "Bank Initiated"
        },
        "sort": [
          2
        ]
      },
      {
        "_index": "real-estate-sales",
        "_type": "sales",
        "_id": "207",
        "_score": 4.347633,
        "_source": {
          "transactionType": "Normal"
        },
        "sort": [
          3
        ]
      },
      {
        "_index": "real-estate-sales",
        "_type": "sales",
        "_id": "211",
        "_score": 4.347633,
        "_source": {
          "transactionType": "Normal"
        },
        "sort": [
          3
        ]
      }
    ]
  }
}