Using Python Requests With Elasticsearch

Summary

Elastic provides a great elasticsearch library for python. However if you have lots of other rest services you might prefer to standarise on using python requests for interacting with all of your services.

This article illustrates how to use python requests with elasticsearch.

Setup

I have a docker-compose in the following github repo with nginx(with a self signed SSL), kibana and elasticsearch running. https://github.com/swarmee/projects/tree/master/elastic-stack-6.4

This docker-compose file includes a short lived container that loads up a nested template and some sample data. So you should just be able to run docker-compose up to bring everything up.

Remember that your host needs to have the following setting.

sudo sysctl -w vm.max_map_count=262144

Once everything is booted up you will be able to access:

• Kibana @ https://localhost/kibana with the username: test and password: test
• Elasticsearch @ https://localhost/ again with the username: test and password: test

The first time you open up Kibana it will force you to setup a default index pattern for kibana --> please use real-estate-sales and saleDate as the time series.

Installing Python Requests

Python requests can be installed using sudo apt-get install python-requests

or using pip (if you are lucky enough to have it) sudo pip install requests

Confirm Connection Between Python and Elasticsearch

Now fire up your python IDE (or vi) and copy in the below.

import requests

r = requests.get('https://localhost/search', auth=('test', 'test'),verify=False)

print(r.text)

You should get the basic elasticsearch response.

Now to Do the Normal Python Thing

We can search nested indexes in elasticsearch then output the JSON.

import json 
import requests
import urllib3
urllib3.disable_warnings()

host = 'localhost'
requestHeaders = {'user-agent': 'my-python-app/0.0.1', 'content-type': 'application/json'}
requestURL = 'http://%s:9200/real-estate-sales/_search' % (host)

nameSearchTerm = input('Enter Name to search for -->  ')

def populateJson(nameSearchTerm):
    return {
  "query": {
    "nested": {
      "path": "role.party.name",
      "query": {
        "match": {
          "role.party.name.fullName": nameSearchTerm
        }
      }
    }
  },
  "aggs": {
    "partyNesting": {
      "nested": {
        "path": "role.party"
      },
      "aggs": {
        "partyFilter": {
          "filter": {
            "nested": {
              "path": "role.party.name",
              "query": {
                "match": {
                  "role.party.name.fullName": nameSearchTerm
                }
              }
            }
          },
          "aggs": {
            "nameNesting": {
              "nested": {
                "path": "role.party.name"
              },
              "aggs": {
                "nameDetails": {
                  "terms": {
                    "field": "role.party.name.fullName.keyword",
                    "size": 10
                  }
                }
              }
            },
            "addressNesting": {
              "nested": {
                "path": "role.party.address"
              },
              "aggs": {
                "nameDetails": {
                  "terms": {
                    "field": "role.party.address.streetAddress.keyword",
                    "size": 10
                  }
                }
              }
            },
            "accountNesting": {
              "nested": {
                "path": "role.party.account"
              },
              "aggs": {
                "nameDetails": {
                  "terms": {
                    "field": "role.party.account.number",
                    "size": 10
                  }
                }
              }
            },
            "identificationNesting": {
              "nested": {
                "path": "role.party.identification"
              },
              "aggs": {
                "nameDetails": {
                  "terms": {
                    "field": "role.party.identification.identifier.keyword",
                    "size": 50
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "size": 0
}

requestBody = populateJson(nameSearchTerm)
r = requests.get(requestURL,
                 json=requestBody,
                 auth=('test', 'test'),
                 verify=False,
                 headers=requestHeaders)
r = r.json()

print(json.dumps(r , sort_keys = True, indent = 2, ensure_ascii = False))