Python – AWS Lambda interacts with Elasticsearch

AWS Lambda interacts with Elasticsearch… here is a solution to the problem.

AWS Lambda interacts with Elasticsearch

I’m new to the AWS space, and I’m trying to develop a proof of concept that allows lambda functions to interact with elasticSearch (an AWS service) and S3 buckets.

I’m not sure if I really understand how it works. We have to set up an IAM role linked to the Lambda function, but no users, so first question:

  • Who is running the Lambda function?

I set up the Elasticsearch connection like this (using https://github.com/DavidMuller/aws-requests-auth ):

def connectES(esEndPoint):
print ('Connecting to the ES Endpoint {0}'.format(esEndPoint))
try:
    # let's talk to our AWS Elasticsearch cluster
    auth = BotoAWSRequestsAuth(
                aws_host='vpc-poc-lambda-3opu7gwdw7bwvtmmazjmdgo7gi.eu-west-3.es.amazonaws.com',
                aws_region='eu-west-3',
                aws_service='es')
    esClient = Elasticsearch(
        hosts=[{'host': esEndPoint, 'port': 443}],
        use_ssl=True,
        verify_certs=True,
        connection_class=RequestsHttpConnection,
        http_auth=auth)
    return esClient
except Exception as E:
    print("Unable to connect to {0}".format(esEndPoint))
    print(E)
    exit(3)

It seems to work, but I don’t understand what credentials it uses.
Then I successfully created an index:

esClient.indices.create('test')

But when I try to index a document like this, I get the error :

esClient.index(index='test', doc_type='post', id=123456, body={
            'author': 'John Doe',
            'blog': 'Learning Elasticsearch',
            'title': 'Using Python with Elasticsearch',
            'tags': ['python', 'elasticsearch', 'tips'],
        })

[WARNING]   2018-02-15T10:39:28.460Z    7f1cc69d-123c-11e8-be8b-cbbfad79068b    PUT https://vpc-****-****.eu-west-3.es.amazonaws.com:443/test/post/123456 [status:406 request:0.039s ]
Document not indexed
Error:  string indices must be integers

I don’t really see why it doesn’t work, wish there was an easy way to interact with Lambda’s other AWS services.

Can you help me?
Thanks

Solution

With the help of a colleague, I finally found the problem.
When you try to use an older version of the ElastiSearch Python library with a new version of the ElasticSearch server, an error is attached.
I > the library ( ) Upgrade to the latest version, which now works fine.

I’m sorry if I bothered you and hope this helps people like me.