AWS Lambda interacts with Elasticsearch
I’m new to the AWS space, and I’m trying to develop a proof of concept that allows lambda functions to interact with elasticSearch (an AWS service) and S3 buckets.
I’m not sure if I really understand how it works. We have to set up an IAM role linked to the Lambda function, but no users, so first question:
- Who is running the Lambda function?
I set up the Elasticsearch connection like this (using https://github.com/DavidMuller/aws-requests-auth ):
def connectES(esEndPoint):
print ('Connecting to the ES Endpoint {0}'.format(esEndPoint))
try:
# let's talk to our AWS Elasticsearch cluster
auth = BotoAWSRequestsAuth(
aws_host='vpc-poc-lambda-3opu7gwdw7bwvtmmazjmdgo7gi.eu-west-3.es.amazonaws.com',
aws_region='eu-west-3',
aws_service='es')
esClient = Elasticsearch(
hosts=[{'host': esEndPoint, 'port': 443}],
use_ssl=True,
verify_certs=True,
connection_class=RequestsHttpConnection,
http_auth=auth)
return esClient
except Exception as E:
print("Unable to connect to {0}".format(esEndPoint))
print(E)
exit(3)
It seems to work, but I don’t understand what credentials it uses.
Then I successfully created an index:
esClient.indices.create('test')
But when I try to index a document like this, I get the error :
esClient.index(index='test', doc_type='post', id=123456, body={
'author': 'John Doe',
'blog': 'Learning Elasticsearch',
'title': 'Using Python with Elasticsearch',
'tags': ['python', 'elasticsearch', 'tips'],
})
[WARNING] 2018-02-15T10:39:28.460Z 7f1cc69d-123c-11e8-be8b-cbbfad79068b PUT https://vpc-****-****.eu-west-3.es.amazonaws.com:443/test/post/123456 [status:406 request:0.039s ]
Document not indexed
Error: string indices must be integers
I don’t really see why it doesn’t work, wish there was an easy way to interact with Lambda’s other AWS services.
Can you help me?
Thanks
Solution
With the help of a colleague, I finally found the problem.
When you try to use an older version of the ElastiSearch Python library with a new version of the ElasticSearch server, an error is attached.
I > the library ( ) Upgrade to the latest version, which now works fine.
I’m sorry if I bothered you and hope this helps people like me.