Leveraging AWS Elasticsearch

Recently I had an use case where I had to expose AWS DynamoDB search as an API.  I could have gone with the common approach of using AWS API Gateway integrating with AWS Lambda which runs the search against the AWS DynamoDB as  




however this approach wouldn't be scalable under load. 

Alternative to the above solution would be to use AWS Elasticsearch Service. ElasticSearch is an open source product from Elastic that’s is designed to help us search in a way that is highly available and abstracted from datastore. AWS Elasticsearch Service is AWS hosted ElasticSearch that takes care of set-up and management of the back end server and provides us with an endpoint that we can get developing with. AWS Elasticsearch Service makes it easy to deploy, secure, operate, and scale Elasticsearch for log analytics, full text search, application monitoring, and more. AWS Elasticsearch Service is a fully managed service that delivers Elasticsearch’s easy-to-use APIs and real-time analytics capabilities alongside the availability, scalability, and security that production workloads require. The service offers built-in integrations with Kibana, Logstash, and AWS services including Amazon Virtual Private Cloud (VPC), AWS Key Management Service (KMS), Amazon Kinesis Data Firehose, AWS Lambda, Amazon Cognito and Amazon CloudWatch so that you can go from raw data to actionable insights quickly and securely.


 Above solution can be re-architected as 





APIGW2ES Lambda function is to pass the search request from API Gateway to Elasticsearch. Example to get info from Elasticsearch via AWS Lambda can be referred here 

 DDB2ES Lambda function is to poll the DynamoDB for any addition of record and send to Elasticsearch for indexing. Example for pushing records from DynamoDB to ES can be referred here  

Comments

Post a Comment

Popular posts from this blog

Validate Cognito tokens in Kong

Assuming Kong environment is set up and operating as expected, this blog helps to Validate Cognito tokens in Kong.  JWT token issued by popular  identity  solutions such as Auth0 , Amazon Cognito etc., can be easily Authorized by kong.  Here are the steps to validate JWT token issued by Auth0 in Kong.  In this blog, I am going to focus on how to validate JWT token issued by Amazon Cognito.  Assuming that Amazon Cognito user pools are set up and operating as expected.  Every  successful authentication of a user, Amazon Cognito issues  ID Token Access Token ID token is represented as a JSON Web Key Token (JWT).  The token contains claims about the identity of the authenticated user. For example, it includes claims such as  name ,  family_name ,  phone_number , etc., For more information about standard claims, see the  OpenID Connect specification . Extract the public key from Amazon Cognito public endpoi...
Read more

Tracing Requests in AWS Serverless Applications

Image
Operating Serverless APIs using Amazon API Gateway, AWS Lambda and Microservices (running in ECS) can be bit challenging if logging is not given any forethought. Best practice is to decide in while designing the application what needs to be logged, how the log messages can be correlated and finally where is it should be sent for aggregation.  Its difficult to reconcile log events for a serverless API sent across multiple layers of application stack using CloudWatch log groups and log streams. Tracking down logs for a specific request or tailing request logs for a specific request can sometimes be a cumbersome experience.  Debugging for a specific issue or Tracing a specific request can be made easy by using the x-amzn-requestid and x-amzn-trace-id that comes for free when using AWS API Gateway. Its easy to leverage these Ids and keep track of your request.   In the following deployment diagram  How to capture the request ID for a particular API req...
Read more