Lambda Event Filtering with a DynamoDB Stream

January 14, 2022

Introduction

In this article, we will take a look at a use-case that allows us to pre-filter events by certain conditions before they invoke our AWS Lambda functions. We will be leveraging a DynamoDB stream with multiple AWS Lambda functions attached and multiple Lambda Event Source Mappings. The entire PoC (proof-of-concept) uses IaC written with Serverless Framework.

AWS Architecture


Folder Structure

Above is an overview of what a local PoC folder structure will look like. Further down in this document you will see what each file looks like.

DynamoDB Table

Above we are creating our DynamoDB table with stream enabled.

Two Lambdas + IAM Role for DynamoDB Stream

Above we are creating our two Lambda functions and attaching an IAM policy with permissions to log to CloudWatch and interact with our DynamoDB stream. The index.js is a one line log.

AWS Lambda Event Source Mappings

Above we are attaching two separate Lambda event filters.

One for Lambda A which is looking for the following:

  • eventName of "INSERT"
  • PK prefix of “Message#”
  • SK prefix of “Channel#”
  • messageId exists

If “messageId” doesn’t exist, “PK” doesn’t start with “Message#”, SK doesn’t start with “Channel#”, and eventName is not “INSERT” while in the new DynamoDB record, it will not trigger Lambda A.

This event would trigger Lambda A as it meets our filter requirements

The second Lambda event filter which would trigger Lambda B is looking for the following:

  • eventName of “INSERT”
  • PK prefix of “Feed#”

If “PK” doesn’t start with “Feed#” and eventName is not “INSERT” in the new DynamoDB record, it will not trigger Lambda B.

This event would trigger Lambda B as it meets our filter requirements

If one of our conditions is not met then our Lambda functions with filters will not be invoked. The trickiest part of this setup is getting the Filter Pattern correct. For this, I leveraged the AWS documentation and trial + error.

Conclusion

By leveraging Lambda Event Mapping Filters on a DynamoDB Stream we are able to only trigger the necessary Lambda when our conditions are met. Which is a huge improvement over triggering every time DynamoDB has an event take place then filtering inside the Lambda itself.

By doing this filtering outside the Lambda we are saving money (as we are not billed for the Lambda Event Mapping Filter), saving engineering hours to write the filtering code and test, reducing overall noise from millions of non-relevant Lambda invocations happening, and ensuring downstream maintenance is minimal.

If you want to download a copy of the template, you can find it on our website.

Serverless Handbook
Access free book

The dream team

At Serverless Guru, we're a collective of proactive solution finders. We prioritize genuineness, forward-thinking vision, and above all, we commit to diligently serving our members each and every day.

See open positions

Looking for skilled architects & developers?

Join businesses around the globe that trust our services. Let's start your serverless journey. Get in touch today!
Ryan Jones - Founder
Ryan Jones
Founder
Speak to a Guru
arrow
Edu Marcos - CTO
Edu Marcos
Chief Technology Officer
Speak to a Guru
arrow
Mason Toberny
Mason Toberny
Head of Enterprise Accounts
Speak to a Guru
arrow

Join the Community

Gather, share, and learn about AWS and serverless with enthusiasts worldwide in our open and free community.