Introduction

Idempotent APIs or Application Programming Interface means that a request can be resent, retried or reprocessed multiple times without additional side effects. So, if the exact same request (path, parameters, method, etc) is sent multiple times to the API service, the result will be the same for each request.

As an example, say we just paid for an online purchase but we lost power before we receive a confirmation. It is possible the payment went through successfully, even after we went offline. But we wanted to make sure we paid for the order so when power is back, we retry the payment.

Without idempotency, we could be charged twice for the same order.

With Idempotency in place, it will recognize our retry request as a duplicate and will not perform any processing other than to return the original response from the original request.

The goals of this post:

• To understand idempotency and its importance in API development
• Learn how to use AWS Powertools to help us build idempotent functions
• Walk through a short demo on how to use the idempotency utility from AWS Powertools in a project

Importance of Idempotency

EDA systems are asynchronous, distributed, scalable, loosely coupled.

As a tradeoff, it’s possible for some unintended side effects to occur:

• Retries are part of error handling best practice and can result in duplicate requests sent.
• Services like EventBridge and SQS guarantee at least once delivery. These services highly distributed, scalable and available, so it is very possible for duplicate deliveries to happen.
• A backend or third party service can become unavailable or a timeout could occur in the middle of processing a request. A user might resend the request again, thinking the initial one died or hung up, which can cause duplicates.

AWS Powertools for Lambda

AWS Powertools for Lambda is a suite of tools created to help developers build solutions that adhere to serverless best practices and well-architected framework. The utilities provided also simplify integrations with other AWS services like Secrets Manager and Systems Parameter Store. Powertools supports the following languages: Python (most extensive), Javascript/Typescript, .NET, Java.

How Idempotency Works in Lambda Powertools

In this example, we have a Serverless API that registers users by their User ID. Users can only register once.

The API Endpoint "/users/register/{userId}” is hosted in API Gateway. The userId value is a path parameter. The registerUser function is idempotent and is responsible for storing registered users in the RegisteredUsers DynamoDB table.

Scenario: A duplicate user registration request has been resent for User ID value: “User123”. The User ID value is used as the idempotency key.

Request Flow:

1. API Request sent to API Gateway
   1. Method: POST
   2. userId path parameter value: “User123”
2. The request is proxied by API Gateway to a Lambda Function. The idempotency utility is implemented within the Lambda Function code. We will take a closer look at this later. For now, just know that the Lambda Function is idempotent.
3. The idempotency utility kicks in. It checks if the Lambda Function has already executed a request that has “userId = User123” by checking the Idempotency Key in the Dynamo Table.
4. Because this is a duplicate request, it is found in the Idempotency table. The original response, which is stored in the same table is returned to the Lambda Function.
5. The Lambda Function does not run the registration process and simply returns the original response back.

Adding Idempotency to Lambda Functions

Check out the Powertools Idempotency Docs.

1. We will pick userId as our Idempotency Key which the Powertools utility will use as the Primary Key for the Idempotency Table. Make sure that this value does not change between requests we consider duplicates. In this scenario, the userId value will be the same across all requests. Refer to Choosing Idempotency Key for guidelines on choosing an appropriate idempotency key.
2. Install the Powertools and DynamoDB dependencies. DynamoDB is needed by the Idempotency utility for storing data.

npm i @aws-lambda-powertools/idempotency @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb

         3. Create a DynamoDB table that will be used by the Idempotency utility. By default, the Idempotency utility will use “id” as the attribute name of the primary key and “expiration” as the Time To Live (TTL) attribute. The default attribute names can be customized.

         4. Ensure that the Lambda Execution Role is granted permissions to the Idempotency and Registration DynamoDB tables. These permissions are required.

dynamodb:GetItem
dynamodb:PutItem
dynamodb:UpdateItem
dynamodb:DeleteItem

         5. Use the “makeIdempotent” wrapper to wrap a function that we need to be idempotent. In our example, we are making the function “registerUser” idempotent.

// import dependencies
import { makeIdempotent, IdempotencyConfig } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import { DynamoDBClient, PutItemCommand } from "@aws-sdk/client-dynamodb";

// Initialize the Idempotency persistence layer
const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: process.env.IDEMPOTENCY_TABLE_NAME,
});

// Setup the idempotency configuration
const idempotencyConfig = new IdempotencyConfig();

// DynamoDB client
const ddb = new DynamoDBClient({ region: "us-east-1" });

// Performs DynamoDB operations to store user registration
// AWS Docs: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/dynamodb/command/PutItemCommand/
export const addUserRegistration = 
    async (userId) => {
      const registrationDate = Date.now();

      console.log(`Saving registration for: ${userId}`);
      
      const putItemCommandInput = {
        TableName: process.env.REGISTRATION_TABLE_NAME,
        Item: {
          "userId": { S: userId },
          "registrationDate": { S: registrationDate.toString() }
        },
        ConditionExpression: "attribute_not_exists(userId)",
        ReturnValues: "NONE"
      };

      const putItemCommand = new PutItemCommand(putItemCommandInput);
      const putItemResponse = await ddb.send(putItemCommand);
      console.log(`Registration saved: ${JSON.stringify(putItemResponse)}`);

      return registrationDate;
    };

// registerUser function is called by the Lambda Handler which with userId as input.
export const registerUser = 
  makeIdempotent(
    async (userId) => {
      console.log(`Registering UserId: ${userId}`);

      const registrationDate = await addUserRegistration(userId);
      const response = {
        statusCode: 200,
        body: JSON.stringify({
          registered: true,
          registrationDate,
          userId
        })
      };

      return response;
    },
    {
      persistenceStore,
      idempotencyConfig
    }
  );

export const lambdaHandler = 
  async (event, context) => {
    // Register the Lambda context
    idempotencyConfig.registerLambdaContext(context);

		/*
			Call the idempotent function with the userId value as input argument.
			The value for userId is included in the API Gateway event payload as a Path Parameter.
		*/
    console.log("Processing event: ", JSON.stringify(event));
    const response = await registerUser(event.pathParameters.userId);
    console.log("Response: ", JSON.stringify(response));

    return response;
  };

         6. Let’s test the API. The POST method will need to include API key in the request header. Our first request has been processed successfully.

$ curl --request POST https://<api-id>.execute-api.us-east-1.amazonaws.com/dev/registerUser/User123 --header "x-api-key:api-key-value"

{"registered":true,"registrationDate":1708370508821,"userId":"User123"}

Powertools Idempotency in Action

Let’s check to see if we implemented idempotency correctly for our API.

         1. In the Idempotency table, we see the response returned by the POST request is stored in the data field.

         2. In the Registration table, we see the registered user ID and the registration date in Epoch. The registration date is the same value that is returned in the response.

         3. Let us look at the CloudWatch logs for the registerUser Lambda Function.

We can see the log messages printed out indicating that the submitted user registration is being saved to DynamoDB.

We actually see the response from DynamoDB after storing the record.

"timestamp","level","message"

"2024-02-19T19:21:47.785Z","INFO","Processing event API Gateway Event"

"2024-02-19T19:21:48.821Z","INFO","Registering UserId: User123"

"2024-02-19T19:21:48.821Z","INFO","Saving registration for: User123"

"2024-02-19T19:21:49.055Z","INFO","Registration saved: {""$metadata"":{""httpStatusCode"":200,""requestId"":""<requestId>"",""attempts"":1,""totalRetryDelay"":0}}" <---- RESPONSE FROM DYNAMO

         4. It has been about 20 mins since the first request. The default expiration for an idempotency record is 1 hour. Let us resend the request to register “User123” again. We should get the exact same result as the first request and we did.

$ curl --request POST https://<api-id>.execute-api.us-east-1.amazonaws.com/dev/registerUser/User123 --header "x-api-key:api-key-value"

{"registered":true,"registrationDate":1708370508821,"userId":"User123"}

         5. Let’s look at the CloudWatch Logs for the Lambda Function again. This time, we do not see the log messages for storing the registration and we simply see the stored response returned back.

"timestamp","level","message"

"2024-02-19T19:47:36.428Z","INFO","Processing event API Gateway Event"

"2024-02-19T19:47:37.529Z","INFO","Response:  {""body"":""{\""registered\"":true,\""registrationDate\"":1708370508821,\""userId\"":\""User123\""}"",""statusCode"":200}" <----ORIGINAL RESPONSE FROM DYNAMO

Additional Tips

1. Refer to this guide to get the default list of attributes used by the Idempotency DynamoDB table. These attribute names can be customized by configuring the DynamoDBPersistenceLayer object.
2. Powertools docs illustrates the request flows for different execution scenarios so we understand how the idempotency behavior changes based on the request.

The default behavior of how idempotency works can be customized using the IdempotencyConfig object. Refer to the docs for more detailed information. Some examples include making idempotency key required, changing the default expiration time for idempotency records in DynamoDB, caching configuration, etc.

         3. Remember to assign the lambda context to the IdempotencyConfig object in the lambda handler so that the idempotency utility can properly handle retries caused by function timeouts.

idempotencyConfig.registerLambdaContext(context);

When a function times out, the status of the idempotency record remains as InProgress to block concurrent processing of the same request. This means that retries will also be blocked. To address this, Powertools utilizes the lambda context to determine function timeout settings to allow retry requests that occur after the function timeout period has elapsed.

         4. The DynamoDB table used to store idempotency records includes the Lambda Function name in the Primary Key. This means the Idempotency table can be shared by several Lambda Functions.

         5. Check out the Integration with the Powertools Batch Utility. This enables processing each record or payload in a batch in an idempotent manner.

Conclusion

In this blog we learned what Idempotency is and why it is important when building event driven architectures. We learned about AWS Powertools for Lambda which is a developer toolkit that helps us adhere to best practices and the well-architected framework when working with Serverless. Finally, we went through a short demo to learn how to implement idempotency for our Serverless API and verified that it is working as expected.

👉 Serverless is an excellent choice for building Event Driven Architectures and Idempotency is a key design principle for building Well-Architected Serverless systems/applications.

Idempotency is that “thing” that’s nice to have in case we need it rather than need it and not have it 😊

‍