Introduction

In this article, we will take a look at a use-case that represents Serverless Orchestration workflow with AWS Step Functions using Serverless Framework.

The entire PoC (proof-of-concept) uses IaC written with Serverless Framework. You can download the template from our website, HERE.

AWS Architecture

Above is an AWS Serverless architecture diagram showing Step Functions orchestrating different Lambda functions through multiple task steps. Later on in the article, we will talk in more detail about how this architecture’s workflow steps work which includes a Serverless Framework IaC template.

What is AWS Step Function?

"AWS Step Functions is a serverless orchestration service that lets you combine AWS Lambda functions and other AWS services to build business-critical applications. Through Step Functions' graphical console, you see your application’s workflow as a series of event-driven steps." - Source

• AWS Step Functions lets you coordinate individual tasks into a visual workflow, so you can build and update apps quickly.
• The workflows you build with Step Functions are called state machines, and each step of your workflow is called a state.
• Tasks perform work, either by coordinating another AWS service or an application that you can host basically anywhere.

How do Step Functions work?

Define your visual workflow in Workflow Studio and the console will auto-generate the code for you. You can also code it yourself using code snippets in the console or locally with VS Code.

Specify the resources, such as Lambda functions and SNS topics, that you want to trigger in your workflow.

Start an execution to visualize and verify that the steps of your workflow are operating as intended. Inspect and debug your execution history from the console directly.

Step Function States

• Task - A single unit of work using different AWS services
• Choice - adds branching logic to a state machine
• Parallel - Fork and join data across tasks
• Wait - delays the state machine from continuing for a specified time
• Fail - stops the execution of the state machine and marks it as a failure
• Succeed - Stops an execution successfully
• Pass - Passes its input to its output
• Map - used to run a set of steps for each element of an input array

Step Function Workflow Types

Standard

Durable, checkpointed workflows for machine learning, order fulfillment, IT/DevOps automation, ETL jobs, and other long-duration workloads.

Express

Event-driven workflows for streaming data processing, microservices orchestration, IoT data ingestion, mobile backends, and other short duration, high-event-rate workloads.

Why use Step Functions?

• Different way to trigger workflow steps
• Workflow progress visualization
• Built-in error handling and retry
• Handles complex workflows with lots of steps
• Allows manual steps
• Handles long-lived workflows up to 1 year
• Manage the state between executions of stateless functions
• Decouples the multiple microservices steps
• Scales & perform parallel executions
• Execution event history
• Function orchestration
• Branching / Parallel processing
• Many AWS Services Support

Provisioning AWS Steps Function Workflow Using Serverless Framework

Serverless Framework plugin serverless-step-functions will help you define a step function state machine definition directly within the serverless.yml.

Step Functions visual workflow generated JSON is in the form of Amazon State Languages and that same spec is able to be used with the serverless-step-functions plugin after translating to YAML.

All JSON is valid YAML. We are able to translate JSON snippets to YAML using an online tool Code Beatify and CLI tool yq.

Below mentioned Serverless Framework IaC code snippet will create these resources:

• 1 State Machine with Standard type Workflow with different states e.g Choice, Task, etc.
• 4 Lambda Functions (add, subtract, multiply, validateResult)

service: sls-step-functions
frameworkVersion: '2'

plugins:
  - serverless-step-functions

provider:
  name: aws
  runtime: nodejs12.x
  lambdaHashingVersion: 20201221
  stage: dev
  region: ap-south-1

functions:
  add:
    handler: functions/add.handler
    timeout: 3

  subtract:
    handler: functions/subtract.handler
    timeout: 3

  multiply:
      handler: functions/multiply.handler
      timeout: 3

  validateResult:
    handler: functions/validateResult.handler
    timeout: 3

stepFunctions:
  stateMachines:
    LambdaStepFuncDemo:    
      name: StepFunctionLambdaOrchestrationPOCDemo
      definition:
        Comment: "Step Function Lambda Orchestration POC Demo"
        StartAt: choose
        States:
          choose:
            Type: Choice            
            Choices:
              - And:
                  - Variable: $.operation
                    StringEquals: "add"
                  - Variable: $.number1
                    NumericGreaterThanEquals: 1
                  - Variable: $.number2
                    NumericLessThan: 100
                Next: add
              - And:
                  - Variable: $.operation
                    StringEquals: "subtract"
                  - Variable: $.number1
                    NumericGreaterThanEquals: 1
                  - Variable: $.number2
                    NumericLessThan: 100
                Next: subtract              
              - And:
                  - Variable: $.operation
                    StringEquals: "multiply"
                  - Variable: $.number1
                    NumericGreaterThanEquals: 1
                  - Variable: $.number2
                    NumericLessThan: 100
                Next: multiply      
          add:
            Type: Task
            Resource: 
              Fn::GetAtt: [add, Arn]
            Next: validateResult
            ResultPath: $.resultNumber
          subtract:
            Type: Task
            Resource: 
              Fn::GetAtt: [subtract, Arn]
            Next: validateResult
            ResultPath: $.resultNumber
          multiply:
            Type: Task
            Resource: 
              Fn::GetAtt: [multiply, Arn]
            Next: validateResult
            ResultPath: $.resultNumber
          validateResult:
            Type: Task
            Resource: 
              Fn::GetAtt: [validateResult, Arn]              
            InputPath: $.resultNumber
            TimeoutSeconds: 1
            Retry: 
              - ErrorEquals:
                  - States.Timeout
                IntervalSeconds: 1
                BackoffRate: 2.0
                MaxAttempts: 2
            Catch: 
              - ErrorEquals:
                  - NumberIsBig
                ResultPath: $.error
                Next: numberIsBig
            End: true
          numberIsBig:
            Type: Fail

This POC Demo Step Function Standard Workflow Works Like This

Step Function workflow starts with choice state. Based on choice states and operation input, different steps of lambda tasks (e.g add/ subtract/multiply) will trigger. Add/subtract/multiply tasks lambda steps will then call validateResult Lambda task.

The validateResult lambda task checks the following:

• From add/subtract/multiply steps if resultNumber > 100 then raise exception NumberIsBig which will be caught by the NumberIsBig fail step
• Incase resultNumber is not valid then return InvalidNumberError exception
• resultNumber < 0 will raise exception with timeout
• For other success scenarios it will return success resultNumber

Steps Functions State Machine Execution Cases

Below we will look at a couple of different scenarios that can take place using an input JSON value, result, and the visual AWS Step Functions view from the console.

Case: add, succeeded

Input JSON Value

{
  "number1": 5,
  "number2": 2,
  "operation": "add"
}

Result Success

["output": 7]

Case: add, failed

Input JSON Value

{
  "number1": 99,
  "number2": 2,
  "operation": "add"
}

Result Failed

[output resultNumber > 100]

Case: subtract, succeeded

Input JSON Value

{
  "number1": 5,
  "number2": 2,
  "operation": "subtract"
}

Result Success

["output": 3]

Case: multiply, succeeded

Input JSON Value

{
  "number1": 5,
  "number2": 2,
  "operation": "multiply"
}

Result Success

["output": 10]

Case: multiply, failed

Input JSON Value

{
  "number1": 15,
  "number2": 7,
  "operation": "multiply"
}

Result Failed

[output resultNumber > 100]

Steps Function State Machines and Lambda Console Screenshots

Step Functions State Machines

Below you can see the AWS Step Function console including the state machine we made which has all the steps mentioned earlier in the article.

Steps Function Orchestration Tasks Relevant Lambda Functions

Below you can see the AWS Lambda console showing the list of Lambda used as Task states within Step Function State Machine.

State Machine Task Lambda: Validate Result

Below you can see the code of Validate Result Lambda Function.

Conclusion

AWS Step Functions is a low-code, visual workflow service that developers use to build distributed applications, Serverless Orchestration workflows, automate IT business processes, and build data or machine learning pipelines using different AWS services.

Workflows manage failures, retries, parallelization, service integrations, and observability so developers can focus on higher-value business logic.

Serverless Framework is easily able to provision a Step Functions State Machine workflow using the Serverless Step Functions plugin through infrastructure as code.

‍