AWS Step Functions with the callback pattern - Ruby with Serverless Framework example

Introduction

In one of our previous blogposts we explored the topic of AWS Step Functions service namely Workflow Studio. Today we're going to look at a life-saving feature of this service, specifically the Callback pattern.

As we can see in the AWS blog post callback "automate workflows for applications with human activities and custom integrations with third-party services".

For the purpose of this article I have created a repository which you can find here. It is an application written in Ruby with Serverless Framework which demonstrates the use of callback pattern inside AWS Step Functions.

The application is very simple. It is a simple state machine with one AWS Lambda. It randomly returns success or failure.

General content

Callbacks help build asynchronous workflows that stop AWS Step functions from moving to the next state until the tasks we specify are completed.

A task might need to wait for a human approval, integrate with a third party, or call legacy systems. For tasks like these, you can pause Step Functions indefinitely, and wait for an external process or workflow to complete. More information here.

Step Functions on its side generates a taskToken and then stops execution until it gets back a SendTaskSuccess or SendTaskFailure call, and then resume the execution. The token itself is of string type, approximately 620 characters. Remember, that it needs to be a Task state, as well as there are no additional costs.

Take into note that not every AWS service supports the callback pattern. In addition, there is no support for the AWS Step Functions Express type. The whole list could be found here.

As Step Functions takes care of token generation for us, on our side we just need to handle returning the success or failure. For this, we just need to use two methods from AWS SDK States for Ruby:

• send_task_success for inform that the task identified by the taskToken completed successfully.
• send_task_failure to inform that the task identified by the taskToken has failed.
• additionally we can use send_task_heartbeat in order to check whether the task is still making progress.

We necessarily require task_token for these methods, as well as for the method with success, we need to pass the parsed JSON for output.

The code should look more or less like this:

As we are using the Serverless framework, the Step Functions plugin was used (but you can also use AWS Step Functions with workflow studio), and from the SLS position create only the resources you need, such as Lambda.

It is worth remembering that you cannot directly pass the ARN of Lambda that you want it to be as a callback pattern. We have to do it a bit differently.

We need to specify our Lambda under the Parameters as FunctionName, and pass into it the Payload containing the Step Function $$.Task.Token.

Be sure to set the IAM roles needed for proper operation:

If you want to see how it works in real life, clone the repository and follow the instructions. After the deployment process, go to the Step Functions dashboard and start new execution of the newly created State Machine.

The current implementation randomly selects whether to have success or failure, so it is best to run the execution several times.

Summary

We learned on a simple example how to build in Ruby and Serverless Framework a simple state machine that uses callback pattern.