A key piece in foodpanda’s shift toward devops is finding ways to host and run code without spending too much time maintaining infrastructure and machines.

One of Amazon’s hottest features, AWS Lambda, is proving to be a big helper in this direction. Lambda functions are a core part of the AWS serverless architecture initiative where you simply do not manage any servers anymore. You simply code and from the moment you deploy, Amazon takes care of all the resource provisioning, scaling and execution.

I can’t give you an easy guide how to write good code, but I’ll share how we manage the second part of working with Lambda functions in foodpanda: deploying.

Current deployment scenario

foodpanda is a global company, we operate in more than 40 countries across most continents (and AWS regions for the matter), so multi-regional set ups are a requirement for each project we have.

Typically with an EC2 + autoscaling group setup, we build a new version of an application on Travis and then go region by region and deploy the new version of the application. That deployment will replace the previous version on the environment.

The main flow with this process goes like this:

  1. Code and tag release
  2. Build application on Travis CI and store artifact in repository
  3. Release by connecting to region and running deploy scripts
  4. Repeat Step 3 for each region

Lambda deployment

With Lambda, since we only pay for the time and memory used for execution, we choose a different flow:

  1. Code and tag release
  2. Build application on Travis CI and store artifact in repository
  3. Immediately after Step 2, deploy that artifact in every region as a new function

There are two main differences. First, the deployment is no longer manual work but done by machines. Second, we automatically have all regions consistent and the we have same version available everywhere.

Using Lambda to deploy Lambda

The whole solution uses only AWS resources and even better, it all fits into one CloudFormation template.

Here is the expected result:

Auomatic Lambda deployment schema

  1. Travis uploads the built Lambda function to S3
  2. S3 triggers an event on a SNS Topic for new artifacts
  3. A Lambda function called CreateLambdaFunctionOnNewArtifact is subscribed to messages on this topic
  4. The Lambda function:
    1. Checks the artifact
    2. Iterates over all known regions
    3. Creates a new version of the Lambda function with the version in the name in each one of them

Since Lambda functions by default have access to the AWS SDK, the iterating over AWS regions, creating the functions programmatically and using the location of the artifact in S3 works seamlessly together.

Let’s go through the basic CloudFormation resources we need to wire this together. Starting with the Artifact bucket where built functions are uploaded. As you see, we configure the notifications at this point.

"ArtifactsBucket" : {
 "Type" : "AWS::S3::Bucket",
 "Properties" : {
   "NotificationConfiguration" : {
     "TopicConfigurations" : [ {
       "Event" : "s3:ObjectCreated:*",
       "Topic" : { "Ref": "NewArtifactTopic" }
     } ]
   }
 }
}

The SNS topic we referenced in the previous snippet adds the subscription of the Lambda deployment function. We could also directly link the S3 notifications to trigger Lambda, but we have multiple apps being uploaded to one artifacts bucket. For each of those apps we have a separate deployment function. Branching out the notification to multiple Lambdas is super simple thanks to SNS.

"NewArtifactTopic": {
  "Type" : "AWS::SNS::Topic",
  "Properties" : {
    "DisplayName" : "New artifact uploaded to bucket",
    "Subscription" : [
      {
        "Endpoint": { "Fn::GetAtt" : [ "CreateLambdaFunctionOnNewArtifact", "Arn" ] },
        "Protocol": "lambda"
      },
      ... you have the possibility of trigger more functions than just one... 
    ],
    "TopicName" : "NewArtifactTopic"
  }
},

Finally, this is the Lambda which I referenced in the subscription endpoint above.

"CreateLambdaFunctionOnNewArtifact" : {
 "Type" : "AWS::Lambda::Function",
 "Properties" : {
   "Code" : {
     "S3Bucket" : "cloudformation-templates-bucket.s3.amazonaws.com",
     "S3Key" : { "Fn::Join" : [ "/", [
       "%TEMPLATE-VERSION%",
       "createLambda.zip"
     ] ] }
   },
   "Description" : "Creates a new Lambda function when an artifact is uploaded to S3",
   "Handler" : "createLambda.handler",
   "MemorySize" : "128",
   "Role" : { "Fn::GetAtt" : [ "LambdaLambdaCreator", "Arn" ] },
   "Runtime" : "nodejs",
   "Timeout" : "60"
 }
}

As you can see in the Role field of the Lambda function resource definition, I omitted one important part: IAM roles and policies to make this all click together. I will not go into details about each one, but here is a list of Resources we needed for proper authorization:

  • NewArtifactTopicPolicy to give S3 the sns:Publish permission to publish on SNS
  • InvokeLambdaCreationLambdaBySnsPermission to give SNS the lambda:InvokeFunctionpermission to execute the deployment function (the policy limits the invoke permission only to the one function
  • LambdaLambdaCreator is an IAM role which is assigned to the deployment function. It allows it to save logs, fetch objects from S3, iam:PassRole to give a role to the function it creates and,ec2:DescribeRegions to iterate over all regions and finally lambda:CreateFunction to actually create functions.
  • We also create a dedicated IAM Travis user with a role and s3:PutObject permissions to the artifacts bucket.

The deployment Lambda function itself is a node.js script which uses the AWS SDK for Javascript to call all the necessary APIs. If all of this is put into place, we just configure our Travis build with the AWS access key and secret which allows it to upload the build artifact to S3 and we can start watch the machine do it’s job.

You can see the results below; there are two automatically deployed versioned functions (starting with logistics-matching) and the function which created them (global-CreateMatchingLambdaFunction...).

Example of deployed functions and the functions that deployed them

There is also one other function in the picture. As I mentioned earlier in the article, we use the SNS topic to notify multiple deployment functions from one artifact bucket. In this case, we also create Elastic Beanstalk application versions with exactly the same setup, to be later prepared for manual deploying in two clicks. Extending it to another app or deployment set up is just a matter of adding a new subscription to the SNS topic.

Posted by on 8 Feb 2016

Lead dev @ foodpanda, currently creating a new generation last-mile delivery platform.

Leave a Reply