Deploying ruby gems with native extensions on AWS Lambda using the serverless toolkit
An earlier article by Vinoth Kumar Natarajan explains how to use ruby gems with native extensions (like nokogiri) on aws lambda using AWS Serverless Application Model. In this article, I will brief how to do the same using the serverless framework.
- Install the serverless framework
$npm install -g serverless
2. Create a new ruby service
Let’s call our service “scrapeservice” for we are going to use “nokogiri” gem which is used to parse html/xml documents. Nokogiri is a ruby gem with a native extension and it is not straightforward to use such gems like we will use normal ruby gems.
$serverless create -t aws-ruby -p scrapeservice
3. The service you just created is already working!
Let’s change into the service directory, deploy it and invoke a function(auto-generated with the serverless create command)
But before that we need to have set aws credentials
$cd scrapeservicescrapeservice$export AWS_ACCESS_KEY_ID=<your-key-here>scrapeservice$export AWS_SECRET_ACCESS_KEY=<your-secret-key-here>scrapeservice$sls deploy
$sls invoke -f hello
Now explore the contents of the service. There is one yml file and one handler.rb
4. Let’s add our code
In handler.rb, replace the hello function with scrape function. Also add relevant require statements
create a Gemfile to use nokogiri gem
5. Building gem dependencies
Like it is explained here, building gem dependencies can be quite tricky. Lambda layers can be used for this.
With the following command, we can install and build them using a docker (lambci) which closely replicates the live AWS Lambda
docker run --rm -it -v $PWD:/var/gem_build -w /var/gem_build lambci/lambda:build-ruby2.5 bundle install --path=.
After this, you can notice that the gems are installed by the docker and are placed in our local project folder
Notice the new ruby folder that has been created.
6. Adding the gem as lambda layers
Modify the functions part in the serverless.yml file.
i) to create a layer with the built gems
path — the name of the folder that was created after we ran the docker to build the gems
gemlayer— name of the layer that you are going to create
ii) change handler.hello to handler.scrape (The function you wrote in handler.rb)
iii) have your function use the layer that you created (gemlayer)
how to specify the layer to be used by the function?
“<your-layer-name-with-capitalized-first-letter>LambdaLayer”
A mistake in this naming convention can waste a lot of time. Beware!
iv) set GEM_PATH environment variable for the lambda function. The gems usually sit inside “opt” folder
The functions and layers portion of your serverless.yml now looks like below
7. Deploy and verify if lambda function and layer is created correctly
$ sls deploy
You can see the layer in aws user interface to verify
Also, see a lambda function created and associated with this layer
and the environment variable you had set
You can now execute the function through the UI and see a success message
or invoke through the command line
$ sls invoke -f scrape
There you go.
P.S In the 7th step, I had an issue
Error: The security token included in the request is invalid.
I then had to reconfigure aws credentials using aws configuration command
Francium Tech is a technology company laser focussed on delivering top quality software of scale at extreme speeds. Numbers and Size of the data don’t scare us. If you have any requirements or want a free health check of your systems or architecture, feel free to shoot an email to contact@francium.tech, we will get in touch with you!