End-to-End Image Resizing Pipeline with AWS S3, Lambda, and SNS: A Step-by-Step Guide

AWS Infra Diagram

Introduction:

In today’s digital age, images play a crucial role in various applications and websites. However, managing and serving images of different sizes can be a challenging task. To overcome this challenge, we can leverage the power of cloud computing and automation to resize images on-the-fly. In this blog post, we will explore how to automate image resizing using AWS S3, Lambda, and SNS, enabling efficient image management and delivery.

Prerequisites:

Before diving into the implementation, let’s ensure we have the following prerequisites in place:

AWS Account: To follow along with the steps outlined in this blog post, you will need an AWS account. If you don’t have one, you can sign up for free on the AWS website.

Basic Knowledge of AWS Services: Familiarity with AWS S3 (Simple Storage Service), Lambda (Serverless Compute), and SNS (Simple Notification Service) will be helpful. If you are new to these services, don’t worry — we will cover the necessary concepts and steps in detail.

Implementation Options:

In this blog post, we will cover only implementation options: manual setup through the AWS Management Console and the other approach is infrastructure-as-code using Terraform.

1. Manual Setup: We will guide you through the step-by-step configuration process using the AWS Management Console. This approach is suitable for users who prefer a hands-on approach and want to understand the individual components and their configurations.

2. Terraform: For users who prefer infrastructure-as-code and want to automate the provisioning of resources, we will provide instructions for implementing the solution using Terraform. This approach allows for repeatable and version-controlled deployments of the Image Resize Automation.

To follow along with the manual setup implementation, please refer to the detailed instructions provided in this blog post. If you prefer the Terraform approach, you can access the complete Terraform code and instructions on our GitHub repository: https://github.com/AmanPathak-DevOps/Terraform-for-AWS/blob/master/Non-Modularized/Resize-Image-Using-LambdaFunction-S3-SNS/. The repository includes the necessary Terraform scripts and configuration files to set up the Image Resize Automation.

By following the upcoming sections of this blog, you’ll gain insights into both manual and Terraform-based implementations of the automated AWS Image Resize Automation and email notification system. Whether you prefer a hands-on or automated approach, this solution will empower you to effortlessly stay resized your images and optimize your storage.

Now that we have our prerequisites in order, let’s dive into the implementation details and automate the image resizing process step by step. By the end of this blog post, you will have a fully functional system that can automatically resize images and store them in a separate bucket, while notifying a subscribed email address about the successful resizing process.

Let’s get started!

In this project, we will be creating two buckets. The first bucket will be used to store the original images that we want to resize. The second bucket will be dedicated to storing the resized images. The resizing process will be automated through an AWS Lambda function, which will automatically upload the resized images to the second bucket.

Creating the first S3 bucket by clicking on Create bucket.

2. Enter the name of Bucket 1 and click on Create bucket.

3. Create the second S3 bucket by clicking on Create bucket.

4. Enter the name of Bucket2 and click on Create bucket.

5. So, Here we have created two Buckets.

You can check the below screenshot.

6. Now, let’s create an SNS topic and an SNS subscription before creating a lambda function.

Click on Create topic.

7. Select Standard type SNS topic and enter the suitable name for your SNS topic and click on Create topic.

8. As you can see in the below screenshot, we have configured the SNS topic. Now, click on Create subscription to add your email address.

9. Select the Protocol type as email and add your mail address in Endpoint and click on Create Subscription.

10. After creating Subscription, you will get a notification on your given mail address if not, please check your spam box. The mail will look like the below screenshot.

Click on Confirm subscription to get the notification for the resized image.

11. Once you confirm your subscription. The status should be Confirmed from Pending status.

Here, we configured the SNS topic and subscription.

12. Now, Let’s create a Lambda function that will help us to resize the images and store the images.

Click on Create function.

13. Enter the function name, runtime(I am using Python 3.9), and architecture, and for role select the first option, So we can configure from our own all the access for the services and click on Create function.

14. First of all, replace the code with our code that will resize the image

Code link- Lambda-Code, After replacing the code, deploy it by clicking on Deploy.

15. Now, we need to configure lots of things in the lambda function. Let’s start with runtime settings, that is, exactly under the code itself.

Click on Edit

16. Now, add the handler name according to your Python code name and click on Save.

My lambda function code name is CreateThumbnail and handler name is handler. So, I have written CreateThumbnail.handler.

17. In our Python code, we have to provide Bucket2- resized-bucket name and SNS topic Arn. But instead of hardcoding it, we will use Lambda environment variables and we will pass our information.

To do that, Go to the configuration section, under click on Environment variables, and click on edit.

18. Add the SNS topic arn and S3 bucket name of Bucket2 according to your service name and on and click on Save.

19. In the lambda function, we have to increase the timeout and Memory size because the lambda function needs more time and memory to perform the resizing task.

To do that, Go to the Configuration tab-> General configuration section -> click on edit. Add the timeout 1 min and give memory 256MB. It would be enough to perform our task.

20. Now, we need to add one PIL layer for our Python code.

Click on Add a layer.

21. Add the ARN of the layer arn:aws:lambda:us-east-1:770693421928:layer:Klayers-p39-pillow:1 and click on Save.

22. Let’s do the IAM role configuration for our Lambda function. Go to IAM -> roles -> ‘search for your lambda name role’ and click on the role.

23. Here we have only one permission that is related to see the logs of our lambda function in loggroup. But we need two more permissions which is SNSFullAccess to send the mail to the person and S3FullAccess to get the images and store the images in the respective buckets.

To do that, Click on Add permissions and select Attach policy.

24. Now, find the two permissions one by one SNSFullAccess and S3FullAccess, and check the box on the left of the permissions. After doing both, click on Add permissions.

The final IAM role has the below permissions list.

25. Finally, We have to add a trigger for the Lambda function. So whenever someone uploads an image to Bucket1. The lambda function will trigger and start the process of resizing the image and doing the other respective things.

26. Add the S3 as source, Bucket1 which will trigger the Lambda function, and click on Add.

27. Now, we have configured all the things related to the Lambda function. It’s time to test it.

To do that click on Test -> Configure test event.

28. Add the Event-name, the template should be s3 put replace the few things that are given in the below screenshot, and click on Save.

29. Now, don’t run the lambda manually. Just upload an image to Bucket1 and see the magic below.

Outputs:

Uploading an Image to the Bucket1.

The Image is of 1.9MB in the Bucket 1.

The Image has been resized automatically from 1.9MB to 355.3KB.

Hurray, We have received the mail as well.

In conclusion, we have successfully implemented an automated image resizing system using AWS S3, Lambda, and SNS. This solution allows us to store the original images in one bucket and automatically resize and store them in another bucket. The integration of Lambda and SNS provides real-time notifications about the resizing process. By leveraging these powerful AWS services, we have simplified the image management workflow, improved efficiency, and enhanced user experience. Stay tuned for more exciting projects and follow us on social media for further updates and engaging content. Join us in the cloud revolution and unlock the potential of automation for your applications.

To stay updated with more exciting tutorials, AWS best practices, and cloud-related content, make sure to follow our blog and subscribe to our newsletter. By staying connected, you will unlock a wealth of knowledge and discover new ways to leverage cloud technologies for your projects.

Remember, the power of automation and cloud computing is at your fingertips. Embrace it, innovate, and unlock the full potential of your applications and services.

Feel free to reach out to me if you have any doubts or queries.

Stay tuned for more exciting content and happy cloud computing!

LinkedIn- https://www.linkedin.com/in/aman-devops/

GitHub- https://github.com/AmanPathak-DevOps/

Don’t miss out on future updates and exclusive content. Subscribe to our newsletter now and be part of the cloud revolution.

Together, let’s transform the way we build and deploy applications in the cloud!

Happy Learning!

Aman Pathak