Message Queues in AWS SQS

Top companies like amazon , google have successfully shift their architecture from the monolithic to microservice. A monolithic architecture is highly coupled means there is high dependency among the components. With high dependency there are some challenges like maintenance, technology enhancement and security. If you want to adopt a technology for the some features of application not for the whole application then it would be difficult or impossible for you.

For example you have made a application totally in java. It is working fine. Now you want to develop the chat feature of the application in python then it will affect the entire application and i think probably you don’t want this.

E-commerce website like amazon and social platform like facebook there are not using one single language instead there are using different languages for different functionality of application. Using different languages for different functionality has been possible for these companies with the microservices.

Monolithic as single code base serves the customer while in microservices a application is broken in different services to serve the customers.

Let’s follow an example: I want to build a video ranking application that allows users to upload videos, and have them ranked by other users.

Microservice Architecture — divide and conquer

Imagine that I choose a Microservice Architecture. The application is divided into two or more small services, where each microservice is developed separately.

The finished application is the sum of its microservices. One service let’s say A is for uploading videos on S3, another service B changing the format of video for uploading on our video ranking app ( same as Instagram changes the format of user uploaded image) and service C to upload for ranking of the video. In order for this to work microservices need a way to communicate with each other. To make sure that the data is delivered in a reliable fashion between the services.

And to make sure to not lose data in case of system failure, we need to centralize the management of the data. The solution is to send the data as a message and to use a Message Queue as a broker. Messaging or Message Queueing is a service-to-service communication allowing one service to communicate with another service by sending messages to each other.

The service A upload video on a S3 bucket and produces the message of the address of s3 video object in the message queue. Service B downloads the message form the queue, retrieves the video and formatted it like MPEG-2 which YouTube uses and store it in some other S3 bucket. After that service B produces the message of the address of s3 formatted video object in the message queue. Service C downloads the message, retrieves the formatted video and upload in our video ranking app or platform for ranking by other users.

The service which produces the message in the Message Queue is called producer and the service which downloads or consumes the video is called consumer.

Services not only downloads message actually they poll the message queue. Poll involves continuous checking of messages in the message queue for the duration poll configured, download messages and delete messages if configured for delete.

The time for which service poll downloads the message, for that time message becomes invisible in the queue. This time is called visibility timeout.

We have generally two types of queue:

• Normal Queue: In this the order for sending message is not fixed.
• FIFO: In this the message which comes first that goes out first.

Now if a message in the queue is failed to process then what happens with it.

Actually the messages those failed to process resides in a special queue call dead-letter queue. The main task of a dead-letter queue is handling message failure. A dead-letter queue lets you set aside and isolate messages that can’t be processed correctly to determine why their processing didn’t succeed.

AWS provides Message Queue as service through the AWS SQS. Most of companies are using AWS SQS to decoupled the microservice architecture of their application.

From AWS official documentation why oyster.com is using AWS:

“We were already using Amazon S3 to store the images, so using Amazon Elastic Compute Cloud (Amazon EC2) to process the images was a natural choice,” Seidman says. Chris McBride, a software engineer at Oyster, adds, “We wanted a cloud environment that could be ramped up for the large processing jobs and downsized for the smaller daily jobs.”

While the company is still running one local server, the bulk of the processing work now takes place on the AWS Cloud. Oyster is using a customized Amazon Linux AMI within Amazon EC2. Within this new environment, the company connects to Amazon S3 and Amazon Simple Queue Service (Amazon SQS) using boto, a Python interface to AWS. The images themselves are processed with the ImageMagick software available in the AMI package.

Oyster uses Amazon EC2 instances and Amazon SQS in an integrated workflow to generate the sizes they need for each photo. The team processes a few thousand photos each night, using Amazon EC2 Spot Instances. When Oyster processes the entire collection, it can use up to 100 Amazon EC2 instances. The team uses Amazon SQS to communicate the photos that need to be processed and the status of the jobs.

NASA is also using AWS SQS and AWS SNS for decoupling in their architecture. The use of Message Queue component is unavoidable in the modern architecture of the application. I try to give overview of how actually Message Queues work in the applications.

Hope you enjoyed this story. We learned from top companies prefer architecture to Message queues terminology in AWS context.

Bye!