SQS, SNS, Kinesis, Active MQ

Decoupling mechanism

When we deploy multiple applications they will inevitably need to communicate. There is two ways of doing the application.

1. Synchronous communication: Direct connection for direct communication between application
2. Asynchronous / event based communication: Application is connected to a queue, then the message will come asynchronous and the application will react to that message to do whatever it needs to do

Synchronous can be a problem if there is sudden spike of traffic. It is better to decouple your application so that your application can scale independently, without worrying about direct communication

Amazon SQS (Simple queue service)

It is a queue, it will have messages. There will be a producer that will be sending messages to the SQS queue. There can be multiple producer sending the messages.

Message can be anything.

Consumer will be polling (checking for messages) the messages from the queue (hey do you have any messages for me), then will process the message then delete the queue.

Standard queues

The first services in AWS. Used for decoupling applications (If two applications are directly connected, we break it up by introducing an intermediate service to deliver the messages asynchronously to scale better).

For standard queue you get unlimited throughput (send as many as you want) and unlimited messages in queue.

Max retention of message: 4 days, up to 14 days. Low latency for sending and receiving it < 10ms for publish and receive.

Size < 256KB

Duplicate message can happen! Standard queue is delivery at least once occasionally service. You would need to take that into account in your application.

It is also best effort ordering, so messages can be out of order not in the order that they are sent.

Sending/reading messages

Application will produce message by using the SDK (SendMessage API).

Consumers can be running on EC2 instances, your own server, or AWS lambda (You can trigger AWS lambda via queues). Consumer poll for messages and can get up to 10 messages at a time and process them i.e. insert them into RDS database or just print it out.

After consuming message you will delete the message using DeleteMessage API.

Multiple consumers

SQS can have multiple consumers and process messages in parallel. So in the case that the consumer doesn't process the message fast enough it will be received by other consumer, since the message hasn't been deleted yet.

This is why it is at least once delivery and also best effort ordering.

ASG with SQS

You can scale consumer using auto scaling group polling from SQS. The metric to scale from is the Queue Length.ApproximateNumberOfMessages. Once the length go over a certain level it will trigger the auto scaling group to spin up more EC2 instances to handle those messages. So you can handle more messages per second.

Example decoupling application

If you have an application that receives request to process a video and store it into a S3 bucket. If you have many request then it might not be able to handle it quickly. So you can decouple your application separating the request from the handling of the request, by storing the request into a SQS then process it with another application.

This is so that your frontend application will not be lagged or stuck at receiving request and HANDLING it at the same time. You can scale the frontend and backend independently.

SQS Security

It has in-flight encryption using HTTPS.

At-rest encryption using KMS keys.

client-side encryption needs to be done by client itself.

Access control is done by IAM policies, but you also have SQS access policies just like S3 bucket policies for allowing other services to access the SQS queue.