SQS, SNS, Kinesis, Active MQ

Decoupling mechanism

When we deploy multiple applications they will inevitably need to communicate. There is two ways of doing the application.

1. Synchronous communication: Direct connection for direct communication between application
2. Asynchronous / event based communication: Application is connected to a queue, then the message will come asynchronous and the application will react to that message to do whatever it needs to do

Synchronous can be a problem if there is sudden spike of traffic. It is better to decouple your application so that your application can scale independently, without worrying about direct communication

Amazon SQS (Simple queue service)

It is a queue, it will have messages. There will be a producer that will be sending messages to the SQS queue. There can be multiple producer sending the messages.

Message can be anything.

Consumer will be polling (checking for messages) the messages from the queue (hey do you have any messages for me), then will process the message then delete the queue.

There are two types of SQS, standard queues and FIFO queues.

Standard queues

The first services in AWS. Used for decoupling applications (If two applications are directly connected, we break it up by introducing an intermediate service to deliver the messages asynchronously to scale better).

For standard queue you get unlimited throughput (send as many as you want) and unlimited messages in queue.

Max retention of message: 4 days, up to 14 days. Low latency for sending and receiving it < 10ms for publish and receive.

Size < 256KB

Duplicate message can happen! Standard queue is delivery at least once occasionally service. You would need to take that into account in your application.

It is also best effort ordering, so messages can be out of order not in the order that they are sent.

Sending/reading messages

Application will produce message by using the SDK (SendMessage API).

Consumers can be running on EC2 instances, your own server, or AWS lambda (You can trigger AWS lambda via queues). Consumer poll for messages and can get up to 10 messages at a time and process them i.e. insert them into RDS database or just print it out.

After consuming message you will delete the message using DeleteMessage API.

Multiple consumers

SQS can have multiple consumers and process messages in parallel. So in the case that the consumer doesn't process the message fast enough it will be received by other consumer, since the message hasn't been deleted yet.

This is why it is at least once delivery and also best effort ordering.

ASG with metric of SQS

You can scale consumer using auto scaling group polling from SQS. The metric to scale from is the Queue Length.ApproximateNumberOfMessages. Once the length go over a certain level it will trigger the auto scaling group to spin up more EC2 instances to handle those messages. So you can handle more messages per second.

SQS as database buffer!

With the same setup as the previous statement. If we have request going into an auto scaling group and then the EC2 instances write say transactions into database in order to store records of the transactions. Sudden spike of requests will OVERWHELM the databases and some writes to the database will fail! How do we solve this? We use SQS as a buffer to store the messages that we need to write, because it is infinitely scalable, can store unlimited amount of messages. Then we can poll the messages with another auto scaling group to then process the transactions, and then the transaction are only deleted if the insertion succeed. If it failed then we don't delete the message and just try again.

Without the SQS buffer there are chances that the request writing to the database will fail and the transaction is lost! If it is done asynchronously, which mostly is.

Decoupling frontend and backend application

If you have an application that receives request to process a video and store it into a S3 bucket. If you have many request then it might not be able to handle it quickly. So you can decouple your application separating the request from the handling of the request, by storing the request into a SQS then process it with another application.

This is so that your frontend application will not be lagged or stuck at receiving request and HANDLING it at the same time. You can scale the frontend and backend independently.

SQS Security

It has in-flight encryption using HTTPS.

At-rest encryption using KMS keys.

client-side encryption needs to be done by client itself.

Access control is done by IAM policies, but you also have SQS access policies just like S3 bucket policies for allowing other services to access the SQS queue.

Message visibility timeout

After a message is polled by a consumer, it becomes invisible to other consumers for some time. That timeout is 30 seconds, which means the message has 30 seconds to be processed. But if the message hasn't been deleted it will be "put back to the queue", and other consumer can receive the same message again.

This is how the message can be deliver multiple time time if it hasn't processed fast enough.

If your application know it is processing it and needs a little bit more time before it finishes consumer can call ChangeMessageVisiblity to change the default invisible time. The visibility timeout if you set it high and consumer crash seeing the message again will take long time. If too long, then duplicate messages can occur.

Long polling

Consumer can wait for messages to arrive if there are none in queue. This is done to reduce the number of API calls to SQS queues and decrease latency for your application for the time for message to arrive to your application. This is referred as long polling.

Long polling can last from 1 to 20 seconds. 20 seconds preferred.

You can configure it at the queue level (every poll you do will be long poll) or you do it at API level by specifying how long you will be waiting.

FIFO queues

First in first out. The message order will be guaranteed as oppose to standard queue which ordering isn't guaranteed but best effort. When you poll for messages it will receive the message in the order it was sent.

It has limited throughput, can only sent 300 messages per second or 3000 messages per second with batching.

You will only receive the messages once! And messages are processed in the order sent by the consumer.

SNS (Simple notification service)

One producer that can sent same message to many receiver! Publish-subscribe pattern.

You will have your producer sent a message to the SNS Topic, and the topic can be subscribed by consumer who would like to receive the message, there can be many subscriber, and process the message.

Producer send message to one SNS topic. Consumer listen to the topic that they want. You can have 12m+ subscription per topic which is a lot!

SNS can sent data to subscriber which includes email,email subscriber, mobile notification,notification subscriber, HTTP endpoint, SQS, lambda, Kinesis data firehose.

Lots of services will publish messages too, and they will sent it to a specified SNS topic. ASG scaling event for example.

Publish

To publish message you use SDK, you create topic, create subscription, then you publish to the topic

Security

In-flight encryption, at-rest encryption using KMS keys. If client wants to handle encryption they have to do it.

IAM policies to regulate access to SNS resources. Also can define SNS policies to define what other services can access SNS.

SNS + SQS Fan out pattern

You want message to sent to multiple SQS. If you sent it individually then it might not be guaranteed. Use fan out pattern!

Push once in SNS topic, SQS will be the subscriber to the topic. This will guarantee the message be sent to the SQS topic.

This is fully decoupled, no data loss. SQS allow for data persistence, delaying the process work since is stored in queues.

You need to allow SNS to write to the SQS. SQS can also be in other region as well and it will work!

S3 event to multiple queues

You can only sent S3 event like an object being created to one location, but what if you want to sent it to multiple places? Then you can use the SNS + SQS fan out pattern. Let the event notification be publish to SNS. Then you have SQS as subscribers, then that event notification can then go to multiple places where ever you like! Services, services, other lambda!

SNS to S3 via Kinesis data firehose

Kinesis data firehose let you deliver streaming data (data that are continuously being produced, logs of customers, ...) to a destination. You can apply transformation before storing the data to say S3 bucket.

SNS can also be used to sent data to Kinesis. So you sent the data to SNS topic, then Kinesis data firehose can subscribe to it. because it let you set a destination for the streaming data you can set destination to say a S3 bucket to store it there. Or do some transformation before storing it into S3.

FIFO topic + SQS FIFO

Same like FIFO queue, the messages will be ordered for a topic that is published to.

You get ordering and no duplication. However you can only have SQS FIFO as the subscriber and limited throughput just like FIFO queue.

You use this to do fan out for SQS FIFO

Messaging filtering

You can also add a JSON policy used as a filtering policy for each subscriber. Based on the JSON policy the subscriber will receive message that it filters for.

If a subscriber doesn't have a filter policy it will receive every message.

Amazon Kinesis