Amazon Kinesis

Amazon Kinesis Data Streams is a high performance, scalable data streaming service and is our default choice for stream-based integration between components in AWS. Kinesis is a fully managed service which runs without the need to have a VPC to host instances, but capacity does need to be pre-provisioned, rather than scaling smoothly and automatically.

As with other stream-based systems, records are retained for a configured retention period, and can be read any number of times by any consumer during that time. This is in contrast to queue-based systems like SQS and RabbitMQ which delete records once delivered to consumers.

Each Kinesis stream distributes records between a configured number of shards. The number of shards determines the maximum throughput and the number of concurrent consumers which can process records in parallel - but it also determines the cost, since each shard is charged at an hourly rate.

Compared to SQS, distinct features of Kinesis are:

• Kinesis has stream semantics as described above, as opposed to the queue model of SQS.
• Kinesis's performance profile must be tuned by choosing the number of shards and partition key, while SQS scales smoothly and elastically.
• Kinesis can handle higher message rates while preserving FIFO ordering, while SQS standard queues can elastically scale almost without limit if FIFO ordering is not required.

Compared to Kafka,

• Kinesis is a managed service, which significantly reduces operational complexity and cost.
• Kafka is open source and can be run in any cloud or physical data centre.