Amazon Kinesis Data Streams is a high performance, scalable data streaming service and is our default choice for stream-based integration between components in AWS. Kinesis is a fully managed service which runs without the need to have a VPC to host instances, but capacity does need to be pre-provisioned, rather than scaling smoothly and automatically.
As with other stream-based systems, records are retained for a configured retention period, and can be read any number of times by any consumer during that time. This is in contrast to queue-based systems like SQS and RabbitMQ which delete records once delivered to consumers.
Each Kinesis stream distributes records between a configured number of shards. The number of shards determines the maximum throughput and the number of concurrent consumers which can process records in parallel - but it also determines the cost, since each shard is charged at an hourly rate.
Compared to SQS, distinct features of Kinesis are:
- Kinesis has stream semantics as described above, as opposed to the queue model of SQS.
- Kinesis's performance profile must be tuned by choosing the number of shards and partition key, while SQS scales smoothly and elastically.
- Kinesis can handle higher message rates while preserving FIFO ordering, while SQS standard queues can elastically scale almost without limit if FIFO ordering is not required.
Compared to Kafka,
- Kinesis is a managed service, which significantly reduces operational complexity and cost.
- Kafka is open source and can be run in any cloud or physical data centre.