ReadProvisionedThroughputExceeded using Kinesis

ReadProvisionedThroughputExceeded

When using Kinesis it is possible you'll get an ReadProvisionedThroughputExceeded when not carefully designing the shards consumption of messages.

That exception indicates "The number of GetRecords calls throttled for the stream over the specified time period".

A cause for this can be you have to many consumer applications at the same time issue GetRecord calls to the same shard.

One recommended solution is to increase the number of shards. But that has a price-tag attached to it. Another solution is process (batches) of records in parallel by spinning off threads within the processRecords() method in KCL. But that will require careful orchestration of threads to keep checkpointing correctly. An interesting analysis of why using KCL and especially version 2 can be found here.

For a more detailed analysis, see the longer blogpost here.