ReadProvisionedThroughputExceeded using Kinesis

ReadProvisionedThroughputExceeded

When using Kinesis it is possible you'll get an ReadProvisionedThroughputExceeded when not carefully designing the shards consumption of messages.

That exception indicates "The number of GetRecords calls throttled for the stream over the specified time period".

A cause for this can be you have to many consumer applications at the same time issue GetRecord calls to the same shard.

One recommended solution is to increase the number of shards. But that has a price-tag attached to it. Another solution is process (batches) of records in parallel by spinning off threads within the processRecords() method in KCL. But that will require careful orchestration of threads to keep checkpointing correctly. An interesting analysis of why using KCL and especially version 2 can be found here.

For a more detailed analysis, see the longer blogpost here.

ProvisionedThroughputExceededException using KCL (Kinesis Client Library)

ProvisionedThroughputExceededException

When using the Amazon Kinesis Client Library (KCL) one would not expect the table created by KCL itself to keep track of its streams, shards and checkpoints to ever be under-configured.

But it can be! Then you get the ProvisionedThroughputExceededException. 

Normally it will recover by retrying itself. But if it can't, other solutions exist:

1. Change the read/write provisioned for that table to 'on-demand'. 
2. Make sure your hashkeys are evenly distributed, as mentioned here not-evenly distributed hash-keys.
3. Modify how many consumers (applications) are reading from the same shard(s) at about the same time, either by changing that design or increase the number of shards.

For a more detailed analysis, see the longer blogpost here.