Sunday, March 31, 2019

ProvisionedThroughputExceededException using KCL (Kinesis Client Library)

ProvisionedThroughputExceededException

When using the Amazon Kinesis Client Library (KCL) one would not expect the table created by KCL itself to keep track of its streams, shards and checkpoints to ever be under-configured.
But it can be! Then you get the ProvisionedThroughputExceededException

Normally it will recover by retrying itself. But if it can't, other solutions exist:
  1. Change the read/write provisioned for that table to 'on-demand'. 
  2. Make sure your hashkeys are evenly distributed, as mentioned here not-evenly distributed hash-keys.
  3. Modify how many consumers (applications) are reading from the same shard(s) at about the same time, either by changing that design or increase the number of shards.
For a more detailed analysis, see the longer blogpost here

No comments: