Wednesday, October 2, 2019

Snyk .snyk ignore file with multiple entries format

"Snyk enables you to find, and more importantly fix known vulnerabilities in your open source."

Some vulnerabilities you'd want to ignore, because for example they are (for you) either false positives, or there is no workaround for them (yet). You can specify them via the so-called CLI snyk ignore command, or in the UI manually, or via a .snyk ignore file. An introduction focused on NPM can be found here.
The format with an example for one entry is specified here:

But what is the format for multiple entries? Most of the time with yaml (-like) formats you might expect a list, so each entry should be prefixed with a dash "-". But that is not the case here. The correct format for specifying multiple entries in the .snyk ignore file is:
version: v1.5.0
    - '* >':
      reason: 'Fix in progress'
      expires: '2020-01-05T08:00.00.000Z'
    - '* >':
      reason: 'Windows issue, systems run on Linux'
      expires: '2020-02-06T09:00.00.000Z'
Note each block after the 'ignore' line is indented with two spaces.

Saturday, July 27, 2019

PACT Consumer Driven Contract Testing: how to allow any body in the response

Consumer Driven Contract testing is a way to ensure that services (such as an API provider and a client) can communicate with each other. Without contract testing, the only way to know that services can communicate is by using expensive and brittle integration tests.
PACT is a contract testing tool.  

For response matching you want to be as loose as possible with the matching for the response (will_respond_with(...)) though. This stops the tests being brittle on the provider side.
 Most of the time you don't care about the exact values of the (JSON or XML) response, but you do care about the types of the values, e.g a string or a number.
In that case you'll be using 'type matching'.

Sadly for JVM matchers there's no such handy method that with one invocation makes sure only types are validated, as is for example with Ruby/Groovy/Javascript/Node: Pact::SomethingLike.
Not much documentation on the Java/JVM matchers can be found at the official Pact site itself. Mostly the examples are for Ruby/Groovy/Javascript/Node.

For  the Java and Virtual Machine integration you'd use pact-jvm with matchers like .stringType() for the body using the PACT DSL or this lambda extension.

But hard to find was how to specify allowing either an empty body in the response or any data in the body in the response, while the consumer does not require a body at all (or in other words: is fine with an empty body or any fields in it).
Solution: for that you need to completely omit the .body() in the consumer contract definition.
If you specify a .body(new PactDslJsonBody()), the contract generated matcher will specify "body" : {}, and therefor requiring an empty body. And if then the provider (test) generates one or more fields in the response, you'll see this as the message in the failing test:

Expected an empty Map but received Map(..... fields added by provider ...)

So full example which accepts an empty body or a body with elements in it:

.consumer("Some Consumer")
.hasPactWith("Some Provider")
.given("a certain state on the provider")
    .uponReceiving("a request for something")

Tuesday, July 9, 2019

Maven Failsafe plugin build error: SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?

When you get this exception during running mvn clean verify, which executes both the Surefire and Failsafe plugin (for which the last one uses the Surefire plugin!):

SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?

it can mean several things. For example as the message says maybe somewhere in your integration tests you call System.exit, which you shouldn't do. That one is easy to find. Also this Stackoverflow post covers that plus some other options, like shortage of memory.

It has also higher chance on happening when using the docker-maven image and 3.x-openjdk-8 or 3.y-openjdk-10.

But if that doesn't fix it, what then?  Then due to this issue you probably have to add this property in your pom.xml in the configuration of both the Surefire and the Failsafe plugin:


Thus not only for the Failsafe plugin, for both! :)

PS: adding that configuration property also is a workaround for, also reported here:

Sunday, March 31, 2019

ReadProvisionedThroughputExceeded using Kinesis


When using Kinesis it is possible you'll get an ReadProvisionedThroughputExceeded when not carefully designing the shards consumption of messages.

That exception indicates "The number of GetRecords calls throttled for the stream over the specified time period".

A cause for this can be you have to many consumer applications at the same time issue GetRecord calls to the same shard.

One recommended solution is to increase the number of shards. But that has a price-tag attached to it. Another solution is process (batches) of records in parallel by spinning off threads within the processRecords() method in KCL. But that will require careful orchestration of threads to keep checkpointing correctly. An interesting analysis of why using KCL and especially version 2 can be found here.

For a more detailed analysis, see the longer blogpost here.

ProvisionedThroughputExceededException using KCL (Kinesis Client Library)


When using the Amazon Kinesis Client Library (KCL) one would not expect the table created by KCL itself to keep track of its streams, shards and checkpoints to ever be under-configured.
But it can be! Then you get the ProvisionedThroughputExceededException

Normally it will recover by retrying itself. But if it can't, other solutions exist:
  1. Change the read/write provisioned for that table to 'on-demand'. 
  2. Make sure your hashkeys are evenly distributed, as mentioned here not-evenly distributed hash-keys.
  3. Modify how many consumers (applications) are reading from the same shard(s) at about the same time, either by changing that design or increase the number of shards.
For a more detailed analysis, see the longer blogpost here

Wednesday, March 20, 2019

Lessons learned using AWS Kinesis to process business events and commands (messages)


When using AWS Kinesis as the means of communicating business events, several challenges arise. For the basic key concepts of Kinesis see here

Of course the "normal" use case to use Kinesis is described as here in the section "When should I use Amazon Kinesis Data Streams, and when should I use Amazon SQS?" Another comparison between Kinesis and SQS can be found here.

Especially built for Kafka-like stream-processing of millions of events, where usually the producers (e.g think IoT devices) are much faster than the consumer applications.
Usually business events are not generated millions per minute or second. Events from IoT devices are usually not considered business events.
Any next time I would not recommend using Kinesis as a transport mechanism for business events; the main reason for me is its intended use-case is not matching that type of use. Plus the many technical challenges I had to solve, for most of which any "regular" messaging tool like RabbitMQ would suffice.
Additionally, usually one wants business events to arrive almost instantly - or at least as fast as possible. Kinesis is fast but e.g RabbitMQ is often sufficient as transport for business events.
And yes RabbitMQ is now also available as a managed solution - hosted at AWS but managed not by AWS itself. For example with CloudAMQP.

In the below discussion a message can be an event or a command (though most often you'd want the commands to be synchronous, since they should only be delivered to one service anyway)

High level Kinesis architecture overview

Below is a high level overview of Kinesis' architecture: 

There can be multiple streams, and within each stream are one or more shards. Messages in a shard are guaranteed to be delivered at the consumer in the order they were published onto the shard.
The partition-key used while publishing determines into what shard the message is published. Messages with the same partition key get always published into the same shard.

A design decision one has to make is: do I make one stream were all services publish their messages, or do I want a stream per XYZ? Where XYZ could be for example a Bounded Context.
No good reason was found to split into multiple streams, so I decided to go for one Kinesis stream all services publish on.

Lessons learned

When using Kinesis to handle business events between micro-services several challenges were to overcome. Below the lessons learned are described.

More than five different applications processing the same shard at the same time

Kinesis limits documentation  states that each shard within a stream supports up to five read transactions per second. And if you need more than that, it is recommended to increase the number of shards.
If you don't comply to that limit, you'll see ReadProvisionedThroughputExceeded or ProvisionedThroughputExceededException exceptions appear in your logs. This can easily be reached if you have large documents (e.g 1MB) in your DynamoDB too; even with the AWS Console you then can't even view such a large document in your browser!
Increasing the number of shards will spread the published messages across more shards and thus reduce the number of read transactions per second per shard.
BUT: in a microservices architecture you can easily have 10 or more services. And all these services need to process all messages on all shards.
So with for example 10 services, each of those services (container or lambda) will need to poll each shard regularly in some form. And the more services, the more chance that more than 5 services poll any given shard per second... Causing the above ReadProvisionedThroughputExceeded exception.
Thus: increasing the number of shards won't help, because still each service will need to poll each shard.

One workaround for this could be:
  • have each service (container, lambda) publish to one central stream. Note that multiple streams in the end will also not help you, since you'll reach that 5 reads per second limit soon as the number of services increases to 5 or higher
  • have a stream per service that needs to consume messages
  • have a smart lambda that reads from the central stream and re-publishes each message on each of the per-service-streams using the same partitionkey the publisher used
  • in case of error in that smart lambda: the lambda should store the failed events somewhere, e.g DynamoDB for later investigation. But when such an error occurs, that means what is received by the consumer is not in the same order anymore as the order the publisher published the messages (the consumer misses a message! And might get it in later when the investigation decides to republish it on the central stream)
  • note that this introduces at least one second delay because of the smart lambda being allowed to poll once a second. And then if the consuming service is also a lambda, another second delay is introduced.
The above solution makes sure each service-stream has only one type of application consuming and any read limits can be fixed by increasing the number of shards (because only one service will be reading from it anyway).
Note Enhanced Fanout was not released yet then, which seems to also solve the described issue.

Options tried by changing the configurations: 
  1. KCL: increasing withIdleTimeBetweenReadsInMillis  and withIdleTimeBetweenCallsInMillis: only helps in a limited way plus reduces throughput.
  2. KCL: same for withMaxrecords(). Even if there are no records, all services will still from time to time have to poll the shard...
  3. Increasing AWS provisioned maximum could of course help but what's the final maximum there? It will probably need to be increased for every X new services.

Increasing throughput

When the consumer needs to be really fast in processing messages from a Kinesis stream shard, one can spin off a new thread per record in the batch. But several things have to be taken into account when doing that:
  • checkpointing is still at record (or batch) level. So you still have to in some way checkpoint only if you know all message before that moment are processed fine too. So you'll need some thread-orchestration to determine to what record to checkpoint
  • if you care about the order, then a thread per record causes processing-ordering issues; so in that case you'll have to first group all records together that apply to the same entity, then start a thread to process these as a whole 

No direct Kinesis consumer

A totally different approach is to have your container services not be bothered with Kinesis at all, and put Lambdas in front of them, which invoke the services via regular REST calls. This alleviates the consumers completely from processing messages.
An example framework that provides this is Zalando's Nakadi. Though implemented for Kafka, still a viable option. This architecture is definitely something to consider.
A more advanced solution could be to have the lambdas put the messages in an SQS queue. This provides even more decoupling and if your service is not available, the lambdas can still deliver the messages into the SQS; care has to be taken of how to support multiple consumers reading from the same SQS queue. Of course then you almost must start to wonder why you are using Kinesis when you are putting a regular queue behind it...

Kinesis Client Libray (KCL) challenges

For the Java consumers the recommended option by AWS is to use the Kinesis Client Library, KCL for short. A short high level introduction can be found here.
This library takes away a lot of standard challenges in a distributed solution. See below's KPL link to implementing efficient producers for an explanation.
But still quite a few challenges exist. E.g how does it handle multiple instances running a Worker? Should you checkpoint per batch or per record?
Are batches provided to the worker one at at time, sequentially or in parallel? How to handle failed messages?
Should workers have a unique ID?

Here's a summary of lessons learned:
  • a shard will be consumed by a record processor thread (not necessarily always the same thread!) of one single worker only at any given moment. And thus batches will never be handed over to processRecords() in parallel; first batch 1, then batch 2 etc.   consumeShard() is protected via synchronized.
  • starting multiple workers on same host won't improve throughput; one on each host seems most effective, otherwise Workers start maybe also stealing leases too many times/too much
  • give each worker a unique ID per host so you can see which host has what lease
  • seems best to pass in your own Executor to the KCL when using Spring, so Spring knows about that threadpool. 
Note that checkpointing too much (or querying your own DynamoDB database while processing each message) can easily cause a ProvisionedThroughputExceededException.
Therefore you also don't want to do that too often. In the end this is the high level algorithm (pseudo code) that one can use to cover all the above concerns:

In processRecords(records) {
   lastSuccessfulRecord = null;
   for (each record in records) {
      try {
         parse record into domain object;
         process(domain object);
         lastSuccessfulRecord = record;
      } catch (ex) {
   checkpointBatch(); // All records in the batch successfully processed
Another question for next implementation will be: should the service really be doing smart checkpointing (i.e only checkpoint when record is processed successfully; if not successful, keep processing but at next restart of service start from the last checkpoint)? Because the "normal" messaging implementation of putting a message on a Dead Letter Queue might be a better solution. Though in that solution you'll have to think about edge scenarios, like when a message A to modify entity E is put on the DLQ, but a following message B related to entity E is processed successfully; should message A be replayed (published) on the main stream again? Or should as soon as one message for E is put onto the DLQ, all messages after that for E also be put on the DLQ?

Kinesis Producer Library (KPL) challenges

For the Java producers the recommended option by AWS is to use the Kinesis Producer Library, KPL for short. For key concepts see here.
See here for a thorough explanation of its advantages with example code.

The only challenge there was: how large should the batch size be?  It seems to depend mostly on how well your consumers can handle batches.
In the end I set it to values between 10 and 100.
Note that the standard AWS Kinesis SDK does not implement handling batches. That means if you use the KPL with batchsize 10, then when these batches of records are received by e.g Javascript lambdas using the standards SDK, they by default can't process them correctly because batches are not supported; unless you write code to unwrap the batch.
As a temporary workaround there the batchsize could be set to size 1 of course, which then makes sure each consumer gets to process one message at a time and not in a batch.

Hope these lessons learned help somebody while implementing Kinesis.

Wednesday, April 4, 2018

gitlab-runner cache key bug: cache not being created anymore in gitlab steps


Besides auto-updating also to the most recent version of the Docker image of maven:3-jdk-8, the Gitlab runners were also always updated to the most recent version.

Again not best-practice and again I learned the hard way: builds suddenly starting to fail.

The issue and workarounds/solutions

 Because suddenly indeed the docker image build of the application started to fail with this error:

Step 9/12 : ADD service1/target/*.jar app.jar
ADD failed: no source files were specified
ERROR: Job failed: exit code 1

It turned out the service1/target directory was empty (or missing, I forgot).
Investigating some more showed that the cache produced in previous steps in Gitlab was not there anymore. The version that this started appearing was:

Running with gitlab-runner 10.1.0 (c1ecf97f)
on gitlab-runner-hosted (70e74c0e)
From older successful builds I saw that at the end of the previous step, when the cache is created, you see something like this:   

Creating cache developbranch:1...
WARNING: target/: no matching files
service1/target/: found 87 matching files
service2/target/: found 33 matching files
untracked: found 200 files
Created cache
Job succeeded

But those loglines were now suddenly missing! And I noticed that the gitlab-runner version changed between the last successful build and this failing one.
So I had an area to focus on: the caching didn't work anymore.

The cache definition in the gitlab-ci.yml was:

cache:  key: "$CI_COMMIT_REF_NAME"
  paths:    - target/
    - service1/target/
    - service2/target/
  untracked: true

I suspected maybe the environment variable of the key: field being empty or something.
But when I added logging in other script steps, the $CI_COMMIT_REF_NAME variable was filled with the value 'developbranch'.  So it is not empty.
Then I had an epiphany and prefixed the above environment variable with a string, making the cache key: definition look like this:

cache:  key: prefix-"$CI_COMMIT_REF_NAME"
  paths:    - target/
    - service1/target/
    - service2/target/
  untracked: true

In the above you can see I prefixed the key with the hardcoded string "prefix-".  And indeed that did it, the creating of the cache worked again and looked like this:

Creating cache prefix-developbranch:1...
WARNING: target/: no matching files
service1/target/: found 87 matching files
service2/target/: found 33 matching files
untracked: found 200 files
Created cache
Job succeeded

So also here I learned (as I really already knew): don't auto-update but do that in a controlled way, so you know when to expected potentially failing builds.

Friday, March 30, 2018

Gitlab maven:3-jdk-8 Cannot get the revision information from the scm repository, cannot run program "git" in directory error=2, No such file or directory


In a Gitlab project setup the most recent Docker image for maven was always retrieved from the internet before each run by having specified image: maven:3-jdk-8 in the gitlab-ci.yml. The image details can be found here.

Of course this is not a best-practice; your build can suddenly start failing at a certain point because an update to the image might have something changed internally causing things to fail.
What you want is controlled updates. That way you can anticipate on builds failing and plan the upgrades in your schedule.

The issue and workarounds/solutions

And indeed suddenly on March 29 2018 our builds started failing with this error:

[ERROR] Failed to execute goal org.codehaus.mojo:buildnumber-maven-plugin:1.4:create (useLastCommittedRevision) on project abc: Cannot get the revision information from the scm repository :
[ERROR] Exception while executing SCM command.: Error while executing command. Error while executing process. Cannot run program "git" (in directory "/builds/xyz"): error=2, No such file or directory

That message is quite unclear: is git missing? Or is the directory wrong? Or could the maven buildnumber plugin not find the SCM repository?
After lots of investigation it turned out the maven:3-jdk-8 image indeed had changed about 18 hours before.
And after running the maven command in a local version of that Docker image indeed the same error occured!  Awesome, the error was reproducable.
And after installing git again in the image with:

- apt-get update
- apt-get install git -y

the error disappeared!  But a new one appeared:

[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
This also hadn't happened before. After some searching it turned out it might be the surefire and failsafe plugins being outdated.
So I updated them to 2.21.0 and indeed the build succeeded.

Here's the issue reported in the Docker Maven github. UPDATE: it is caused by an openjdk issue (on which the maven:3-jdk-8 is based upon.

This issue made us realize we really need an internal Docker repository. And so we implemented that :)

One disadvantage about Docker images is that you can't specify a commit hash to use. Yes you can specify a digest instead of a tag, but that is a unique UUID hashcode only. You can't see from that hashcode anymore the (related) tagname.

Saturday, February 3, 2018

How to prevent Chromium from rebooting a Raspberry Pi 3 model B Rev 1.2

I tried to use a Raspberry Pi 3 model B Rev 1.2 as a dashboard for monitoring a couple of systems using Chromium as browser.

Tip: use  this to have it never turn off the display:
sudo xset s off
sudo xset -dpms
sudo xset s noblank

I had only two tabs open all the time and was using the Revolver browser extension to rotate the tabs. One tab had the default Datadog page open, another a custom dashboard within Kibana that refreshed every 15 minutes.
Using all default settings, within a few hours the Pi would reboot out of its own! So something got it to do that.

It seemed the browser (tabs) or the Javascript in them were just leaking so much memory that the Pi ran out of memory.  I tried multiple times with the same default setup, but the behavior was the same each time.

So I tried a couple of other things:
  1. Have the Revolver plugin fully reload the page. Still a reboot of the Pi, though it took a bit longer

  2. Added --process-per-site to the startup shortcut of Chromium. This causes Chrome to create less processes and that should reduce the memory usage a bit. But still a reboot of the Pi; though again it took a bit longer.
    Note that this also comes with its own weaknesses.

  3. Added --disable-gpu-program-cache to the startup shortcut of Chromium. Again still rebooted the Pi after a while.

  4. Tried other browsers like Midori and Firefox Iceweasel.  Midori does not have a Revolver-like plugin, so it didn't fit the requirements. Firefox's only add-on that should work gave some kind of "invalid format" error (don't remember exactly) when trying to install it. The other add-ons for Firefox were not compatible with Iceweasel.

So in the end I did not find a solution :(  I just built a cron-job that would restart the browser every 5 hours.
If you found a way to fix this problem, let the world know in the comments!

Friday, December 29, 2017

Upgrading Dell M4700 from 500G HDD to 1T Samsung Evo 850 SSD v-nand

Below is a list of things encountered when upgrading a Dell Precision M4700 laptop with Windows 7 with a 500G HDD to a 1 terabyte Samsung Evo v-nand 850 SSD.

  • Used HDClone 7 Free Edition to make a copy of the current harddisk. Copied onto a Samsung T5 external SSD.  HDClone very nicely copies over everything, even from a live (running) Windows 7 machine. It creates all partitions also on the external SSD; these are visible as separate drives when reconnecting the USB drive. It can be even made bootable, but I didn't need that.

  • Swapped in the SSD as shown here:
    Note how the SDD has to "click" with the notches of the bracket (metal frame).

  • Put in the Dell Windows 7 SP1 DVD. It installed. For quite a while it shows "Windows is loading files...", but in the end it got through, after about 5-10 minutes.

  • After logging in, the wireless device was not detected. So no internet. Also other drivers were not installed yet or failed:

  • So: "Ethernet controller", "Network controller", "PCI Simple Communications Controller", "SM Bus Controller", "Universal Serial Bus (USB) Controller", "Unknown device"

  • Then tried to use the Dell 'Resource Media' DVD to install the drivers. But the usage of the program that starts then is just plain impossible to understand. E.g see the screenshot below:

    The touchpad is marked as installed (I think that the checkbox indicates that). When I then installed any other driver, no checkbox appeared at all on the left of any of them.
    Plus, what is the order to install the drivers? Found this post but seems a lot of manual work. Plus you have to know which devices you have in your machine to know which driver matches. Eg this post shows what driver to look for for one specific error. That list you should be able to find on the bill when you ordered your M4700.

    In the end managed to get the ethernet driver installed (filter on the word 'ethernet' on the Dell drivers page, you should find "Intel I2xx/825xx Gigabit Ethernet Network Controller Drivers")

  • Then managed to get the internet connection working via a wired cable after installing the above ethernet driver. Then used the Dell's Analyze Detect Drivers option  to update the correct drivers in the right order.
    All but the 'Unknown device' errors where gone in the Device Manager. Didn't dare to update the BIOS since that was working fine before.

  • After that about 180 windows updates to install and then all worked fine. Score of machine is now 7.3 on a scale of maximum 7.9 (no idea what it was before I upgraded):

    But I do notice the difference in startup for example: complete Windows 7 startup from powered off state is about 10-15 seconds. Not bad :)

  • And then the tedious job of installing all non-OS software began...
Lessons learned for next time:
  • HDClone is very handy
  • Export all browsers's favorites before taking out the old disk. You can find the favorites back, but not in an easy importable format.
  • Impossible to understand the Dell Resource Media DVD.  
  • Keep the service tag of your M4700 ready.