gitlab-runner cache key bug: cache not being created anymore in gitlab steps

Introduction

Besides auto-updating also to the most recent version of the Docker image of maven:3-jdk-8, the Gitlab runners were also always updated to the most recent version.

Again not best-practice and again I learned the hard way: builds suddenly starting to fail.

The issue and workarounds/solutions

 Because suddenly indeed the docker image build of the application started to fail with this error:

Step 9/12 : ADD service1/target/*.jar app.jar
ADD failed: no source files were specified
ERROR: Job failed: exit code 1

It turned out the service1/target directory was empty (or missing, I forgot).

Investigating some more showed that the cache produced in previous steps in Gitlab was not there anymore. The version that this started appearing was:

Running with gitlab-runner 10.1.0 (c1ecf97f)
on gitlab-runner-hosted (70e74c0e)

From older successful builds I saw that at the end of the previous step, when the cache is created, you see something like this:   

Creating cache developbranch:1...
WARNING: target/: no matching files
service1/target/: found 87 matching files
service2/target/: found 33 matching files
untracked: found 200 files
Created cache
Job succeeded

But those loglines were now suddenly missing! And I noticed that the gitlab-runner version changed between the last successful build and this failing one.

So I had an area to focus on: the caching didn't work anymore.

The cache definition in the gitlab-ci.yml was:

cache:  key: "$CI_COMMIT_REF_NAME"
  paths:    - target/
    - service1/target/
    - service2/target/
  untracked: true

I suspected maybe the environment variable of the key: field being empty or something.

But when I added logging in other script steps, the $CI_COMMIT_REF_NAME variable was filled with the value 'developbranch'.  So it is not empty.

Then I had an epiphany and prefixed the above environment variable with a string, making the cache key: definition look like this:

cache:  key: prefix-"$CI_COMMIT_REF_NAME"
  paths:    - target/
    - service1/target/
    - service2/target/
  untracked: true

In the above you can see I prefixed the key with the hardcoded string "prefix-".  And indeed that did it, the creating of the cache worked again and looked like this:

Creating cache prefix-developbranch:1...
WARNING: target/: no matching files
service1/target/: found 87 matching files
service2/target/: found 33 matching files
untracked: found 200 files
Created cache
Job succeeded

So also here I learned (as I really already knew): don't auto-update but do that in a controlled way, so you know when to expected potentially failing builds.

Gitlab maven:3-jdk-8 Cannot get the revision information from the scm repository, cannot run program "git" in directory error=2, No such file or directory

Introduction

In a Gitlab project setup the most recent Docker image for maven was always retrieved from the internet before each run by having specified image: maven:3-jdk-8 in the gitlab-ci.yml. The image details can be found here.

Of course this is not a best-practice; your build can suddenly start failing at a certain point because an update to the image might have something changed internally causing things to fail.

What you want is controlled updates. That way you can anticipate on builds failing and plan the upgrades in your schedule.

The issue and workarounds/solutions

And indeed suddenly on March 29 2018 our builds started failing with this error:

[ERROR] Failed to execute goal org.codehaus.mojo:buildnumber-maven-plugin:1.4:create (useLastCommittedRevision) on project abc: Cannot get the revision information from the scm repository :
[ERROR] Exception while executing SCM command.: Error while executing command. Error while executing process. Cannot run program "git" (in directory "/builds/xyz"): error=2, No such file or directory

That message is quite unclear: is git missing? Or is the directory wrong? Or could the maven buildnumber plugin not find the SCM repository?

After lots of investigation it turned out the maven:3-jdk-8 image indeed had changed about 18 hours before.

And after running the maven command in a local version of that Docker image indeed the same error occured!  Awesome, the error was reproducable.

And after installing git again in the image with:

- apt-get update
- apt-get install git -y

the error disappeared!  But a new one appeared:

[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?

This also hadn't happened before. After some searching it turned out it might be the surefire and failsafe plugins being outdated.

So I updated them to 2.21.0 and indeed the build succeeded.

Here's the issue reported in the Docker Maven github. UPDATE: it is caused by an openjdk issue (on which the maven:3-jdk-8 is based upon.

This issue made us realize we really need an internal Docker repository. And so we implemented that :)

One disadvantage about Docker images is that you can't specify a commit hash to use. Yes you can specify a digest instead of a tag, but that is a unique UUID hashcode only. You can't see from that hashcode anymore the (related) tagname.

Lessons learned - Jackson, API design, Kafka

Introduction

This blogpost describes a bunch of lessons learned during a recent project I worked on.

They are just a bunch grouped together, too small to "deserve" their own separate post :)

Items discussed are: Jackson, JSON, API design, Kafka, Git.

Lessons learned

• Pretty print (nicely format) JSON in a Linux shell prompt:
  
  cat file.json | jq
  
  You might have to 'apt-get install jq' first.
  
  
• OpenOffice/LibreOffice formulas:
  
  To increase date and year by one from cell H1031: =DATE(YEAR(H1031)+1; MONTH(H1031)+1; DAY(H1031))
  
  Count how many times a range of cells A2:A4501 has the value 1: =COUNTIF(A2:A4501; 1)
  
  
• For monitoring a system a split between a health-status page and performance details is handy. The first one is to show issues/metrics that would require immediate action. Performance is for informational purposes, and usually does not require immediate action.
  
  
• API design: Even if you have a simple method that just returns a date (string) for example, always return JSON (and not just a String of that value). Usueful for backwards compatibility: more fields can be easily added later.
  
  
• When upgrading gitlab, it (gitlab) had changed a repository named 'users' to 'users0'. Turns out 'users' is a reserved repository name in gitlab since version 8.15.
  
  To change your local git settings to the new users0 perform these steps to update your remote origin:
  
  # check current setting
  $ git remote -vorigin  https://gitlab.local/gitlab/backend/users (fetch)
  origin  https://gitlab.local/gitlab/backend/users (push)
  
  # change it to the new one
  $ git remote set-url origin https://gitlab.local/gitlab/backend/users0
  
  # see it got changed
  $ git remote -v
  origin  https://gitlab.local/gitlab/backend/users0 (fetch)
  origin  https://gitlab.local/gitlab/backend/users0 (push)
  
  
• Jackson JSON generating (serializing): probably a good practice is to not use @JsonInclude(JsonInclude.Include.NON_EMPTY)  or NON_NULLm since that would mean a key will be just not in the JSON when its value is empty or null. That could be confusing to the caller: sometimes it's there sometimes not.  So just leave it in, so it will be set to null.   Unless it would be a totally unrelated field like: amount and currency. If amount is null, currency (maybe) doesn't make sense, so then it could be left out.
  
  
• Java:
  
  Comparator userComparator = (o1, o2)->o1.getCreated().compareTo(o2.getCreated());
  
  can be replaced now in Java 8 by:
  
  Comparator userComparator = Comparator.comparing(UserByPhoneNumber::getCreated);
  
  
• Kafka partitioning tips: http://blog.rocana.com/kafkas-defaultpartitioner-and-byte-arrays
  
  
• Kafka vs RabbitMQ:
  
  - Kafka is optimized for producers producing lots of data (batch-oriented producers) and consumers that are usually slower that the producers.
  - Performance: Rabbit: makes about 20K/s  Kafka: up to 150K/s.
  - Unlike other message system, Kafka brokers are stateless. This means that the consumer has to maintain how much it has consumed.