Lessons learned - Jackson, API design, Kafka

Introduction

This blogpost describes a bunch of lessons learned during a recent project I worked on.

They are just a bunch grouped together, too small to "deserve" their own separate post :)

Items discussed are: Jackson, JSON, API design, Kafka, Git.

Lessons learned

• Pretty print (nicely format) JSON in a Linux shell prompt:
  
  cat file.json | jq
  
  You might have to 'apt-get install jq' first.
  
  
• OpenOffice/LibreOffice formulas:
  
  To increase date and year by one from cell H1031: =DATE(YEAR(H1031)+1; MONTH(H1031)+1; DAY(H1031))
  
  Count how many times a range of cells A2:A4501 has the value 1: =COUNTIF(A2:A4501; 1)
  
  
• For monitoring a system a split between a health-status page and performance details is handy. The first one is to show issues/metrics that would require immediate action. Performance is for informational purposes, and usually does not require immediate action.
  
  
• API design: Even if you have a simple method that just returns a date (string) for example, always return JSON (and not just a String of that value). Usueful for backwards compatibility: more fields can be easily added later.
  
  
• When upgrading gitlab, it (gitlab) had changed a repository named 'users' to 'users0'. Turns out 'users' is a reserved repository name in gitlab since version 8.15.
  
  To change your local git settings to the new users0 perform these steps to update your remote origin:
  
  # check current setting
  $ git remote -vorigin  https://gitlab.local/gitlab/backend/users (fetch)
  origin  https://gitlab.local/gitlab/backend/users (push)
  
  # change it to the new one
  $ git remote set-url origin https://gitlab.local/gitlab/backend/users0
  
  # see it got changed
  $ git remote -v
  origin  https://gitlab.local/gitlab/backend/users0 (fetch)
  origin  https://gitlab.local/gitlab/backend/users0 (push)
  
  
• Jackson JSON generating (serializing): probably a good practice is to not use @JsonInclude(JsonInclude.Include.NON_EMPTY)  or NON_NULLm since that would mean a key will be just not in the JSON when its value is empty or null. That could be confusing to the caller: sometimes it's there sometimes not.  So just leave it in, so it will be set to null.   Unless it would be a totally unrelated field like: amount and currency. If amount is null, currency (maybe) doesn't make sense, so then it could be left out.
  
  
• Java:
  
  Comparator userComparator = (o1, o2)->o1.getCreated().compareTo(o2.getCreated());
  
  can be replaced now in Java 8 by:
  
  Comparator userComparator = Comparator.comparing(UserByPhoneNumber::getCreated);
  
  
• Kafka partitioning tips: http://blog.rocana.com/kafkas-defaultpartitioner-and-byte-arrays
  
  
• Kafka vs RabbitMQ:
  
  - Kafka is optimized for producers producing lots of data (batch-oriented producers) and consumers that are usually slower that the producers.
  - Performance: Rabbit: makes about 20K/s  Kafka: up to 150K/s.
  - Unlike other message system, Kafka brokers are stateless. This means that the consumer has to maintain how much it has consumed.