Lessons learned Docker microservices architecture with Spring Boot

Introduction

During my last project consisting of a Docker microservices architecture, built with Spring Boot, using RabbitMQ as communication channel, I learned a bunch of lessons, here's a summary of them.

Architecture

Below is a high level overview of the architecture that was used.

Docker

• Run 1 process/service/application per docker container (or put stuff in init.d but that's not intended use of docker)


• Starting background processes in the CMD cause container to exit. So either have a script waiting at the end (e.g tail -f /dev/null) or keep the process (i.e the one prefixed with CMD) running in the foreground. Other useful Dockerfile tips you can find here


• As far as I can tell Docker checks if Dockerfile has changed, and if so, creates a new image instance (diffs only?)


• Basic example to start RabbitMq docker image, as used in the build tool:
  
  $ docker pull 172.18.19.20/project/rabbitmq:latest
  $ docker rm -f build-server-rabbitmq
  $ # Map the RabbitMQ regular and console ports
  $ docker run -d -p 5672:5672 -p 15672:15672 --name build-server-rabbitmq 172.18.19.20/rabbitmq:latest


• If there's no docker0 interface (check by running command ifconfig) then probably there are ^M characters in the config file at /etc/default/docker/docker.config. To fix it, perform a dos2unix on that file.


• Check for errors at startup of docker in /var/log/upstart/docker.log


• If your docker push <image> asks for a login (and you don't expect that) or it returns some weird html like "</html>" then you're probably missing the host in front of the image name, e.g: 172.18.19.20:6000/projectname/some-service:latest


• Stuff like /var/log/messages is not visible in a Docker container, but is in its host! So look there for example to find out why a process is not starting/gets killed at startup without any useful logging (like we had with clamd)


• How to remove old dangling unused docker images: docker rmi $(docker images --filter "dangling=true" -q --no-trunc)

Spring Boot

• Some jackson2 dependencies were missing from the generated Spring Initializr project, noticed when creating unittests. These dependencies were additionally needed in scope test:
  
  <dependency>
    <groupid>com.fasterxml.jackson.core</groupid>
    <artifactid>jackson-databind</artifactid>
    <version>2.5.0</version></dependency>
  <dependency>
    <groupid>com.fasterxml.jackson.core</groupid>
    <artifactid>jackson-annotations</artifactid>
    <version>2.5.0</version></dependency>
  <dependency>
    <groupid>com.fasterxml.jackson.core</groupid>
    <artifactid>jackson-core</artifactid>
    <version>2.5.0</version></dependency>
  
  Not sure anymore why these then didn't get <scope>test</scope> then... Guess it was needed also in some regular code... :)


• In the Spring Boot AMQP Quick Start the last param has name .with(queueName) during binding, but that's the topic key! (which is related to the binding key used at sending), so not the queue name.


• Spring Boot Actuator's /health will check all related dependencies! So if you have a dependency in your pom.xml to a project which uses spring-boot-starter-amqp, /health will check now for an AMQP queue being up! So add a for those if you don't want that.


• Spring Boot's default AppAplicationIT probably needs a @DirtiesContext for your tests, otherwise the tests might re-use or create more beans than you think (we saw that in our message receiver tests helper class).


• @Transactional in Spring: by default only for unchecked exceptions!! It's documented but still a thing to watch out for.


• And of course: Spring's @Transactional does not work on private methods (due to proxy stuff it creates)


• To see in Spring Boot the transaction logging, put this in application.properties:
  
  logging.level.org.springframework.jdbc=TRACE
  logging.level.org.springframework.transaction=TRACE
  
  Note that by default @Transactional just rolls back, it does not log anything, so if you don't log your runtime exceptions, you won't see much in your logs.


• mockMvc from spring is not really invoking from "outside", our spring sec context filter (for which you can use @Secured(role)) was allowing calls while no authentication was provided for. RestTemplate seems to work from "the outside".


• Scan order can mess up @ControllerAdvice error handler it seems. Had to change the order sometimes:
  
  Setup:
  - Controller is in: com.company.request.web.
  - General error controller is in com.company.common package.
  
  Had to change
  @ComponentScan(value = {"com.company.security", "com.company.common", "com.company.cassandra", "com.company.module", "com.company.request"})
  
  to
  
  @ComponentScan(value = {"com.company.security", "com.company.cassandra", "com.company.module", "com.company.request", "com.company.common"})
  
  Note that the general error controller has now been put in last


• Spring Boot footprint seems relatively big especially for microservices. At least 500MB or something is needed, so we have quite big machines for about 20 services. Maybe plain Spring (iso Spring boot) might be more lightweight...

Bamboo build server

• When Bamboo gets slow and the CPU seems quite busy and memory availability on its server seems fine, increase the Xmss and Xmsx (or related). Found this out because the java Bamboo process was running out of heap sometimes, increasing heap also fixed performance.


• To have Bamboo builds fail on quality gates not met in SonarQube, install in Sonar the build breaker plugin. See the plugin docs and Update Center. This FAQ says so.

Stash

• The Stash (now called Bitbucket) API: in /rest/git/1.0/projects/{projectKey}/repos/{repositorySlug}/tags a 'slug' is just a repository name. 

Microservices with event based architecture

• When you do microservices, IMMEDIATELY take into account during coding + reviews that multiple instances can do concurrent access to database.
  
  This has affect on your queries. Most likely correct implementation for uniqueness check on inserts:
  1- add unique constraint
  2- run insert
  3- catch uniqueness exception --> you know it already exists. Solution with SELECT NOT EXISTS is not guaranteed unique.


• Also take deleting of data (e.g user deletes himself) into account from the start. Especially when using events and/or eventual consistency in combination with an account-balance or similar. Because what if one services in the whole chain of things to execute for a delete fails? Has the user still some money left on his/her account then? In short: take care of CRUD.


• Multiple services are sending the same event? That can indicate 2 services are doing the same thing --> Not good probably.


• Microservices advantages:
  
  - Forces you to better think about where to put stuff in comparison to monolith where you more often can be tempted to "just do a quick fix".
  - language independency for service implementation: choose the best language for the job
  
  Disadvantages:
  - more time needed for design
  - eventual consistency is quite tough to understand & work with, also conceptually
  - infrastructure is more complex including all communication between services
  
  More cons can be found here.

Tomcat

• Limit the maximum size of what can be posted to a servlet is not as easy as it seems for REST services:
  
  - maxPostSize in Tomcat is enforced only for specific contenttype: Tomcat only enforces that limit if the content type is application/x-www-form-urlencoded
  
  - And the other 3 below XML options are for multipart only:
  
  <multipart-config>
    <!-- 52MB max -->
    <max-file-size>52428800</max-file-size>
    <max-request-size>52428800</max-request-size>
    <file-size-threshold>0</file-size-threshold></multipart-config>
  
  So that one won't work for uploading just a byte[]. The only solution is in the servlet (e.g Spring @Controller) you'll have to check for the limit you want to allow.


• maxthreads seems set to be unlimited by default or something. 50 seems to perform better. (workerthreads) 

Security

• To securely generate a random number: SecureRandom randomGenerator = SecureRandom.getInstance("NativePRNG");


• Good explanation of secure use of a salt to use for hashing can be found here. 

Cassandra

• Unique constraints are not possible in Cassandra, so there you will even have to implement unique constraints in the business logic (and make it eventually consistent)


• CassandraOperations query for one field:
  
  Select select = QueryBuilder.select(MultiplePaymentRequestRequesterEntityKey.ID).from(MultiplePaymentRequestRequesterEntity.TABLE_NAME);
  select.setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM);
  select.where(QueryBuilder.eq(MultiplePaymentRequestRequesterEntityKey.REQUESTER, requester));
  return cassandraTemplate.queryForList(select, UUID.class);
  
  See also here.


• Note below two keys don't seem to get picked up by Cassandra in the Spring Data Cassandra version 1.1.4.RELEASE:  
  
  <groupid>org.springframework.data</groupid>
  <artifactid>spring-data-cassandra</artifactid>
  
  @PrimaryKeyColumn(name = OTHER_USER_ID, ordinal = 0, type = PrimaryKeyType.PARTITIONED)
  @CassandraType(type = DataType.Name.UUID)
  private UUID meUserId;
  
  @PrimaryKeyColumn(name = ME_USER_ID, ordinal = 1, type = PrimaryKeyType.CLUSTERED)
  @CassandraType(type = DataType.Name.UUID)
  private UUID meId;
  This *does* get picked up: put it into a separate class:
  
  @Data
  @AllArgsConstructor
  @PrimaryKeyClass
  public class HistoryKey implements Serializable {
  
    @PrimaryKeyColumn(name = HistoryEntity.ME_USER_ID, ordinal = 0, type = PrimaryKeyType.PARTITIONED)
    @CassandraType(type = DataType.Name.UUID)
    private UUID meUserId;
  
    @PrimaryKeyColumn(name = HistoryEntity.OTHER_USER_ID, ordinal = 1, type = PrimaryKeyType.PARTITIONED)
    @CassandraType(type = DataType.Name.UUID)
    private UUID otherUserId;
  
    @PrimaryKeyColumn(name = HistoryEntity.CREATED, ordinal = 2, type = PrimaryKeyType.CLUSTERED, ordering = Ordering.DESCENDING)
    private Date created;
  
  }


• Don't use Cassandra for all types of use cases. An RDMS still has its value, e.g for ACID requirements. Cassandra is eventually consistent.

Kubernetes

• See my diagram in this post.

Miscellaneous

• Use dig for DNS resolving problems


• Use pgAdmin III for PostgreSQL GUI


• To stop SonarQube complaining about unused private fields when using Lombok @Data annotation: add to each of those classes @SuppressWarnings("PMD.UnusedPrivateField")


• Managed to not need transactions nor XA transactions for message publishing, message reading, store db, message sending, by using the confirm + ack mechanism.
  And allow message to be read again. DB then sees: oh already stored (or do an upsert).
  So,when processing message from the queue:
  1- store in db
  2- send message on queue
  3- only then ack back to queue that read was successful


• Performance: instantiate the Jackson2 ObjectMapper once as static, not in each call, so:
  

  private static final ObjectMapper mapper = new ObjectMapper();

• Javascript: when an exception occurs in a callback and it is not handled, processing just ends. Promises have better error handling.


• clamd would not start correctly; it would try to start but then show 'Killed' when started via the commandline. Turns out it runs out of memory when starting up.  Though we had enough RAM (16G total, 3G free), it turns out clamd needs swap configured!


• Linux bash shell script to loop through projects for tagging with projects with spaces in their name:
  
  PROJECTS="
    project1
    project space2
  ";
  IFS=$'\n'
  for PROJECT in $PROJECTS
  do
    TRIM_LEADING_SPACE_PROJECT="$(echo -e "${PROJECT}" | sed -e 's/^[[:space:]]*//')"
    echo "Cloning '$TRIM_LEADING_SPACE_PROJECT'"
    git clone --depth=1 http://$USER:$GITPASSWD@github.com/projects/$TRIM_LEADING_SPACE_PROJECT.git
  done


• OpenVPN in Windows 10: Sometimes it hangs on "Connecting..."  It doesn't show the popup to enter username/pwd. Go to View logs. Then when you see: Enter management password in the logs: ???? you have to kill the OpenVPN Daemon under Processes tab (windows taskmanager). The service is stopped when exiting the app but that's not enough!


• javascript/nodejs log every call that comes in:
  
  app.use(function (req, res, next) {
    console.log('Incoming request = ' + new Date(), req.method, req.url);
    logger.debug('log at debug level');
    next()
  }


• If ever your mouse is suddenly not working anymore your VirtualBox guest, kill the process in your guest-machine mentioned in comment 5 here. After that the mouse works again in your vbox guest.


• Fix Firefox to version 45.0.0 for selenium driver tests:
  
  sudo apt-get install -y firefox=45.0.2+build1-0ubuntu1
  sudo apt-mark hold firefox


• Setting the cookie attribute Secure (indicating cookie should only be sent over httpS) can be seen when using curl to request the URL(s) that should send that cookie plus the new attribute, even when using HTTP. See also my previous post.
  
  But when using a browser and HTTP, you probably won't see the secure cookie appear in the cookie store. This is (probably) because the browser knows not to store it in that case because it's HTTP being used.


• Idempotency within services is key for resilience and be able to resend an event or perform an API call again.

JBoss EAP 6.2.0 + Camel + CXF + JSF 2.2.8 Project Summary

Introduction

This blogpost is a short summary of a project I did a short while ago in 2014.

Tools used

1. Oracle's jdk1.8.0_05 with target 7
   
2. JBoss EAP 6.2.0
   
3. Testng 6.8 (instead of JUnit)
   
4. Camel 2.13
   
5. CXF (for calls to external webservices)
   
6. Less (CSS similar a bit to SASS)
   
7. Spring 4.0
   
8. Lombok
   
9. Mockito 1.0
   
10. Hamcrest 1.3
    
11. DBUnit 2.5
    
12. Embedded tomcat 7 for integration tests with maven 3
    
13. Sonar, Confluence, Sourcetree (for Git over SVN)
    
14. Spring-rabbit 1.2 for accessing RabbitMQ
    
15. JSF 2.2.8
    
16. Hibernate 5.1.1
    
17. Hsqldb 2.3

Lessons learned

1. My first time use of the Maven plugin dependency which gives warnings of undeclared and used or unused declared dependencies:
   
   mvn dependency:analyze-only
   
   Very handy. You *can't* trust on analyze-only though! In the end you still need to perform a full build to make sure all still works. The plugin can only determine dependencies.
   
   It is possible that some "hidden" non-explicit dependency on an @Entity class creates a FK, which causes the project (with the missing explicit ) hbm.ddl.auto=create not be able to drop a table because it doesn't know about that FK (and thus can't drop that and thus not the table).
   
   Luckily in log (perhaps you need to set the hibernate level to log all queries/statements) you can see what exactly is being dropped and thus can you spot the missing 'drop contraint xyz' line, where xyz is the constraint the current project doesn't know about. Add it as to the project's pom.xml and the statement should appear. Note that mvn dependency:analyze-only will complain about it as 'unused declared dependency'. Probably this FK dependency is an incorrect project setup anyway.
   
   
   
2. Sonar's //NOSONAR does not seem to ignore code-coverage violations... Only issues (e.g the Blocker, Critical etc rules).
   
   
3. Camel 2.13: example expression to only have file moved if same file prefix file but with extension .ready already exists:
   
   // Reference: http://camel.apache.org/file-language.html
   private static final String FILE_READY_TO_BE_PICKED_UP_EXPRESSION = "doneFileName=$simple{file:name.noext}.ready";
   
   
4. Parsing in jodatime:
   
   private final DateTimeParser[] parsers = {
       DateTimeFormat.forPattern(SOME_DATE_FORMAT).getParser()
   };
   
   DateTimeFormatter formatter = new DateTimeFormatterBuilder().append(null, parsers).toFormatter().withZone(DateTimeZone.UTC);
   And the other direction:
   
   String nowText = DateTime.now().toString(SOME_DATE_FORMAT);
   
   
5. Suppose you have two maven profiles A and B. When running mvn clean install -PA,B it appeared only the <exclude> of the last profile was being used (as seen when adding the -X parameter when running the mvn command).
   
   Problem was that both profiles had the same <id> in the <execution> part!
   
   Solution: prefix the execution ids to make them unique. E.g:
   
   <id>A-tests</id>
   and
   <id>B-tests</id>
   
   
6. TestNG (and JUnit vx.y in a bit different way but same mechanism) supports groups to divide tests in for example unit vs integration tests.
   But what if you have a combination of previous unittests which don't have that annotation yet, and you don't want to have to update all tests classes?
   Because if you have specified in your root pom.xml, then the tests w/o the groups attribute in the @Test annotation  won't get picked up by the maven-sure-fire plugin.
   
   For that you need this specified with your maven-surefire-plugin configuration:
   
   <plugin>
   ...
   <configuration>
       <!-- Need to specify an empty variable. Leaving the tags empty does not make it overrule the defined in the inmotiv-root/pom.xml -->
       <groups>${emptyProperty}</groups>
   </configuration>
   ...
   </plugin>
   Note the ${emptyProperty} property. It has *no* value set!
   
   Would you remove the ${emptyProperty}, so you have <groups><groups/>, or replace the line with <groups/> you'll see the test class won't get run! My guess is that maven just removes the empty tag, but won't if you put in a variable...
   
   
7. When renaming files by change the casing (uppercase or lowercase) that are already git controlled, use
   
   git mv -f FILE file
   otherwise GIT won't see it.

Best of this Week Summary 19 July - 25 July 2010

• A short introduction to using the open source project Jericho to parse HTML. Jericho includes text extraction from HTML markup, rendering, formatting and compacting HTML.
  


• A nice list of both strategic and technical factors you should consider when making decisions about integration in the blogpost EAI: when tools can help.
  


• The BBC used interesting semantic technologies on their soccer Worldcup 2010 website. See the architecture below:
  
  


• Pretty basic article on using DateFormat in a multi-threading environment. Tricky because they are not synchronized. You should definitely also consider Joda Time and the new Date and Time APIs for Java 7 (JSR-310).
  


• An explanation of how Sonar feedback metrics can be integrated into the code/test/commit/build cycle for a quality feedback loop.
  

Best of this Week Summary 1 March - 7 March 2010

• Of course is the contract between webservices more important than their internals.
  


• How the static analysis tool Coverity was turned into a commercial product. Includes caveats, lessons learned etc.
  Check also how you can measure code quality with Sonar.
  


• The updated 2010 list of most dangerous programming errors.
  


• Four presentations from John Resig on jQuery, and especially Query 1.4.
  

Best of this Week Summary 25 May - 31 May 2009

• Facebook is now also supporting registration/login via a GMail account and OpenID, see the image below. I especially like that they've implemented it with a lightbox ("popup") so the user doesn't get as much confused anymore, as was the case in the old/standard implementation where the user is completely redirected to Google or the OpenID provider's website. Note that it is actually more "Facebook Connect" like this way! See here another example where OpenID is combined with OAuth to enable a popup login.
  
  
  
• Understanding how the JVM uses native memory on Windows and Linux. The extensive article explains what native memory is, how the Java runtime uses it, what running out of it looks like (so you're not running out of heap space!), and how to debug a native OutOfMemoryError on Windows and Linux. A companion article covers the same topics for AIX systems.
  


• Quite big news was of course Google's announcement of Google Wave at the Google I/O conference. It has been built with GWT. A good description can be found here. It's open source with plugin-like APIs with many integration possibilities. See the 80 minutes video for the full details. It hopes to become the replacement for email... Servers can be run by anybody. Wonder how Google is thinking of making money with it. Ads, just like in GMail? Maybe they are going to charge you for using their Wave server instances (SAAS version)? An interview with Wave's creators can be found here. And six reasons why Wave could be game-changing.
  


• Eight generic best practices for scalable high performance systems.
  


• Are you any of these two tools with almnost the exact same name? SonarJ is a plug-in for Eclipse that helps you validate your code against a software architecture, using static analysis (free for projects up to 500 classes). And now for the confusion: check also Sonar: enables to collect, analyze and report metrics on source code. It leverages the existing ecosystem of quality open source tools (ex. Checkstyle, PMD, Maven, Cobertura …), to offer a fully integrated solution to development environments and continuous integration tools.