Wednesday, December 9, 2015

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.255.235.17 (Timeout during read)

In a recent project, seemingly randomly, this exception occurred when doing a CQL 'select' statement from a Spring Boot project to Cassandra:

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.255.235.17 (Timeout during read), /10.255.235.16 (Timeout during read))
...


After a lot of research, some people seemed to have reported the same issue. But no clear answer anywhere. Except that some Cassandra driver versions might be the cause of it: they mark (all) the node(s) as down and don't recognize it when it becomes available again.

But, the strange this is we have over 10 (micro) services running, all running at with least 2 instances. But only one of these services had this timeout problem. So it almost couldn't be the driver.... Though it did seem to be related with not using the connection for a while, because often our end-to-end tests just ran fine, time after time. But after a few hours, the tests would just fail. Then we didn't see the pattern yet...

But, as a test, we decided to let nobody use the environment against which the end-to-end tests run for a few hours; especially also because some of the below articles do mention as a solution to set the heartbeat (keep-alive) of the driver.

And indeed, the end-to-end tests started failing again after the grace period. Then we realized it: all our services have a Spring Boot health-check implemented, which is called every X seconds. EXCEPT the service that has the timeouts; it only recently got connected with Cassandra!

After fixing that, the error disappeared! Of course depending on the healthcheck for a connection staying alive is not the ideal solution. A better solution is probably setting the heartbeat interval on the driver on Cluster creation:

var poolingOptions = new PoolingOptions()
  .SetCoreConnectionsPerHost(1)
  .SetHeartBeatInterval(10000);
var cluster = Cluster
  .Builder()
  .AddContactPoints(hosts).
  .WithPoolingOptions(poolingOptions)
  .Build();


In the end it was the firewall which resets all TCP connections every two hours!

References

Tips to analyse the problem:

Similar error reports



Wednesday, December 2, 2015

iOS9 / iOS 9 / iOS 9.1 / ATS 9: An SSL error has occurred and a secure connection to the server cannot be made due to old sha1 signed certificate

In iOS 9 and higher apps, a higher level of ciphers is required for a certificate for Forward Secrecy.

Before iOS 9, it was possible to let a site within a webview forward/redirect to another SSL protected site.
For example it was possible to let another site redirect to this one in iOS 8: https://www.securesuite.co.uk/

But since iOS 9 it is not allowed anymore and you'll get an error like:

An SSL error has occurred and a secure connection to the server cannot be made." UserInfo={NSUnderlyingError=0x7f9855dcb520 {Error Domain=kCFErrorDomainCFNetwork Code=-1200 "An SSL error has occurred and a secure connection to the server cannot be made." UserInfo={_kCFStreamErrorDomainKey=3, NSLocalizedRecoverySuggestion=Would you like to connect to the server anyway?, _kCFNetworkCFStreamSSLErrorOriginalValue=-9802, _kCFStreamPropertySSLClientCertificateState=0, NSLocalizedDescription=An SSL error has occurred and a secure connection to the server cannot be made., NSErrorFailingURLKey=https://www.securesuite.co.uk


You'll just see a blank/white screen in the webview; no errors or whatsoever on the screen.

The by-default supported list in iOS 9 and higher can be found here: https://developer.apple.com/library/prerelease/ios/documentation/General/Reference/InfoPlistKeyReference/Articles/CocoaKeys.html#//apple_ref/doc/uid/TP40009251-SW35

You can test easily whether a webpage/site is accepted at this site: https://www.ssllabs.com/ssltest/analyze.html

So when run for the above mentioned securesuite site, it tells us even in red the signature is still using the old SHA-1:


The site also even checks different clients like browsers and mobile operating systems like Android and iOS. And see the error in the Handshake Simulation section for ATS 9/ iOS 9: Client requires SHA2 certificate signatures

Running it against https://www.mastercard.com, which has SHA-2 as signature algorithm, the forwarding does work in iOS 8 and ios 9+: 
And the iOS 9 client also likes it:

To still be able to have iOS 9 and higher apps work with those less-secure sites which still use SHA-1, you can specify which domains are "ok-ish", i.e whitelist per domain. 
In the sections in https://developer.apple.com/library/prerelease/ios/documentation/General/Reference/InfoPlistKeyReference/Articles/CocoaKeys.html#//apple_ref/doc/uid/TP40009251-SW35 you can read how to whitelist: "Allowing Insecure Connection to a Single Server" and "Allowing Lowered Security to a Single Server" and "Using ATS For Your Servers and Allowing Insecure Connections Elsewhere".


Sunday, September 27, 2015

JBoss EAP 6.2.0 + Camel + CXF + JSF 2.2.8 Project Summary

Introduction

This blogpost is a short summary of a project I did a short while ago in 2014.


Tools used

  1. Oracle's jdk1.8.0_05 with target 7
  2. JBoss EAP 6.2.0
  3. Testng 6.8 (instead of JUnit)
  4. Camel 2.13
  5. CXF (for calls to external webservices)
  6. Less (CSS similar a bit to SASS)
  7. Spring 4.0
  8. Lombok
  9. Mockito 1.0
  10. Hamcrest 1.3
  11. DBUnit 2.5
  12. Embedded tomcat 7 for integration tests with maven 3
  13. Sonar, Confluence, Sourcetree (for Git over SVN)
  14. Spring-rabbit 1.2 for accessing RabbitMQ
  15. JSF 2.2.8
  16. Hibernate 5.1.1
  17. Hsqldb 2.3

Lessons learned

  1. My first time use of the Maven plugin dependency which gives warnings of undeclared and used or unused declared dependencies:

    mvn dependency:analyze-only

    Very handy. You *can't* trust on analyze-only though! In the end you still need to perform a full build to make sure all still works. The plugin can only determine dependencies.

    It is possible that some "hidden" non-explicit dependency on an @Entity class creates a FK, which causes the project (with the missing explicit ) hbm.ddl.auto=create not be able to drop a table because it doesn't know about that FK (and thus can't drop that and thus not the table).

    Luckily in log (perhaps you need to set the hibernate level to log all queries/statements) you can see what exactly is being dropped and thus can you spot the missing 'drop contraint xyz' line, where xyz is the constraint the current project doesn't know about. Add it as to the project's pom.xml and the statement should appear. Note that mvn dependency:analyze-only will complain about it as 'unused declared dependency'. Probably this FK dependency is an incorrect project setup anyway.


  2. Sonar's //NOSONAR does not seem to ignore code-coverage violations... Only issues (e.g the Blocker, Critical etc rules).

  3. Camel 2.13: example expression to only have file moved if same file prefix file but with extension .ready already exists:

    // Reference: http://camel.apache.org/file-language.html
    private static final String FILE_READY_TO_BE_PICKED_UP_EXPRESSION = "doneFileName=$simple{file:name.noext}.ready";


  4. Parsing in jodatime:

    private final DateTimeParser[] parsers = {
        DateTimeFormat.forPattern(SOME_DATE_FORMAT).getParser()
    };

    DateTimeFormatter formatter = new DateTimeFormatterBuilder().append(null, parsers).toFormatter().withZone(DateTimeZone.UTC);

    And the other direction:

    String nowText = DateTime.now().toString(SOME_DATE_FORMAT);

  5. Suppose you have two maven profiles A and B. When running mvn clean install -PA,B it appeared only the <exclude> of the last profile was being used (as seen when adding the -X parameter when running the mvn command).

    Problem was that both profiles had the same <id> in the <execution> part!

    Solution: prefix the execution ids to make them unique. E.g:

    <id>A-tests</id>
    and
    <id>B-tests</id>

  6. TestNG (and JUnit vx.y in a bit different way but same mechanism) supports groups to divide tests in for example unit vs integration tests.
    But what if you have a combination of previous unittests which don't have that annotation yet, and you don't want to have to update all tests classes?
    Because if you have specified in your root pom.xml, then the tests w/o the groups attribute in the @Test annotation  won't get picked up by the maven-sure-fire plugin.

    For that you need this specified with your maven-surefire-plugin configuration:

    <plugin>
    ...
    <configuration>
        <!-- Need to specify an empty variable. Leaving the tags empty does not make it overrule the defined in the inmotiv-root/pom.xml -->
        <groups>${emptyProperty}</groups>
    </configuration>
    ...
    </plugin>

    Note the ${emptyProperty} property. It has *no* value set!

    Would you remove the ${emptyProperty}, so you have <groups><groups/>, or replace the line with <groups/> you'll see the test class won't get run! My guess is that maven just removes the empty tag, but won't if you put in a variable...

  7. When renaming files by change the casing (uppercase or lowercase) that are already git controlled, use

    git mv -f FILE file
    otherwise GIT won't see it.




Saturday, January 31, 2015

Recreate Vagrant box when original .box is gone from vagrantcloud

Introduction

Suppose you've setup a cool Vagrant image based upon a box from the Vagrant Hashicorp cloud. But suddenly one day the URL is gone! So the original box is gone. E.g this one, which gives a 404 page: https://atlas.hashicorp.com/box-cutter/boxes/ubuntu1404-desktop
Your 'vagrant up' still works, vagrant will just complain it can't find the box anymore, but the VirtualBox image will start up fine.

But you are still getting new team-members on board, and you want them to use the Vagrant setup you created; but the URL is gone so the base box can't be downloaded anymore. What to do?
Well, it turns out you can recreate the .box file which the above URL is pointing to.
This post describes how.

Steps

  1. Locate where the Vagrant box from the URL is stored on your local machine. On Windows 7 it is C:/Users/ttlnews/.vagrant.d/boxes 
    At this location are all the base Vagrant boxes downloaded. So without any changes/provisioning made to them. The exact directory of the above ubuntu1404-desktop URL is: C:/Users/ttlnews/.vagrant.d/boxes/box-cutter-VAGRANTSLASH-ubuntu1404-desktop

  2. Create a zip of the deepest directory in there. In my case it I 7zipped all the files in C:/Users/ttlnews/.vagrant.d/boxes/box-cutter-VAGRANTSLASH-ubuntu1404-desktop/1.0.11/virtualbox So not the directory, just the files, like the box.ovf, metadata.json, *.vmdk, Vagrantfile. The resulting file was: box-cutter-VAGRANTSLASH-ubuntu1404-desktop.box.7z

    This is what it should look like in the .7z file:


  3. Rename the file to box-cutter-VAGRANTSLASH-ubuntu1404-desktop.box.

  4. Put it in c:/tmp or wherever you want; make sure to change accordingly in the below steps.

  5. In the directory where you have your (already existing because you have created it successfully before from the now-defunct box URL) Vagrantfile add the following line below config.vm.box:

    config.vm.box_url = "file:///c:/tmp/box-cutter-VAGRANTSLASH-ubuntu1404-desktop.box"

    Tip 1: when testing this out, it's safer to copy your Vagrantfile + whatever else you had to a new directory, so you won't mess up your working Vagrant setup.

    Tip 2: in that new setup from tip 1, also change the value of config.vm.box in the Vagrantfile, so you can clearly identify this new image.

  6. Run 'vagrant up' and you should see it copy (extract) the files, start a VirtualBox and start provisioning the box!
Main source for this solution: https://docs.vagrantup.com/v2/boxes/format.html