Wednesday, February 10, 2016

Tomcat 7 request processing threading architecture and performance tuning

Tomcat has many parameters for performance tuning (see http://tomcat.apache.org/tomcat-7.0-doc/config/http.html), but for some attribute descriptions it is not a 100% clear how some properties affect each other.

A pretty good explanation regarding acceptCount and maxThreads can be found here: http://techblog.netflix.com/2015/07/tuning-tomcat-for-high-throughput-fail.html 
But that article is missing a ... picture! That's what I tried to create here, with the numbers 1-5 indicating the steps described in the above article:



Just in case should the Netflix URL ever get lost and for easy reference, here's the 5 steps:




The Tomcat attribute maxThreads worked best in my case with value 50, as it won't saturate the machine/CPU. (due to too many workerthreads, many context-switches)


To set maxThreads when using Spring Boot + embedded Tomcat: http://stackoverflow.com/questions/31432514/how-to-modify-tomcat8-acceptcount-in-spring-boot


Other remarks


Referenced material




Wednesday, December 9, 2015

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.255.235.17 (Timeout during read)

In a recent project, seemingly randomly, this exception occurred when doing a CQL 'select' statement from a Spring Boot project to Cassandra:

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.255.235.17 (Timeout during read), /10.255.235.16 (Timeout during read))
...


After a lot of research, some people seemed to have reported the same issue. But no clear answer anywhere. Except that some Cassandra driver versions might be the cause of it: they mark (all) the node(s) as down and don't recognize it when it becomes available again.

But, the strange this is we have over 10 (micro) services running, all running at with least 2 instances. But only one of these services had this timeout problem. So it almost couldn't be the driver.... Though it did seem to be related with not using the connection for a while, because often our end-to-end tests just ran fine, time after time. But after a few hours, the tests would just fail. Then we didn't see the pattern yet...

But, as a test, we decided to let nobody use the environment against which the end-to-end tests run for a few hours; especially also because some of the below articles do mention as a solution to set the heartbeat (keep-alive) of the driver.

And indeed, the end-to-end tests started failing again after the grace period. Then we realized it: all our services have a Spring Boot health-check implemented, which is called every X seconds. EXCEPT the service that has the timeouts; it only recently got connected with Cassandra!

After fixing that, the error disappeared! Of course depending on the healthcheck for a connection staying alive is not the ideal solution. A better solution is probably setting the heartbeat interval on the driver on Cluster creation:

var poolingOptions = new PoolingOptions()
  .SetCoreConnectionsPerHost(1)
  .SetHeartBeatInterval(10000);
var cluster = Cluster
  .Builder()
  .AddContactPoints(hosts).
  .WithPoolingOptions(poolingOptions)
  .Build();


In the end it was the firewall which resets all TCP connections every two hours!

References

Tips to analyse the problem:

Similar error reports



Wednesday, December 2, 2015

iOS9 / iOS 9 / iOS 9.1 / ATS 9: An SSL error has occurred and a secure connection to the server cannot be made due to old sha1 signed certificate

In iOS 9 and higher apps, a higher level of ciphers is required for a certificate for Forward Secrecy.

Before iOS 9, it was possible to let a site within a webview forward/redirect to another SSL protected site.
For example it was possible to let another site redirect to this one in iOS 8: https://www.securesuite.co.uk/

But since iOS 9 it is not allowed anymore and you'll get an error like:

An SSL error has occurred and a secure connection to the server cannot be made." UserInfo={NSUnderlyingError=0x7f9855dcb520 {Error Domain=kCFErrorDomainCFNetwork Code=-1200 "An SSL error has occurred and a secure connection to the server cannot be made." UserInfo={_kCFStreamErrorDomainKey=3, NSLocalizedRecoverySuggestion=Would you like to connect to the server anyway?, _kCFNetworkCFStreamSSLErrorOriginalValue=-9802, _kCFStreamPropertySSLClientCertificateState=0, NSLocalizedDescription=An SSL error has occurred and a secure connection to the server cannot be made., NSErrorFailingURLKey=https://www.securesuite.co.uk


You'll just see a blank/white screen in the webview; no errors or whatsoever on the screen.

The by-default supported list in iOS 9 and higher can be found here: https://developer.apple.com/library/prerelease/ios/documentation/General/Reference/InfoPlistKeyReference/Articles/CocoaKeys.html#//apple_ref/doc/uid/TP40009251-SW35

You can test easily whether a webpage/site is accepted at this site: https://www.ssllabs.com/ssltest/analyze.html

So when run for the above mentioned securesuite site, it tells us even in red the signature is still using the old SHA-1:


The site also even checks different clients like browsers and mobile operating systems like Android and iOS. And see the error in the Handshake Simulation section for ATS 9/ iOS 9: Client requires SHA2 certificate signatures

Running it against https://www.mastercard.com, which has SHA-2 as signature algorithm, the forwarding does work in iOS 8 and ios 9+: 
And the iOS 9 client also likes it:

To still be able to have iOS 9 and higher apps work with those less-secure sites which still use SHA-1, you can specify which domains are "ok-ish", i.e whitelist per domain. 
In the sections in https://developer.apple.com/library/prerelease/ios/documentation/General/Reference/InfoPlistKeyReference/Articles/CocoaKeys.html#//apple_ref/doc/uid/TP40009251-SW35 you can read how to whitelist: "Allowing Insecure Connection to a Single Server" and "Allowing Lowered Security to a Single Server" and "Using ATS For Your Servers and Allowing Insecure Connections Elsewhere".


Sunday, September 27, 2015

JBoss EAP 6.2.0 + Camel + CXF + JSF 2.2.8 Project Summary

Introduction

This blogpost is a short summary of a project I did a short while ago in 2014.


Tools used

  1. Oracle's jdk1.8.0_05 with target 7
  2. JBoss EAP 6.2.0
  3. Testng 6.8 (instead of JUnit)
  4. Camel 2.13
  5. CXF (for calls to external webservices)
  6. Less (CSS similar a bit to SASS)
  7. Spring 4.0
  8. Lombok
  9. Mockito 1.0
  10. Hamcrest 1.3
  11. DBUnit 2.5
  12. Embedded tomcat 7 for integration tests with maven 3
  13. Sonar, Confluence, Sourcetree (for Git over SVN)
  14. Spring-rabbit 1.2 for accessing RabbitMQ
  15. JSF 2.2.8
  16. Hibernate 5.1.1
  17. Hsqldb 2.3

Lessons learned

  1. My first time use of the Maven plugin dependency which gives warnings of undeclared and used or unused declared dependencies:

    mvn dependency:analyze-only

    Very handy. You *can't* trust on analyze-only though! In the end you still need to perform a full build to make sure all still works. The plugin can only determine dependencies.

    It is possible that some "hidden" non-explicit dependency on an @Entity class creates a FK, which causes the project (with the missing explicit ) hbm.ddl.auto=create not be able to drop a table because it doesn't know about that FK (and thus can't drop that and thus not the table).

    Luckily in log (perhaps you need to set the hibernate level to log all queries/statements) you can see what exactly is being dropped and thus can you spot the missing 'drop contraint xyz' line, where xyz is the constraint the current project doesn't know about. Add it as to the project's pom.xml and the statement should appear. Note that mvn dependency:analyze-only will complain about it as 'unused declared dependency'. Probably this FK dependency is an incorrect project setup anyway.


  2. Sonar's //NOSONAR does not seem to ignore code-coverage violations... Only issues (e.g the Blocker, Critical etc rules).

  3. Camel 2.13: example expression to only have file moved if same file prefix file but with extension .ready already exists:

    // Reference: http://camel.apache.org/file-language.html
    private static final String FILE_READY_TO_BE_PICKED_UP_EXPRESSION = "doneFileName=$simple{file:name.noext}.ready";


  4. Parsing in jodatime:

    private final DateTimeParser[] parsers = {
        DateTimeFormat.forPattern(SOME_DATE_FORMAT).getParser()
    };

    DateTimeFormatter formatter = new DateTimeFormatterBuilder().append(null, parsers).toFormatter().withZone(DateTimeZone.UTC);

    And the other direction:

    String nowText = DateTime.now().toString(SOME_DATE_FORMAT);

  5. Suppose you have two maven profiles A and B. When running mvn clean install -PA,B it appeared only the <exclude> of the last profile was being used (as seen when adding the -X parameter when running the mvn command).

    Problem was that both profiles had the same <id> in the <execution> part!

    Solution: prefix the execution ids to make them unique. E.g:

    <id>A-tests</id>
    and
    <id>B-tests</id>

  6. TestNG (and JUnit vx.y in a bit different way but same mechanism) supports groups to divide tests in for example unit vs integration tests.
    But what if you have a combination of previous unittests which don't have that annotation yet, and you don't want to have to update all tests classes?
    Because if you have specified in your root pom.xml, then the tests w/o the groups attribute in the @Test annotation  won't get picked up by the maven-sure-fire plugin.

    For that you need this specified with your maven-surefire-plugin configuration:

    <plugin>
    ...
    <configuration>
        <!-- Need to specify an empty variable. Leaving the tags empty does not make it overrule the defined in the inmotiv-root/pom.xml -->
        <groups>${emptyProperty}</groups>
    </configuration>
    ...
    </plugin>

    Note the ${emptyProperty} property. It has *no* value set!

    Would you remove the ${emptyProperty}, so you have <groups><groups/>, or replace the line with <groups/> you'll see the test class won't get run! My guess is that maven just removes the empty tag, but won't if you put in a variable...

  7. When renaming files by change the casing (uppercase or lowercase) that are already git controlled, use

    git mv -f FILE file
    otherwise GIT won't see it.




Saturday, January 31, 2015

Recreate Vagrant box when original .box is gone from vagrantcloud

Introduction

Suppose you've setup a cool Vagrant image based upon a box from the Vagrant Hashicorp cloud. But suddenly one day the URL is gone! So the original box is gone. E.g this one, which gives a 404 page: https://atlas.hashicorp.com/box-cutter/boxes/ubuntu1404-desktop
Your 'vagrant up' still works, vagrant will just complain it can't find the box anymore, but the VirtualBox image will start up fine.

But you are still getting new team-members on board, and you want them to use the Vagrant setup you created; but the URL is gone so the base box can't be downloaded anymore. What to do?
Well, it turns out you can recreate the .box file which the above URL is pointing to.
This post describes how.

Steps

  1. Locate where the Vagrant box from the URL is stored on your local machine. On Windows 7 it is C:/Users/ttlnews/.vagrant.d/boxes 
    At this location are all the base Vagrant boxes downloaded. So without any changes/provisioning made to them. The exact directory of the above ubuntu1404-desktop URL is: C:/Users/ttlnews/.vagrant.d/boxes/box-cutter-VAGRANTSLASH-ubuntu1404-desktop

  2. Create a zip of the deepest directory in there. In my case it I 7zipped all the files in C:/Users/ttlnews/.vagrant.d/boxes/box-cutter-VAGRANTSLASH-ubuntu1404-desktop/1.0.11/virtualbox So not the directory, just the files, like the box.ovf, metadata.json, *.vmdk, Vagrantfile. The resulting file was: box-cutter-VAGRANTSLASH-ubuntu1404-desktop.box.7z

    This is what it should look like in the .7z file:


  3. Rename the file to box-cutter-VAGRANTSLASH-ubuntu1404-desktop.box.

  4. Put it in c:/tmp or wherever you want; make sure to change accordingly in the below steps.

  5. In the directory where you have your (already existing because you have created it successfully before from the now-defunct box URL) Vagrantfile add the following line below config.vm.box:

    config.vm.box_url = "file:///c:/tmp/box-cutter-VAGRANTSLASH-ubuntu1404-desktop.box"

    Tip 1: when testing this out, it's safer to copy your Vagrantfile + whatever else you had to a new directory, so you won't mess up your working Vagrant setup.

    Tip 2: in that new setup from tip 1, also change the value of config.vm.box in the Vagrantfile, so you can clearly identify this new image.

  6. Run 'vagrant up' and you should see it copy (extract) the files, start a VirtualBox and start provisioning the box!
Main source for this solution: https://docs.vagrantup.com/v2/boxes/format.html



Thursday, November 20, 2014

JasperReports/iReports dynamically show and hide fields conditionally

Introduction

With JasperReports and its visual designer iReports you can have (blocks of) fields collapse based on certain conditions.
You can use a subreport, but that's not necessary. This post describes how to show/hide fields without subreports.

How

Basically you have to do two things:

  1. Put the elements (static text, textfields etc) you want to hide on a frame. Make sure the fields are not behind the frame.
  2. Set the correct properties on the frame, including the expression to show/hide it dynamically. By hiding the frame you hide all the fields on it. 
The most important thing is to make sure no other element besides those that need to be on it is overlapping with the frame! Below is a screenshot to make it clearer.
There are three static and three textfields on the frame. Plus two (separate!) vertical lines.


Note the green, red and black vertical lines in the screenshot: the red and black vertical lines are not on the frame at all. The frame has its own green vertical line. That way JasperReports can shrink it based on the condition. Otherwise it doesn't know what to do with the overlap, and doesn't (can't?) collapse it. Notice also the frame is selected (little square blue boxes on its lines) to make clear from where to where the frame runs.
So basically you have to make sure the frame is all by itself, all its elements only on the frame, and no other elements overlapping the frame.

PS: it might be possible to do it per field, but since I needed at least a label + field to be shown/hidden, I didn't even try that.

To put in the other necessary settings, you have two ways of doing that:

  1. Via the XML:

    Put in the condition of the frame. So if theObject is not null, the frame will be shown. Also make sure you add 'isRemoveLineWhenBlank="true"'. E.g:

  2. Or via the Eclipse properties tab: see below screenshot:


    Notice setting the flag and the expression, see the arrow pointing at $F, that's the start of the same above expression in the XML.


Thursday, July 3, 2014

Vaadin 7 + Spring 3 + JPA project summary and lessons learned

This blogpost is a short summary of the Vaadin 7.0 project I did last year, including tools used, lessons learned and a bunch of screenshots to show the results.

Tools used

  • Spring 3.2 including annotation based configuration, Spring Task*Executor framework
  • JPA2 with Hibernate 4.2 + JPA modelgen, including Envers for auditing
  • JMS 2
  • Vaadin 7.0.1
  • JAXB 2.1
  • Spring WS 2.1
  • Shiro 1.2
  • JUnit 4.8
  • Mockito
  • Eclipse 3.7
  • SOAP + SoapUI
  • MySql 5.5 + H2
  • Tomcat 7
  • Jenkins
  • Fisheye/Crucible
  • Sonar
  • Subversion
  • Maven 3
  • Java 6

Lessons learned

  • Don't only trust SoapUI of being able to validate your WSDL. Also try to generate stubs with Axis2 and JAXWS generators. *AND* actually make sure you can make a call to the webservice, because on our project it did work perfectly fine in SoapUI, but not in Axis nor JAXWS. Example: soap fault in operation but not in porttype defined. It required the project to defining one or more xyz.jaxb files for JAX-WS  (using .episodes dit not work for us). Then use these in the wsimport etc. Related links: Compiling multiple WSDLs that share a common schema and Customizing Java Packages.
  • If sources not found of e.g junit-4.11 (junit.org is also giving a lot of 404s), then run mvn clean dependency:sources, that will download the sources too into the .m2 repo.
  • Try to keep the WSDL and XSD as semantic as possible, so use Strings instead of IDs for for example lookup /reference values like: started, finished. Makes the interface much more readable and understandable by just looking at it.
  • count(case ...) is NOT supported by Hibernate criteria API, when using it it gives: java.lang.IllegalArgumentException: org.hibernate.hql.internal.ast.QuerySyntaxException: unexpected token: case near line 1, column 92 [...  Use sum(case) since that is supported.

    So instead of:

    cb.count(cb.selectCase().when(cb.equal(workOrder.get(WorkOrder_.priority), 1), workOrder.get(WorkOrder_.priority)).otherwise(0)),
    Use:

    cb.sum(cb.selectCase().when(cb.equal(workOrder.get(WorkOrder_.priority), 1), 1).otherwise(0)) 

    SQL equivalent:

    Instead of:

    COUNT(CASE WHEN priority = 1 THEN priority ELSE NULL END) AS prio1,

    Use:

    SUM(CASE WHEN priority = 1 THEN 1 ELSE 0 END) AS prio1,

    Links about this:

    https://forum.hibernate.org/viewtopic.php?f=1&t=992497
    https://forum.hibernate.org/viewtopic.php?p=2393060
    http://stackoverflow.com/questions/11011151/jpa-criteria-query-order-by-enum-values
    Sums replacement for count
    JPA 2.0 spec including 'case expressions' and this one and this one.
    http://stackoverflow.com/questions/775787/plsql-get-sum-for-each-day-of-week-and-total-sum-for-week-in-a-single-query
  • When trying to upgrade from Vaadin from version 7.0.1 to 7.0.4 by updating the version number in the pom.xml, a serial UID problem occurred when generating the widgetset for this new version:

    [INFO]  at com.google.gwt.dev.Compiler.main(Compiler.java:177)
    [INFO] Caused by: java.io.InvalidClassException: com.google.gwt.dev.jjs.ast.JMethod; local class incompatible: stream classdesc serialVersionUID = 5017484276333252513, local class serialVersionUID = 9103713597467037978

    Tried it then with 7.0.2, that one worked fine! 7.0.3 gave the same error. This post talks about maybe the GWT cache being the problem. So I deleted the directory (even though the timestamp on that directory was recent!) and that worked. In my case the directory to delete was:

    D:\workspace\xyz\frontend\app-web\src\main\webapp\VAADIN\gwt-unitCache 

The application

Below you can find a bunch of screenshots of the resulting application Planning Optimizer or ePOp.



The login screen, authentication via Shiro with a custom Realm that does a SOAP call to the backend:


The dashboard so the planner can immediately see the status + what needs attention first:

The Vaadin Charts 90 days look ahead graph shows the status for the coming 90 days. Clicking on a bar shows the orders at the bottom table for that day. A similar 12 weeks graph is available on another tab:



Different ways of looking at the status exist, for example these week-based graphs, the first one based on craft, the second one on priority:
Notice in the above graph the multiple y-axis: the bars are for the left y-axis, the lines for the right y-axis.

And other filters:




When clicking on a row in the bottom table, details for that order can be filled in:

Links (from a whitelist) and documents can be attached too (including drag 'n drop), and in the end a PDF can be generated with all the info (including the attachments + where the links are pointing to!) included:

Wednesday, June 18, 2014

Android and iOS app discovery and search engines overview

Introduction

Recently a friend of mine asked how to check if an app or type of app with certain functionality already exists. He had a great idea for an app (he thinks :) so wanted to figure out: does such an app already exist?
Entering the name of an app is not sufficient of course, you want to search on offered functionality too, since the name of the app might not match the functionality. E.g Tinder is a dating app, but the name Tinder doesn't show that (at least not in its early days!).

I knew there are a couple of websites and apps that provide that, but had a hard time coming up with more than one. This list is pretty ok, but already partly out of date.
So I decided to create this blogpost with a list of the current recent websites and their apps and some(!) app-only search apps that provide app discovery.

Main focus of this post will be on app search engine websites, not apps! Mainly because the research for an existing app you'd probably like to do behind a computer (laptop or PC), not a phone or tablet, since it requires quite some typing. E.g all types of ways of describing the functionality.
Also most focus will be on Android, but also some iOS search engines will be mentioned.

Below is a list of app search engines I found, including a screenshot and my short take on it. If any search engine is missing, please leave a comment and I'll check it out. If appropriate I'll add it to this page!

App discovery and search engines

1 Standard stores

Of course the standard app stores have their own search engine, either online and/or on the platform itself, e.g:


There are quite some more 3rd party app stores like SlideMe, but usually these are considered less important than the "big ones", so chances are minimal that there's an app in those stores which is not in the big app stores. But for adult content you mighthave to check those.

2 AppBrain

URL: http://www.appbrain.com/
Platforms: Android
Has app: yes


3 Quixey

URL: https://www.quixey.com/
Platforms: Android, iOS, Windows, Blackberry
Has app: yes


4 App Annie

URL: http://www.appannie.com/
Platforms: Android, iOS, Amazon, Mac
Has app: yes but can't tell from the screenshots if it includes searching etc



This is more intended for app analytics and ASO (app store optimization) and SEO but still usuable for checking what type of apps exist, e.g by keyword.
App Annie recently purchased Distimo, another well known app statistics/analytics platform.

Most parts are for free, "enterprise" is not.

5 AppGravity

URL: http://appgravity.com/
Platforms: Android, iOS
Has app: yes



6 AndroLib

URL: http://www.androlib.com/
Platforms: Android
Has app: no


Still refers to "Android Market" instead of Google Play.

7 AppsZoom

URL: http://www.appszoom.com/
Platforms: Android, iOS
Has app: yes


8 MobileDevHQ (formerly known as AppStoreHQ)

URL: http://www.mobiledevhq.com/
Platforms: Android, iOS
Has app: no


Similar to AppAnnie more intended for ASO and analytics and keywords optimilization, but can still come in handy. 

9 Regular search engines

More and more regular search engines are optimizing for finding apps. For for example Google, Apple and Amazon it's of course in their own interest to help people find apps for their platform.

10 App only

There are also many apps out there trying to help you find apps. I did not investigate those, but a query you could run to find a list is this one

11 DIY

If none of the above work for you, you could consider writing your own search engine/crawling engine. Here's a script for the Google Play store.


Saturday, February 8, 2014

Fairphone is in!

Yes it's in (well already for over a month), the Fairphone! Since I was one of the first 10.185 backers, a special edition with an extra message:





Wednesday, December 11, 2013

Vaadin TextFields in a Table empty/null value not propagated for required field

Setup

Problem

When you create a table in Vaadin with a .setTableFieldFactory(), and you have a Field which is set to .setRequired(true), and a user puts in an empty value for that field after first putting in a non-empty value, the field won't be set to that empty value when iterating through the table to read the current values.

Solution

Make sure you invoke .setInvalidCommitted(true) on that field to have the empty value also be passed through; i.e "committed" even though validation fails.

For example:

urlTable.setTableFieldFactory(new DefaultFieldFactory() {

    @Override
    public Field createField(final Container container, final Object itemId, final Object 
                    propertyId, Component uiContext) {

    if (URL_PROPERTY_ID.equals(propertyId)) {
        
        final TextField urlText = new TextField();
        urlText.setRequired(true);

        // Add the below line to have also the empty string be
        // passed on!!
        // See: https://vaadin.com/forum#!/thread/1129126
        urlText.setInvalidCommitted(true);
        urlText.setImmediate(true);
        urlText.setNullRepresentation("");
        urlText.setInputPrompt("Enter a URL");
        return field;

}

Note the bold line.

And for completeness here an example of the iterating over the table to get the current entered values:

List result = new ArrayList();
int sequence = 1;
for (Iterator iterator = urlTable.getItemIds().iterator(); iterator.hasNext();) {

    // Get the current item identifier
    String itemId = (String) iterator.next();

    // Now get the actual item from the table.
    Item item = linksTable.getItem(itemId);
    @SuppressWarnings("unchecked")
    Property urlProperty = item.getItemProperty(URL_PROPERTY_ID);
    String url = urlProperty.getValue();

    // Fill the object
    UrlInfo urlInfo = new UrlInfo();
    urlInfo.setUrl(url);
    // ... set other stuff
    result.add(urlInfo);
    sequence++;
}

return result;

Related resources

The forum thread that put me on the right track can be found here.
Long thread related to required vs validation problems can be found here.
@NotNull bean validator annotation combined with setRequired() also has some issues.