Saturday, July 27, 2013

JPA @Version annotation: when is the version updated?


In JPA you can annotate a field of an entity with @Version to specify that property as its optimistic lock value.
But the above documentation does not say when the version number is validated and updated (increased).
The short answer is: at commit time!

I came across this because I was having trouble passing on the updated version number back to the caller, after a merge() operation.

The setup

Example and the solution

A client class (not in a transaction) invokes the method 

   /** 
    * Updates the order and returns the new version number for the order
    *
    * @param order the entity to update. Must be provided with the current version of the order.
    *             If not provided, StaleObjectStateException will be thrown
    */
   Date updateMe(Order orderWithNewValues);

on a service class, which is annotated with Spring @Service and Spring @Transactional

That method invokes 

   /** 
    * Similar java doc
    */
   Date updateMe(SomeEntity orderWithNewValues)

in a DAO which is annotated with Spring @Transactional(propagation = Propagation.MANDATORY) and @Repository.  The DAO searches for the entity to update using its name, sets the version number to the currently known version number, updates it and should return the new version number.

The class Order is annotated (amongst other things) with @Version as follows:

   @Version
   @Column(nullable = false)
   @Temporal(TemporalType.TIMESTAMP)
   private Date version;
   ...
   DateTime getVersion() {
       return version;
   }

The implementation of the DAO method implementing the update originally was:

   Order order = orderDao.findByName(name);
   if (order == null) {
       throw new IllegalArgumentException("No order with name '" + name + "' exists");
   }

   Order updatedOrderNotManaged = new Order(order);
   updatedOrderNotManaged.setVersion(orderWithNewValues.getVersion());
   updatedOrderNotManaged.setTotal(orderWithNewValues.getTotal());

   Order updatedOrderManaged = orderDao.merge(updatedOrderNotManaged);

   return updatedOrderManaged.getVersion();


Expecting that the last line (return updatedOrderManaged.getVersion()) would return the new version number, since the non-managed entity was now managed.

But it isn't! What seems to be the reason is that:
  • The JPA implementation (in this case Hibernate) can do some caching, deferring the actual database access to a later moment, causing the changes to not hit the database yet.
  • Not hitting the database also doesn't cause the @Version to be "triggered" (it seems).
  • So the version is still at its old value!
But after the service method had returned the (old) version to the client calling it (so the transaction was committed), in the database you can see the new increased timestamp in the version column.

The solution is to force the JPA implementation to flush() to the database, such that the @Version annotation gets triggered and thus the version gets updated.
Therefor the following was added, just before the "return"  statement: 


   orderDao.flush();

That did it, now return updatedOrderManaged.getVersion()) is returning the updated version number. Should the transaction commit fail later, then no harm is done since the whole transaction is rolled back and the client should get an exception or similar so it knows it should ignore the returned version number.


Wednesday, July 10, 2013

Vaadin error: A connector with id 7 is already registered!


This post describes a short analysis and possible solution when getting the error java.lang.RuntimeException: A connector with id 7 is already registered!


Setup

  • Vaadin 7.0.4 with Spring using the SpringVaadinIntegration plugin 1.6.5 and Shiro 1.2.1
  • The UI class has @Component en @Scope("prototype") as annotations
  • All components extend CustomComponent
  • All components but one component also have the @Scope("prototype") annotation
  • One component, let's call that the HelpWindow, it extends Window, has no @Scope annotation, and thus is a Spring singleton. The reason for it being a singleton is that it seemed the only solution to always have a sub-window on top of another sub-window (using UI.addWindow()). At least one situation in the project made the sub-window appear beneath another sub-window sometimes...
  • No @VaadinView used anywhere.
  • Spring 3.1.2
  • Tomcat 7 as application server.

Scenario

This is the reproducible scenario that causes the A connector with id 7 is already registered! error to appear: 
  1. Log in for first time, so authentication is via Shiro: all fine, the HelpWindow is shown.
  2. Log out.
  3. Log in again. I can see the UI constructor and its init() getting invoked.
  4. Exception in the logfile:

    java.lang.RuntimeException: A connector with id 7 is already registered!
    ....... stuff deleted.....
    SEVERE: 

    java.lang.RuntimeException: A connector with id 7 is already registered!
    at com.vaadin.ui.ConnectorTracker.registerConnector(ConnectorTracker.java:131)
    at com.vaadin.server.AbstractClientConnector.attach(AbstractClientConnector.java:599)
    at com.vaadin.ui.AbstractComponent.attach(AbstractComponent.java:554)
    at com.vaadin.ui.Label.attach(Label.java:430)
    at com.vaadin.server.AbstractClientConnector.setParent(AbstractClientConnector.java:586)
    at com.vaadin.ui.AbstractComponent.setParent(AbstractComponent.java:457)
    at com.vaadin.ui.Table.registerComponent(Table.java:2350)
    at com.vaadin.ui.Table.parseItemIdToCells(Table.java:2337)
    at com.vaadin.ui.Table.getVisibleCellsNoCache(Table.java:2147)
    at com.vaadin.ui.Table.refreshRenderedCells(Table.java:1668)
    at com.vaadin.ui.Table.getVisibleCells(Table.java:3921)
    at com.vaadin.ui.Table.beforeClientResponse(Table.java:3155)
    at com.vaadin.server.AbstractCommunicationManager.writeUidlResponse(AbstractCommunicationManager.java:799)
    at com.vaadin.server.AbstractCommunicationManager.paintAfterVariableChanges(AbstractCommunicationManager.java:728)
    at com.vaadin.server.AbstractCommunicationManager.handleUidlRequest(AbstractCommunicationManager.java:599)
    at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:315)
    at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:201)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

    The exception occurs at the moment UI.addWindow(helpWindow) is invoked (not included in this exception-dump).
  5. After this, even appending ?restartApplication to the URL won't save you anymore. Only a full restart of the application server (Tomcat) fixes the problem.
  6. BUT: if after the log out I press ctrl-F5 (so full browser refresh) once, the problem still occurs after logging in again.
  7. BUT: if after the log out I press ctrl-F5 twice, the problem does not occur! All works fine. 

Investigation and Solution

So very strange that Vaadin's ConnectorTracker thinks/sees the HelpWindow's connectorId already registered, because the user logged out and the UI's constructor and its init() method were invoked, so a completely new UI is created.
After pressing ctrl-F5 twice, I did see that the HelpWindow's connectorId is cleared (null).
Also changing the HelpWindow's scope to prototype fixed the problem, no exception; but prototype isn't an option for the project for above mentioned reason.

So something somewhere doesn't get fully cleared at logout time. Potentially the reasoning could be: since the HelpWindow is a singleton, Spring won't re-instantiate it (it's a singleton), Vaadin won't clear its connectorId either for some reason, so the ConnectorTracker (see the stacktrace) detects that it is already registered --> exception!

Several posts on the Vaadin forum indicated that the singleton vs prototype might be the problem. Like this one which indicates a potential problem right there. And in this one another caching-like problem is mentioned.

Also not "everything" getting cleared at logout was pointed out as a potential problem. So for that, the code invoked when logging out I changed from only doing currentUser.logout() and the setLocation() to:

// currentUser.logout() might also clear the Vaadin session, but better safe than sorry...
UI.getCurrent().getSession().close();
UI.getCurrent().getSession().getService().closeSession(VaadinSession.getCurrent());
UI.getCurrent().close();

Subject currentUser = SecurityUtils.getSubject(); // Shiro
currentUser.logout(); // Shiro

// It's apparently essential to do the redirect too
UI.getCurrent().getPage().setLocation(VaadinServlet.getCurrent().getServletContext().getContextPath());

For readability, the try/catch around each statement is omitted. And, as mentioned in the first comment, currentUser.logout() might already be sufficient to clear the session, did not further investigate this.

Together with a colleague another solution came to mind: try @Scope("request")! Using that as scope, within an HTTP request the HelpWindow bean exists only once, but a new one is created for each new HTTP request.
And yes, that did it! And it also keeps the 'sub-window on top of a sub-window' problem away.

Sufficient work-around for now!

PS: note that there might be something not 100% ok regarding scope in the SpringVaadinIntegration add-on; see this discussion on what the scope of the UI should be. The last comment (from the user pointing out this potential issue) also refers to using this post as a reference to integrating Spring with Vaadin....

Tip: talking about injection/CDI: integration Vaadin with JEE 6? This might be a good one to read.




Wednesday, July 3, 2013

Integrating Twitter 1.1 api in Android with oauth including app-only authentication

Introduction

One major change is that all API calls now have to be authenticated (and authorized) via OAuth: "user authorization (and access tokens) are required for all API 1.1 requests". That means any action on Twitter that is related to a user has to happen this way.

Luckily there is also a so called application-only (app-only) authentication for actions that don't require a user as a context. Those are requests on behalf of an application. Think about search and getting a timeline for a user. One thing that's definitely different from the other calls is that this app-only authentication has to use OAuth 2.0, while the user-context based authentication has to use OAuth 1.0a!

In this post I'll describe how to implement both, including a full working Android code example.




App-only Twitter 1.1 Authentication OAuth 2.0

Information about this principle can be found here. As said above, you can use it for accessing the Twitter API 1.1 endpoints that do not require a user action. For example searching for tweets falls in this category; just like getting a tweet from a user's timeline. Tweeting a status update does not fall in this category, because a user has to give permission for this. You can see if a Twitter API call supports application-only authentication if there's a rate-limit specified with it. E.g for the search it says on the right: 450/app. So that's accessible via the app-only API. But getting a user's direct messages isn't, it only has 15/user specified.

The problem with this userless-context API solution I had was my app was using the Signpost library for the Twitter 1.0 API. I preferably would have used that lib for this app-only authentication too, but the Signpost still does not support OAuth 2 yet...
Then I tried to use the Scribe library, which supports OAuth 2.0, but I couldn't get that working quickly. 
Therefore I went back to the basics and used plain HTTP POST and GET methods, using this blogpost as a starting point. All pieces to get it working are in that post, but not available as one complete code set, that's why I made this post. Plus you might come along a few problems too... also described below.

Making it work

Easiest is to refer to the above mentioned Crazy Bob's blogpost to get started and what's happening why.
In my attached example code (see bottom of this post) most of the logic occurs in OAuthTwitterClient.java. In OnCreate() the UI is set up, including the consumer, provider and client for the OAuth 1.0a using Signpost (see next section for details on that part).

When a user clicks on the top button of the application, named '1a) Request bearer-token via app-only oauth2') the getBearerTokenOnClickListener.onClick() method is invoked. There you see the bearer token getting retrieved via requestBearerToken(), as described in Crazy Bob's blogpost.
That should give the bearer token which you need as mentioned here
Then to get a tweet from a Twitter user (in the code I'm using 'twitterapi'), click on the second button named '1b) Get tweet from user via app-only oauth2'. That invokes its listener which invokes fetchTimelineTweet(), which in turn gets the tweet using the bearer token, after which the tweet is displayed on the screen.

Troubleshooting 

I ran into a few issues:
  1. When fetching the tweet, the getInputStream (in readResponse()) was giving a FileNotFoundException. Turns out that since Android 4 ICS (Ice Cream Sandwich), this can occur because the conn.setDoOutput(true) causes the request to be turned into a POST even though the specified request method is GET! See here. The fix is to remove that line.
  2. When invoking requestBearerToken() two times in a row on Android 1.5, the responseCode showed as -1 sometimes; mainly when waiting about 30 seconds before invoking it again. That turned out to be a known issue for Android before 2.2 Froyo. For that the method disableConnectionReuseIfNecessary() is put in place. 
  3. It seems a bearer token lasts "forever", until it is revoked/invalidated. Twitter can do that but you can do it yourself too, via the invalidate_token API call.
PS 1: no the code is not optimal, for example the error handling can be done a lot better
PS 2: and no the UI won't win any "Best Interface" prizes
PS 3: yes all network calls should be done on a non-UI thread really. Yup that's some homework for you too :)


User Context Twitter 1.1 Authentication OAuth 1.0a

For updating a user's status (i.e. tweeting), you have to be authenticated and the user has to have authorized you to perform those type of actions. Usually the user authorization happens by switching to a browser in which the user has to log in to Twitter and say whether the application is allowed to perform the requested actions.
Luckily, for this type of authentication still the Signpost library can be used since its based on OAuth 1.0a.

Making it work

After upgrading to the signpost-core-1.2.1.2.jar and signpost-commonshttp4-1.2.1.2.jars, everything almost worked immediately. Especially make sure to use CommonsHttpOAuthProvider as it says on the Signpost homepage in the Android section.

Since my app still runs on Android SDK 1.5 (yes!) I tested the solution on a 1.5 emulator.
But then I ran into my first problem, authentication failed with: javax.net.ssl.SSLException: Not trusted server certificate.
To get this working, I had to make sure all the Twitter Root CA certificates are known. For that I used this blogpost. See the class CrazyBobHttpClient for the implementation. After putting in all VeriSign certificates one by one (search for the string 'class 3 Public Primary Certification Authority - G' in the .pem file mentioned in this Twitter Developers Forum post to find all 5 of them).
In short, issue the following command to have each one added to your own keystore:

C:\>C:\jdk1.7.0_17\jre\bin\keytool -import -v -trustcacerts -alias 3 -file verisign_cert_class3_g5.cert -keystore twitterkeystore.bks -storetype BKS -provider org.bouncycastle.jce.provider.BouncyCastleProvider -providerpath bcprov-jdk16-145.jar -storepass ez24get

So in the above I'm adding the VeriSign G5 certificate. The -alias 3 you have to increase each time. And notice the password used: 'ez24get', you'll find that in the CrazyBobHttpClient class.

If you see:

Certificate already exists in system-wide CA keystore under alias
Do you still want to add it to your own keystore? [no]:  

answer with: yes

And do the same for the DigiCert certificates, you only need the 3 Root ones. You can find the separate certificates and keystore in the res/raw directory of the example application. The separate certificates don't have to be provided, I just left them there for reference.

After performing all these steps, I was able to authenticate and even tweet. Just as a test I switched back to the DefaultHttpClient(). And the authentication now goes fine using that implementation too! Strange, because AFAIK that one just uses the default system keystore, and before it gave the "not trusted server certificate error"... Strange to me. If anybody knows the reason why, please leave it in the comments!

PS: maybe even cleaner is to add your own created keystore to the existing keystore(s). See the comments at the top in the CrazyBobHttpClient.java.


Outstanding Issues

Well, issues.... more like questions: 
  • Why does the DefaultHttpClient() suddenly work (see previous section)?
  • It would be nice to have an example for app-only authentication using an OAuth 2.0 library like Scribe instead of using this low level POST/GET code.
  • In the LogCat I see these messages appear after each API call:

    07-01 18:59:03.575: W/ResponseProcessCookies(732): Invalid cookie header: "set-cookie: guest_id=v1%3A137270514572832152;  Domain=.twitter.com; Path=/; Expires=Wed, 01-Jul-2015 18:59:05 UTC". Unable to parse expires attribute: Wed, 01-Jul-2015 18:59:05 UTC
    Why do these appear?
  • It should be possible to get the Twitter Timeline for a given user using your application's private access tokens (and thus not requiring the app-only authentication). Of course this is not recommended, because that would mean you'd have to put these tokens in your application. But for some situations it could be an option. See this PHP code on how that can be done.


The Sample Code Application

You can find the complete source code here
It should run on anything from Android 1.5 and higher.
It has been tested by me on:
  • Android 1.5 emulator
  • Android 4.0 Samsung S2
  • Android 4.0 emulator
You might want to remove the scribe-oauth2.0-1.3.0.jar from the libs directory, just left it there for reference. Same goes for the certificates in the res/raw/ directory. Removal is not necessary, it just uses unnecssary storage.

How to run it

  1. Load the project in your favorite IDE or whatever you use to build Android applications.
  2. In OAuthTwitterClient change the values of CONSUMER_KEY and CONSUMER_SECRET to your own application keys.
  3. Also modify the CALLBACK_URL to your application's callback URL. Note: you can also make an OOB (out-of-bounds) call that doesn't require a callback URL, but that's for you to figure out, didn't spend time on that.
  4. Deploy it. You should see a screen like this:


  5. Press button 1a) to get a bearer token. After pressing it should look like:


  6. Press 1b) to get the tweet. That should look like:


  7. Press button 2a) to get authenticated and authorized to post a tweet on behalf of a user. When the browser opens, enter the credentials under which account the tweet should be performed (no screenshot for that one). Then authorize the app (click Authorize app):




    It should get back to:




    Note: I am getting a message that the certificate is not ok. This is I think because the quite old 1.5 emulator does not have that root certificate:




    Because the certificate looks fine: 
  8. Enter a tweet text.
  9. Press button 2b) to tweet the text above it. That should give you:


  10. At the bottom of the screen you can see some statuses of what's going on. If you see any error message, check the LogCat.
  11. That's it!
And here two examples of what it can look like after integration in an app, here in the free Android SongDNA app.
Showing all tweets with the words 'shakira' and 'underneath your clothes' in it.



And the different actions possible on each tweet: