Integrating Twitter 1.1 api in Android with oauth including app-only authentication

Introduction

Since June 11 2013 Twitter has switched off API 1.0 and is now only suppporting version 1.1.

One major change is that all API calls now have to be authenticated (and authorized) via OAuth: "user authorization (and access tokens) are required for all API 1.1 requests". That means any action on Twitter that is related to a user has to happen this way.

Luckily there is also a so called application-only (app-only) authentication for actions that don't require a user as a context. Those are requests on behalf of an application. Think about search and getting a timeline for a user. One thing that's definitely different from the other calls is that this app-only authentication has to use OAuth 2.0, while the user-context based authentication has to use OAuth 1.0a!

In this post I'll describe how to implement both, including a full working Android code example.

App-only Twitter 1.1 Authentication OAuth 2.0

Information about this principle can be found here. As said above, you can use it for accessing the Twitter API 1.1 endpoints that do not require a user action. For example searching for tweets falls in this category; just like getting a tweet from a user's timeline. Tweeting a status update does not fall in this category, because a user has to give permission for this. You can see if a Twitter API call supports application-only authentication if there's a rate-limit specified with it. E.g for the search it says on the right: 450/app. So that's accessible via the app-only API. But getting a user's direct messages isn't, it only has 15/user specified.

The problem with this userless-context API solution I had was my app was using the Signpost library for the Twitter 1.0 API. I preferably would have used that lib for this app-only authentication too, but the Signpost still does not support OAuth 2 yet...

Then I tried to use the Scribe library, which supports OAuth 2.0, but I couldn't get that working quickly. 

Therefore I went back to the basics and used plain HTTP POST and GET methods, using this blogpost as a starting point. All pieces to get it working are in that post, but not available as one complete code set, that's why I made this post. Plus you might come along a few problems too... also described below.

Making it work

Easiest is to refer to the above mentioned Crazy Bob's blogpost to get started and what's happening why.

In my attached example code (see bottom of this post) most of the logic occurs in OAuthTwitterClient.java. In OnCreate() the UI is set up, including the consumer, provider and client for the OAuth 1.0a using Signpost (see next section for details on that part).

When a user clicks on the top button of the application, named '1a) Request bearer-token via app-only oauth2') the getBearerTokenOnClickListener.onClick() method is invoked. There you see the bearer token getting retrieved via requestBearerToken(), as described in Crazy Bob's blogpost.

That should give the bearer token which you need as mentioned here. 

Then to get a tweet from a Twitter user (in the code I'm using 'twitterapi'), click on the second button named '1b) Get tweet from user via app-only oauth2'. That invokes its listener which invokes fetchTimelineTweet(), which in turn gets the tweet using the bearer token, after which the tweet is displayed on the screen.

Troubleshooting 

I ran into a few issues:

1. When fetching the tweet, the getInputStream (in readResponse()) was giving a FileNotFoundException. Turns out that since Android 4 ICS (Ice Cream Sandwich), this can occur because the conn.setDoOutput(true) causes the request to be turned into a POST even though the specified request method is GET! See here. The fix is to remove that line.
2. When invoking requestBearerToken() two times in a row on Android 1.5, the responseCode showed as -1 sometimes; mainly when waiting about 30 seconds before invoking it again. That turned out to be a known issue for Android before 2.2 Froyo. For that the method disableConnectionReuseIfNecessary() is put in place. 
3. It seems a bearer token lasts "forever", until it is revoked/invalidated. Twitter can do that but you can do it yourself too, via the invalidate_token API call.

PS 1: no the code is not optimal, for example the error handling can be done a lot better

PS 2: and no the UI won't win any "Best Interface" prizes

PS 3: yes all network calls should be done on a non-UI thread really. Yup that's some homework for you too :)

User Context Twitter 1.1 Authentication OAuth 1.0a

For updating a user's status (i.e. tweeting), you have to be authenticated and the user has to have authorized you to perform those type of actions. Usually the user authorization happens by switching to a browser in which the user has to log in to Twitter and say whether the application is allowed to perform the requested actions.

Luckily, for this type of authentication still the Signpost library can be used since its based on OAuth 1.0a.

Making it work

After upgrading to the signpost-core-1.2.1.2.jar and signpost-commonshttp4-1.2.1.2.jars, everything almost worked immediately. Especially make sure to use CommonsHttpOAuthProvider as it says on the Signpost homepage in the Android section.

Since my app still runs on Android SDK 1.5 (yes!) I tested the solution on a 1.5 emulator.

But then I ran into my first problem, authentication failed with: javax.net.ssl.SSLException: Not trusted server certificate.

To get this working, I had to make sure all the Twitter Root CA certificates are known. For that I used this blogpost. See the class CrazyBobHttpClient for the implementation. After putting in all VeriSign certificates one by one (search for the string 'class 3 Public Primary Certification Authority - G' in the .pem file mentioned in this Twitter Developers Forum post to find all 5 of them).

In short, issue the following command to have each one added to your own keystore:

C:\>C:\jdk1.7.0_17\jre\bin\keytool -import -v -trustcacerts -alias 3 -file verisign_cert_class3_g5.cert -keystore twitterkeystore.bks -storetype BKS -provider org.bouncycastle.jce.provider.BouncyCastleProvider -providerpath bcprov-jdk16-145.jar -storepass ez24get

So in the above I'm adding the VeriSign G5 certificate. The -alias 3 you have to increase each time. And notice the password used: 'ez24get', you'll find that in the CrazyBobHttpClient class.

If you see:

Certificate already exists in system-wide CA keystore under alias

Do you still want to add it to your own keystore? [no]:  

answer with: yes

And do the same for the DigiCert certificates, you only need the 3 Root ones. You can find the separate certificates and keystore in the res/raw directory of the example application. The separate certificates don't have to be provided, I just left them there for reference.

After performing all these steps, I was able to authenticate and even tweet. Just as a test I switched back to the DefaultHttpClient(). And the authentication now goes fine using that implementation too! Strange, because AFAIK that one just uses the default system keystore, and before it gave the "not trusted server certificate error"... Strange to me. If anybody knows the reason why, please leave it in the comments!

PS: maybe even cleaner is to add your own created keystore to the existing keystore(s). See the comments at the top in the CrazyBobHttpClient.java.

Outstanding Issues

Well, issues.... more like questions: 

• Why does the DefaultHttpClient() suddenly work (see previous section)?
• It would be nice to have an example for app-only authentication using an OAuth 2.0 library like Scribe instead of using this low level POST/GET code.
• In the LogCat I see these messages appear after each API call:
  
  07-01 18:59:03.575: W/ResponseProcessCookies(732): Invalid cookie header: "set-cookie: guest_id=v1%3A137270514572832152;  Domain=.twitter.com; Path=/; Expires=Wed, 01-Jul-2015 18:59:05 UTC". Unable to parse expires attribute: Wed, 01-Jul-2015 18:59:05 UTC
  Why do these appear?
• It should be possible to get the Twitter Timeline for a given user using your application's private access tokens (and thus not requiring the app-only authentication). Of course this is not recommended, because that would mean you'd have to put these tokens in your application. But for some situations it could be an option. See this PHP code on how that can be done.

The Sample Code Application

You can find the complete source code here. 

It should run on anything from Android 1.5 and higher.

It has been tested by me on:

• Android 1.5 emulator
• Android 4.0 Samsung S2
• Android 4.0 emulator

You might want to remove the scribe-oauth2.0-1.3.0.jar from the libs directory, just left it there for reference. Same goes for the certificates in the res/raw/ directory. Removal is not necessary, it just uses unnecssary storage.

How to run it

1. Load the project in your favorite IDE or whatever you use to build Android applications.
2. In OAuthTwitterClient change the values of CONSUMER_KEY and CONSUMER_SECRET to your own application keys.
3. Also modify the CALLBACK_URL to your application's callback URL. Note: you can also make an OOB (out-of-bounds) call that doesn't require a callback URL, but that's for you to figure out, didn't spend time on that.
4. Deploy it. You should see a screen like this:
   
   
   
   
5. Press button 1a) to get a bearer token. After pressing it should look like:
   
   
   
   
6. Press 1b) to get the tweet. That should look like:
   
   
   
   
7. Press button 2a) to get authenticated and authorized to post a tweet on behalf of a user. When the browser opens, enter the credentials under which account the tweet should be performed (no screenshot for that one). Then authorize the app (click Authorize app):
   
   
   
   
   
   
   It should get back to:
   
   
   
   
   
   
   Note: I am getting a message that the certificate is not ok. This is I think because the quite old 1.5 emulator does not have that root certificate:
   
   
   
   
   
   
   Because the certificate looks fine: 
   
8. Enter a tweet text.
9. Press button 2b) to tweet the text above it. That should give you:
   
   
   
   
10. At the bottom of the screen you can see some statuses of what's going on. If you see any error message, check the LogCat.
11. That's it!

And here two examples of what it can look like after integration in an app, here in the free Android SongDNA app.
Showing all tweets with the words 'shakira' and 'underneath your clothes' in it.

And the different actions possible on each tweet:

Best of this Week Summary 15 June - 27 June 2010

• References to two small Google Guice Inversion of Control tutorials. Check also the comments re: Spring's current complexity.
  


• Nice 2D graphics tutorial for Android creating a vintage thermometer.
  A beginners tutorial creating a tic-tac-toe game for Android.
  


• How to combine Wicket panels with Google Analytics.
  

Best of this Week Summary 10 May - 16 May 2010

• Finally a new version of the OWASP Top 10 Web Security Vulnerabilities. Previous versions were from 2007 and 2004.
  


• Lower level 8 potential jQuery optimization tips.
  


• A post explaining implementing RSS feeds with new Spring 3 features like AbstractRssFeedView.
  


• Oracle's ADF is starting to support mobile clients too with ADF Mobile Client. The technology preview for Blackberry is available now. Here's a tutorial PDF that shows how to develop a basic mobile browser application and a native app.
  

Best of this Week Summary 19 April - 25 April 2010

• An introduction to Gizzard, an open sourced sharding framework (store data across multiple computers instead of on just one) which is used by Twitter.
  


• A hands-on tutorial of creating a Spring application that uses Hibernate as JPA provider and JTA for transaction demarcation. A simple Order Processing Message Driven Bean is implemented that showcases this integration. It is deployed on a WebLogic 10.3 server.
  


• Last week Jira from the Apache Foundation was compromised. Here's a description of how the hackers gained access via XSS.
  

Best of this Week Summary 16 November - 22 November 2009

• Did you know Java 5 reached its End of Service Life last month? But luckily not if you've got Java SE for business (support).
  


• This week Google introduced the first version of its Chrome OS. Here's a pretty good explanation on how it differs from current operating systems. And here are 5 (other) points describing them:
  
  
  • "The browser is the OS",
    
  
  
  • "It’s designed to fix itself",
    
  
  
  • "All apps are web apps, no installations", (sounds like Citrix to me ;)
    
  
  
  • "Chrome OS doesn’t support drivers and will not run on your laptop",
    
  
  
  • "Super-fast startup speeds"
    
  
  


• A step-by-step tutorial on creating a CRUD Web application and run it on the Amazon Elastic Compute Cloud. The web application leverages Groovy, Spring, and Hibernate (via the Grails framework), MySQL and Tomcat. For deployment the Eclipse AWS-plugin is used.
  


• Great, here you can download chapters 20, 21, 22 for free from the book "Web Service Contract Design and Versioning for SOA" by Thomas Erl, Anish Karmarkar, Priscilla Walmsley, Hugo Haas, L. Umit Yalcinalp, Canyang Kevin Liu, David Orchard, Andre Tost, James Pasley. These chapters address the issues related to service contract versioning. In those chapters described are versioning of WSDL and schema definitions.
  

Best of this Week Summary 18 May - 24 May 2009

• Tutorial in which you learn to build an SOA application which is listening for incidents coming from web service or files. According to the origin, the content (= incidents) are transformed into their corresponding objects using for the CSV file, a new Camel component : camel-bindy and for the Web Service camel-cxf component. Each message transformed is placed in a queue handled by ActiveMQ engine. All the messages (containing the objects) are next processed by a Bean service who will (with the help of injection of dependency provided by Spring) save the incidents in a DB using Spring and Hibernate frameworks.
  A small Apache Wicket web application running in Jetty Web server provide to the users a screen to consult the incidents created.
  


• This feed is handy to follow to see what's getting more mainstream traction and technology trends. E.g Thought Leadership: Technology Brief May 2009.
  


• All of Dion Hinchcliffe web 2.0 diagrams at one place.
  


• JSON REST webservices best practices described by the Feedly team. A related more general pros & cons of REST are described in this article.
  


• WiQuery aims to integrate the JavaScript frameworks jQuery and jQuery UI with Wicket, a Java framework providing a real way to create web applications with an OO approach. Other integrations like with MooTools, Dojo and Scriptaculous with Prototype can be found here. BTW, a comparison between MooTools and jQuery can be found here.
  


• The distinction between a spring singleton and a java singleton explained.
  

Best of this Week Summary 06 October - 12 October 2008

• One of the many nice editions of The A-Z of Programming Languages series, this time on C#. Yes, yes, that word is not supposed to be on this blog, but this article is still interesting if you're into programming languages. For example the challenges when designing a language. Check also the other editions, for example about: AWK, Forth, Modula-3, Python and Javascript.
  


• An Amazon S3 introduction/beginners guide for setting up hosting of images.
  


• Four cases of CSRF attacks desribed, including one were it was possible to transfer money from a bank account! Check also the mentioned paper on how to prevent them: Cross-Site Request Forgeries: Exploitation and Prevention. The paper explains how to prevent CSRF in your web-application/frameworks: don't let a GET modify anything and pass a random number in the cookie and each form POST. It also describes XSS and the same-origine policy.