Recreate Vagrant box when original .box is gone from vagrantcloud

Introduction

Suppose you've setup a cool Vagrant image based upon a box from the Vagrant Hashicorp cloud. But suddenly one day the URL is gone! So the original box is gone. E.g this one, which gives a 404 page: https://atlas.hashicorp.com/box-cutter/boxes/ubuntu1404-desktop
Your 'vagrant up' still works, vagrant will just complain it can't find the box anymore, but the VirtualBox image will start up fine.

But you are still getting new team-members on board, and you want them to use the Vagrant setup you created; but the URL is gone so the base box can't be downloaded anymore. What to do?

Well, it turns out you can recreate the .box file which the above URL is pointing to.

This post describes how.

Steps

1. Locate where the Vagrant box from the URL is stored on your local machine. On Windows 7 it is C:/Users/ttlnews/.vagrant.d/boxes 
   At this location are all the base Vagrant boxes downloaded. So without any changes/provisioning made to them. The exact directory of the above ubuntu1404-desktop URL is: C:/Users/ttlnews/.vagrant.d/boxes/box-cutter-VAGRANTSLASH-ubuntu1404-desktop
   
   
2. Create a zip of the deepest directory in there. In my case it I 7zipped all the files in C:/Users/ttlnews/.vagrant.d/boxes/box-cutter-VAGRANTSLASH-ubuntu1404-desktop/1.0.11/virtualbox So not the directory, just the files, like the box.ovf, metadata.json, *.vmdk, Vagrantfile. The resulting file was: box-cutter-VAGRANTSLASH-ubuntu1404-desktop.box.7z
   
   This is what it should look like in the .7z file:
   
   
   
   
   
3. Rename the file to box-cutter-VAGRANTSLASH-ubuntu1404-desktop.box.
   
   
4. Put it in c:/tmp or wherever you want; make sure to change accordingly in the below steps.
   
   
5. In the directory where you have your (already existing because you have created it successfully before from the now-defunct box URL) Vagrantfile add the following line below config.vm.box:
   
   config.vm.box_url = "file:///c:/tmp/box-cutter-VAGRANTSLASH-ubuntu1404-desktop.box"
   
   Tip 1: when testing this out, it's safer to copy your Vagrantfile + whatever else you had to a new directory, so you won't mess up your working Vagrant setup.
   
   Tip 2: in that new setup from tip 1, also change the value of config.vm.box in the Vagrantfile, so you can clearly identify this new image.
   
   
6. Run 'vagrant up' and you should see it copy (extract) the files, start a VirtualBox and start provisioning the box!

Main source for this solution: https://docs.vagrantup.com/v2/boxes/format.html

Integrating Twitter 1.1 api in Android with oauth including app-only authentication

Introduction

Since June 11 2013 Twitter has switched off API 1.0 and is now only suppporting version 1.1.

One major change is that all API calls now have to be authenticated (and authorized) via OAuth: "user authorization (and access tokens) are required for all API 1.1 requests". That means any action on Twitter that is related to a user has to happen this way.

Luckily there is also a so called application-only (app-only) authentication for actions that don't require a user as a context. Those are requests on behalf of an application. Think about search and getting a timeline for a user. One thing that's definitely different from the other calls is that this app-only authentication has to use OAuth 2.0, while the user-context based authentication has to use OAuth 1.0a!

In this post I'll describe how to implement both, including a full working Android code example.

App-only Twitter 1.1 Authentication OAuth 2.0

Information about this principle can be found here. As said above, you can use it for accessing the Twitter API 1.1 endpoints that do not require a user action. For example searching for tweets falls in this category; just like getting a tweet from a user's timeline. Tweeting a status update does not fall in this category, because a user has to give permission for this. You can see if a Twitter API call supports application-only authentication if there's a rate-limit specified with it. E.g for the search it says on the right: 450/app. So that's accessible via the app-only API. But getting a user's direct messages isn't, it only has 15/user specified.

The problem with this userless-context API solution I had was my app was using the Signpost library for the Twitter 1.0 API. I preferably would have used that lib for this app-only authentication too, but the Signpost still does not support OAuth 2 yet...

Then I tried to use the Scribe library, which supports OAuth 2.0, but I couldn't get that working quickly. 

Therefore I went back to the basics and used plain HTTP POST and GET methods, using this blogpost as a starting point. All pieces to get it working are in that post, but not available as one complete code set, that's why I made this post. Plus you might come along a few problems too... also described below.

Making it work

Easiest is to refer to the above mentioned Crazy Bob's blogpost to get started and what's happening why.

In my attached example code (see bottom of this post) most of the logic occurs in OAuthTwitterClient.java. In OnCreate() the UI is set up, including the consumer, provider and client for the OAuth 1.0a using Signpost (see next section for details on that part).

When a user clicks on the top button of the application, named '1a) Request bearer-token via app-only oauth2') the getBearerTokenOnClickListener.onClick() method is invoked. There you see the bearer token getting retrieved via requestBearerToken(), as described in Crazy Bob's blogpost.

That should give the bearer token which you need as mentioned here. 

Then to get a tweet from a Twitter user (in the code I'm using 'twitterapi'), click on the second button named '1b) Get tweet from user via app-only oauth2'. That invokes its listener which invokes fetchTimelineTweet(), which in turn gets the tweet using the bearer token, after which the tweet is displayed on the screen.

Troubleshooting 

I ran into a few issues:

1. When fetching the tweet, the getInputStream (in readResponse()) was giving a FileNotFoundException. Turns out that since Android 4 ICS (Ice Cream Sandwich), this can occur because the conn.setDoOutput(true) causes the request to be turned into a POST even though the specified request method is GET! See here. The fix is to remove that line.
2. When invoking requestBearerToken() two times in a row on Android 1.5, the responseCode showed as -1 sometimes; mainly when waiting about 30 seconds before invoking it again. That turned out to be a known issue for Android before 2.2 Froyo. For that the method disableConnectionReuseIfNecessary() is put in place. 
3. It seems a bearer token lasts "forever", until it is revoked/invalidated. Twitter can do that but you can do it yourself too, via the invalidate_token API call.

PS 1: no the code is not optimal, for example the error handling can be done a lot better

PS 2: and no the UI won't win any "Best Interface" prizes

PS 3: yes all network calls should be done on a non-UI thread really. Yup that's some homework for you too :)

User Context Twitter 1.1 Authentication OAuth 1.0a

For updating a user's status (i.e. tweeting), you have to be authenticated and the user has to have authorized you to perform those type of actions. Usually the user authorization happens by switching to a browser in which the user has to log in to Twitter and say whether the application is allowed to perform the requested actions.

Luckily, for this type of authentication still the Signpost library can be used since its based on OAuth 1.0a.

Making it work

After upgrading to the signpost-core-1.2.1.2.jar and signpost-commonshttp4-1.2.1.2.jars, everything almost worked immediately. Especially make sure to use CommonsHttpOAuthProvider as it says on the Signpost homepage in the Android section.

Since my app still runs on Android SDK 1.5 (yes!) I tested the solution on a 1.5 emulator.

But then I ran into my first problem, authentication failed with: javax.net.ssl.SSLException: Not trusted server certificate.

To get this working, I had to make sure all the Twitter Root CA certificates are known. For that I used this blogpost. See the class CrazyBobHttpClient for the implementation. After putting in all VeriSign certificates one by one (search for the string 'class 3 Public Primary Certification Authority - G' in the .pem file mentioned in this Twitter Developers Forum post to find all 5 of them).

In short, issue the following command to have each one added to your own keystore:

C:\>C:\jdk1.7.0_17\jre\bin\keytool -import -v -trustcacerts -alias 3 -file verisign_cert_class3_g5.cert -keystore twitterkeystore.bks -storetype BKS -provider org.bouncycastle.jce.provider.BouncyCastleProvider -providerpath bcprov-jdk16-145.jar -storepass ez24get

So in the above I'm adding the VeriSign G5 certificate. The -alias 3 you have to increase each time. And notice the password used: 'ez24get', you'll find that in the CrazyBobHttpClient class.

If you see:

Certificate already exists in system-wide CA keystore under alias

Do you still want to add it to your own keystore? [no]:  

answer with: yes

And do the same for the DigiCert certificates, you only need the 3 Root ones. You can find the separate certificates and keystore in the res/raw directory of the example application. The separate certificates don't have to be provided, I just left them there for reference.

After performing all these steps, I was able to authenticate and even tweet. Just as a test I switched back to the DefaultHttpClient(). And the authentication now goes fine using that implementation too! Strange, because AFAIK that one just uses the default system keystore, and before it gave the "not trusted server certificate error"... Strange to me. If anybody knows the reason why, please leave it in the comments!

PS: maybe even cleaner is to add your own created keystore to the existing keystore(s). See the comments at the top in the CrazyBobHttpClient.java.

Outstanding Issues

Well, issues.... more like questions: 

• Why does the DefaultHttpClient() suddenly work (see previous section)?
• It would be nice to have an example for app-only authentication using an OAuth 2.0 library like Scribe instead of using this low level POST/GET code.
• In the LogCat I see these messages appear after each API call:
  
  07-01 18:59:03.575: W/ResponseProcessCookies(732): Invalid cookie header: "set-cookie: guest_id=v1%3A137270514572832152;  Domain=.twitter.com; Path=/; Expires=Wed, 01-Jul-2015 18:59:05 UTC". Unable to parse expires attribute: Wed, 01-Jul-2015 18:59:05 UTC
  Why do these appear?
• It should be possible to get the Twitter Timeline for a given user using your application's private access tokens (and thus not requiring the app-only authentication). Of course this is not recommended, because that would mean you'd have to put these tokens in your application. But for some situations it could be an option. See this PHP code on how that can be done.

The Sample Code Application

You can find the complete source code here. 

It should run on anything from Android 1.5 and higher.

It has been tested by me on:

• Android 1.5 emulator
• Android 4.0 Samsung S2
• Android 4.0 emulator

You might want to remove the scribe-oauth2.0-1.3.0.jar from the libs directory, just left it there for reference. Same goes for the certificates in the res/raw/ directory. Removal is not necessary, it just uses unnecssary storage.

How to run it

1. Load the project in your favorite IDE or whatever you use to build Android applications.
2. In OAuthTwitterClient change the values of CONSUMER_KEY and CONSUMER_SECRET to your own application keys.
3. Also modify the CALLBACK_URL to your application's callback URL. Note: you can also make an OOB (out-of-bounds) call that doesn't require a callback URL, but that's for you to figure out, didn't spend time on that.
4. Deploy it. You should see a screen like this:
   
   
   
   
5. Press button 1a) to get a bearer token. After pressing it should look like:
   
   
   
   
6. Press 1b) to get the tweet. That should look like:
   
   
   
   
7. Press button 2a) to get authenticated and authorized to post a tweet on behalf of a user. When the browser opens, enter the credentials under which account the tweet should be performed (no screenshot for that one). Then authorize the app (click Authorize app):
   
   
   
   
   
   
   It should get back to:
   
   
   
   
   
   
   Note: I am getting a message that the certificate is not ok. This is I think because the quite old 1.5 emulator does not have that root certificate:
   
   
   
   
   
   
   Because the certificate looks fine: 
   
8. Enter a tweet text.
9. Press button 2b) to tweet the text above it. That should give you:
   
   
   
   
10. At the bottom of the screen you can see some statuses of what's going on. If you see any error message, check the LogCat.
11. That's it!

And here two examples of what it can look like after integration in an app, here in the free Android SongDNA app.
Showing all tweets with the words 'shakira' and 'underneath your clothes' in it.

And the different actions possible on each tweet:

Vaadin Charts add-on showing custom properties in tooltip hover

Recently I had to investigate if it is possible to show other data than the X and Y value when hovering over a stacked bar graph in the Vaadin Charts add-on. And yes it is possible!
Nowhere I could find an example on how to do this, so that's why I wrote this blogpost. Hope it helps some people :)

Setup:
- Vaadin 7.0.1
- Charts add-on 1.0.1. In version 1.0.0 you can't set the color of the stacked bars using PlotOptions and ContainerDataSeries. This version has a fix for that.
- Note: I had to manually modify some labels of the hover in the screenshot since I can't use the real data used in the project. Same goes for the code, did not check if it still compiles. Let me know if something essential is missing :) Oh yeah and this is all demo-code so no quality complaints please....

Goal:
- Be able to add a custom field to the hover. And even better: be able to add an array to the hover tooltip.

Here's a screenshot of what the goal was. Click on it for a larger image:

The code:
Notice the Grapes ok: 7 Total: 16 in the hover text. These are the standard x and y values you can show easily via:

Tooltip tooltip = new Tooltip();         tooltip.setFormatter("function() { return ''+ this.x +'"+ " '+ this.series.name +': '+ this.y  +' '+'Total: '+ this.point.stackTotal;}");         conf2.setTooltip(tooltip);

But also note the extra this.point.custom = 708. That's an example of adding a custom value. The tooltip code then looks like:

tooltip.setFormatter("function() {"return ''+ this.x +'"+ " '+this.series.name +': '+ this.y +' '+ 'Total: '+ this.point.stackTotal + ' this.point.custom = ' + this.point.custom + "''"

And then below that line you see a set of table-like data: Quality 1: 2 Stored: 3 Interesting: 4. This is the data that comes from three arrays. The tooltip code then looks like this:

tooltip.setFormatter("function() { var qualityValuesArray = JSON.parse(this.point.qualities); " + "var storedValuesArray = JSON.parse(this.point.stored); " + "var interestingValuesArray = JSON.parse(this.point.interesting); " + "return ''+ this.x +'"+ " '+this.series.name +': '+ this.y +' '+ 'Total: '+ this.point.stackTotal + ' this.point.custom = ' + this.point.custom + " + "' Quality 1: ' + qualityValuesArray[0] + " + "'Stored: ' + storedValuesArray[0] + " + "'Interesting: ' + interestingValuesArray[0] + " + "' Quality 2: ' + qualityValuesArray[1] + " + "'Stored: ' + storedValuesArray[1] + " + "'Interesting: ' + interestingValuesArray[1] + " + "' Quality 3: ' + qualityValuesArray[2] + " + "'Stored: ' + storedValuesArray[2] + " + "'Interesting: ' + interestingValuesArray[2] + " + "' Quality 4: ' + qualityValuesArray[3] + " + "'Stored: ' + storedValuesArray[3] + " + "'Interesting: ' + interestingValuesArray[3] + " + "' Quality 5: ' + qualityValuesArray[4] + " + "'Stored: ' + storedValuesArray[4] + " + "'Interesting: ' + interestingValuesArray[4] + " + "' Quality 6: ' + qualityValuesArray[5] + " + "'Stored: ' + storedValuesArray[5] + " + "'Interesting: ' + interestingValuesArray[5] + " + "' Quality 7: ' + qualityValuesArray[6] + " + "'Stored: ' + storedValuesArray[6] + " + "'Interesting: ' + interestingValuesArray[6] + " + "' Quality 8: ' + qualityValuesArray[7] + " + "'Stored: ' + storedValuesArray[7] + " + "'Interesting: ' + interestingValuesArray[7] + " + "'.';}");

The span can also be replaced by a HTML b tag of course.

Too bad you can't use the setHTML() for better formatting, like putting in a HTML table tag. This is a known issue, see here.

Essential in constructing the data is to use the BeanItemContainer, ContainerDataSeries and the addAttributeToPropertyIdMapping() in which you can put custom class (bean) OrderChartItem:

/** * Class with custom value and 3 arrays to be displayed in tooltip */ public class OrderChartItem { private static final int NR_QUALITIES = 8; private final String name; private final int x; private final Number y; private final int someValue; private final Number[] quality = new Number[NR_QUALITIES]; private final Number[] stored = new Number[NR_QUALITIES]; private final Number[] interesting = new Number[NR_QUALITIES]; public OrderChartItem(String name, int x, Number y, int someValue) { this.name = name; this.x = x; this.y = y; this.someValue = someValue; for (int i = 0; i < NR_QUALITIES; i++) { quality[i] = i + 2; stored[i] = i + 3; interesting[i] = i + 4; } } public String getName() { return name; } public int getX() { return x; } public Number getY() { return y; } public int getSomeValue() { return someValue; } public List getQualities() { return java.util.Arrays.asList(quality); } public List getStored() { return java.util.Arrays.asList(stored); } public List getInteresting() { return java.util.Arrays.asList(interesting); } }

I made the arrays the same size because of its table data, but that doesn't have to be of course.

The basic mechanism is:

1. Set up a BeanItemContainer with values of OrderChartItem. Take special care when putting in 0 values. The graph then shows the number 0. To avoid that, put in 'null' as a value.
   
   

   BeanItemContainer containerGood = new BeanItemContainer(OrderChartItem.class); containerGood.addBean(new OrderChartItem("some name good", 1, null, 701));

   
    So the OrderChartItem has the properties that can be reached from the tooltip Javascript. Note the OrderChartItem just gets some name, later it is overwritten by setting the ContainerDataSeries names:
   
   

   seriesGood.setName("good");



2. Then wrap each BeanItemContainer in a ContainerDataSeries:
   
   

   ContainerDataSeries seriesGood = new ContainerDataSeries(containerGood);

3. Then add the custom properties for each container. First the one value:
   
   

   seriesGood.addAttributeToPropertyIdMapping("custom", "someValue");

   Then the arrays:

   seriesGood.addAttributeToPropertyIdMapping("qualities", "qualities");

4. And add the series to the configuration:
   
   

   conf2.addSeries(seriesGood);

In summary: the OrderChartItem is used as data-provider by adding them to the ContainerDataSeries. The properties-mappings of the custom OrderChartItem are then added to the series, after which they are accessible in the tooltip Javascript. Note the "trick" of using JSON.parse() in the formatter and the List getQualities() to get the arrays in the correct format.

Full source code can be found here.

The widgetsets directory has been cleared out because of distribution issues.