Lessons learned using AWS Kinesis to process business events and commands (messages)

Introduction

When using AWS Kinesis as the means of communicating business events, several challenges arise. For the basic key concepts of Kinesis see here. 

Of course the "normal" use case to use Kinesis is described as here in the section "When should I use Amazon Kinesis Data Streams, and when should I use Amazon SQS?" Another comparison between Kinesis and SQS can be found here.

Especially built for Kafka-like stream-processing of millions of events, where usually the producers (e.g think IoT devices) are much faster than the consumer applications.

Usually business events are not generated millions per minute or second. Events from IoT devices are usually not considered business events.

Any next time I would not recommend using Kinesis as a transport mechanism for business events; the main reason for me is its intended use-case is not matching that type of use. Plus the many technical challenges I had to solve, for most of which any "regular" messaging tool like RabbitMQ would suffice.

Additionally, usually one wants business events to arrive almost instantly - or at least as fast as possible. Kinesis is fast but e.g RabbitMQ is often sufficient as transport for business events.

And yes RabbitMQ is now also available as a managed solution - hosted at AWS but managed not by AWS itself. For example with CloudAMQP.

In the below discussion a message can be an event or a command (though most often you'd want the commands to be synchronous, since they should only be delivered to one service anyway)

High level Kinesis architecture overview

Below is a high level overview of Kinesis' architecture: 

There can be multiple streams, and within each stream are one or more shards. Messages in a shard are guaranteed to be delivered at the consumer in the order they were published onto the shard.

The partition-key used while publishing determines into what shard the message is published. Messages with the same partition key get always published into the same shard.

A design decision one has to make is: do I make one stream were all services publish their messages, or do I want a stream per XYZ? Where XYZ could be for example a Bounded Context.

No good reason was found to split into multiple streams, so I decided to go for one Kinesis stream all services publish on.

Lessons learned

When using Kinesis to handle business events between micro-services several challenges were to overcome. Below the lessons learned are described.

More than five different applications processing the same shard at the same time

Kinesis limits documentation  states that each shard within a stream supports up to five read transactions per second. And if you need more than that, it is recommended to increase the number of shards.

If you don't comply to that limit, you'll see ReadProvisionedThroughputExceeded or ProvisionedThroughputExceededException exceptions appear in your logs. This can easily be reached if you have large documents (e.g 1MB) in your DynamoDB too; even with the AWS Console you then can't even view such a large document in your browser!

Increasing the number of shards will spread the published messages across more shards and thus reduce the number of read transactions per second per shard.

BUT: in a microservices architecture you can easily have 10 or more services. And all these services need to process all messages on all shards.

So with for example 10 services, each of those services (container or lambda) will need to poll each shard regularly in some form. And the more services, the more chance that more than 5 services poll any given shard per second... Causing the above ReadProvisionedThroughputExceeded exception.

Thus: increasing the number of shards won't help, because still each service will need to poll each shard.

One workaround for this could be:

• have each service (container, lambda) publish to one central stream. Note that multiple streams in the end will also not help you, since you'll reach that 5 reads per second limit soon as the number of services increases to 5 or higher
• have a stream per service that needs to consume messages
• have a smart lambda that reads from the central stream and re-publishes each message on each of the per-service-streams using the same partitionkey the publisher used
• in case of error in that smart lambda: the lambda should store the failed events somewhere, e.g DynamoDB for later investigation. But when such an error occurs, that means what is received by the consumer is not in the same order anymore as the order the publisher published the messages (the consumer misses a message! And might get it in later when the investigation decides to republish it on the central stream)
• note that this introduces at least one second delay because of the smart lambda being allowed to poll once a second. And then if the consuming service is also a lambda, another second delay is introduced.

The above solution makes sure each service-stream has only one type of application consuming and any read limits can be fixed by increasing the number of shards (because only one service will be reading from it anyway).

Note Enhanced Fanout was not released yet then, which seems to also solve the described issue.

Options tried by changing the configurations: 

1. KCL: increasing withIdleTimeBetweenReadsInMillis  and withIdleTimeBetweenCallsInMillis: only helps in a limited way plus reduces throughput.
2. KCL: same for withMaxrecords(). Even if there are no records, all services will still from time to time have to poll the shard...
3. Increasing AWS provisioned maximum could of course help but what's the final maximum there? It will probably need to be increased for every X new services.

Increasing throughput

When the consumer needs to be really fast in processing messages from a Kinesis stream shard, one can spin off a new thread per record in the batch. But several things have to be taken into account when doing that:

• checkpointing is still at record (or batch) level. So you still have to in some way checkpoint only if you know all message before that moment are processed fine too. So you'll need some thread-orchestration to determine to what record to checkpoint
• if you care about the order, then a thread per record causes processing-ordering issues; so in that case you'll have to first group all records together that apply to the same entity, then start a thread to process these as a whole 

No direct Kinesis consumer

A totally different approach is to have your container services not be bothered with Kinesis at all, and put Lambdas in front of them, which invoke the services via regular REST calls. This alleviates the consumers completely from processing messages.

An example framework that provides this is Zalando's Nakadi. Though implemented for Kafka, still a viable option. This architecture is definitely something to consider.
A more advanced solution could be to have the lambdas put the messages in an SQS queue. This provides even more decoupling and if your service is not available, the lambdas can still deliver the messages into the SQS; care has to be taken of how to support multiple consumers reading from the same SQS queue. Of course then you almost must start to wonder why you are using Kinesis when you are putting a regular queue behind it...

Kinesis Client Libray (KCL) challenges

For the Java consumers the recommended option by AWS is to use the Kinesis Client Library, KCL for short. A short high level introduction can be found here.

This library takes away a lot of standard challenges in a distributed solution. See below's KPL link to implementing efficient producers for an explanation.

But still quite a few challenges exist. E.g how does it handle multiple instances running a Worker? Should you checkpoint per batch or per record?

Are batches provided to the worker one at at time, sequentially or in parallel? How to handle failed messages?

Should workers have a unique ID?

Here's a summary of lessons learned:

• a shard will be consumed by a record processor thread (not necessarily always the same thread!) of one single worker only at any given moment. And thus batches will never be handed over to processRecords() in parallel; first batch 1, then batch 2 etc.   consumeShard() is protected via synchronized.
• starting multiple workers on same host won't improve throughput; one on each host seems most effective, otherwise Workers start maybe also stealing leases too many times/too much
• give each worker a unique ID per host so you can see which host has what lease
• seems best to pass in your own Executor to the KCL when using Spring, so Spring knows about that threadpool. 

Note that checkpointing too much (or querying your own DynamoDB database while processing each message) can easily cause a ProvisionedThroughputExceededException.

Therefore you also don't want to do that too often. In the end this is the high level algorithm (pseudo code) that one can use to cover all the above concerns:

In processRecords(records) {

   lastSuccessfulRecord = null;

   for (each record in records) {

      try {

         parse record into domain object;

         process(domain object);

         lastSuccessfulRecord = record;

      } catch (ex) {

         checkpointUpToIncluding(lastSuccessfulRecord);

      }

   checkpointBatch(); // All records in the batch successfully processed

}
Another question for next implementation will be: should the service really be doing smart checkpointing (i.e only checkpoint when record is processed successfully; if not successful, keep processing but at next restart of service start from the last checkpoint)? Because the "normal" messaging implementation of putting a message on a Dead Letter Queue might be a better solution. Though in that solution you'll have to think about edge scenarios, like when a message A to modify entity E is put on the DLQ, but a following message B related to entity E is processed successfully; should message A be replayed (published) on the main stream again? Or should as soon as one message for E is put onto the DLQ, all messages after that for E also be put on the DLQ?

Kinesis Producer Library (KPL) challenges

For the Java producers the recommended option by AWS is to use the Kinesis Producer Library, KPL for short. For key concepts see here.

See here for a thorough explanation of its advantages with example code.

The only challenge there was: how large should the batch size be?  It seems to depend mostly on how well your consumers can handle batches.

In the end I set it to values between 10 and 100.

Note that the standard AWS Kinesis SDK does not implement handling batches. That means if you use the KPL with batchsize 10, then when these batches of records are received by e.g Javascript lambdas using the standards SDK, they by default can't process them correctly because batches are not supported; unless you write code to unwrap the batch.

As a temporary workaround there the batchsize could be set to size 1 of course, which then makes sure each consumer gets to process one message at a time and not in a batch.

Hope these lessons learned help somebody while implementing Kinesis.

Setting up an Amazon AMI with Java and MySQL on EBS using the AWS Management Console

Introduction

In this post I'll describe the steps I performed to create my own Amazon Machine Image using the AWS Management Console and Windows XP as my local machine.
Just recently Amazon released their AWS Management Console. It's still in beta, but it makes life already so much easier: before it was available, you had to use quite a few scripts to get your own AMI ready.

The final AMI will have installed on it:

• Fedora Core 8
  


• 32-bit architecture
  


• Java JDK 7 (1.7.0)
  


• JEE 5
  


• Tomcat 5.5.27
  


• Apache 2.2.9
  


• MySQL 5.0.45
  

I also will setup MySQL on Elastic Block Storage such that you can shutdown the AMI and not lose your MySQL data.
In the end you should be able to deploy for example a Java .war file (if correctly assembled of course) with the following frameworks without any problem:

• Spring 2.5
  


• Hibernate 3.2.5
  


• Wicket 1.3
  


• Sitemesh 2.2.1
  


• Quartz 1.6.2
  

For the steps described below, I'm assuming you've already setup your keys and know how to start an AMI and get access to it via a browser and putty. If you don't know how to do this, on the AMC homepage there's a good introductionary video on how to do this.

Starting up an instance

You can start with a very basic AMI with only an OS installed on it, or use one that has already a lot more software installed on it.
As a starting point I used the publicly available Java Web Starter AMI. Notice that you can lookup AMIs without being logged in into AWS.
Start the instance such that you see something like this:


Check that Tomcat is running by going to the public IP of the instance. In my case I had to go to http://ec2-174-129-150-80.compute-1.amazonaws.com/. And I do see Tomcat:

Setting up MySQL and Tomcat passwords

In the basic AMI I'm using, MySQL root has no password and the Tomcat Manager login is admin/password. You don't want that in your final version, so let's change that. First login with putty to the instance (don't forget to use the ppk version of the key). You can login as root w/o a password because the key takes care of the authentication. Then change the passwords:

1. MySQL password: login to mysql:
   
   

   mysql -u root

   
   
   And execute:
   
   

   GRANT ALL ON *.* to 'root'@'localhost' IDENTIFIED BY 'mysecretpassword'.

   
   
   Note that we're only allowing root access from the localhost (the AMI itself) and of course replace the 'mysecretpassword' text for your own password. Double check that you can now login with the new password. If not, you might have to restart the AMI (all changes are lost) and try again.


2. Tomcat password:
   
   

   cd /usr/share/tomcat5/conf.

   
   
   Edit tomcat-users.xml, change the password field with value 'password' where it says username="admin" to your desired new password. Restart Tomcat to let the change take effect: /etc/init.d/tomcat5 restart. Go to your instance again with your browser and check that the new Tomcat Manager login works.

If there's anything else you like to change on your AMI, you should do it now, for as long as you don't terminate the AMI instance. Tip: Reboot is fine btw, that keeps the AMI settings!

Setting up MySQL on Amazon EC2 with Elastic Block Store

As a basis for these steps I'll be using "Running MySQL on Amazon EC2 with Elastic Block Store". The steps given in there are written prior to the Amazon AWS Management Console being available, so here I'll describe the changes necessary using the AMC. The steps in the article are also specificly for Ubuntu 8.04 LTS, but as you can see from the AMI I use as a starting point, I'm using Fedora 8. Only some of the commands differ, for the rest the steps work for both OSs. Í'll also be setting up Ext3 as filesystem, not XFS.

1. Create an EBS volume with the AMC: click on Volumes on the left and click on the 'Create Volume' button. Enter the required capacity. AFAIK as long as you're not using the datablocks, you're not charged. So I put in 25GB as size. Select a zone (don't know how much it matters which one you pick, just make sure you stay in the same zone as your AMI). This should result in something like this:
   
   
   Now attach the volume to the instance as device /dev/sdf (in the original article sdh is used) via the 'Attach Volume' button. When successful, the 'Attachment Information' status field has changed to "attached":
   
   


2. Now let's format the volume and mount it. For that I deviated from the article and followed the steps you can also see when you click on the 'Help' button on the (current) EBS Volumes page in AMC. Thus execute in your putty session:
   
   

   mkfs -t ext3 /dev/sdf

   
   
   Hit OK on any questions. Then create a directory to mount the EBS volume on. Let's use a more distintive name than '/vol'. Let's create:
   
   

   mkdir /ebsmnt

   
   
   Then mount it:
   
   

   mount /dev/sdf /ebsmnt

   
   
   Note: at my first effort, I wanted to use /mnt/data (in the Volumes 'Help' button in the AMC they also give /mnt as an example), so I created that directory. But: the bundling command you'll see below does not include the /mnt directory by default! So then the /mnt/data directory doesn't exist on the new AMI, and then the auto-mount from fstab at bootup of the AMI will always fail for that reason!
   


3. Make sure the mount is performed at startup by adding it in /etc/fstab:
   
   

   /dev/sdf /ebsmnt ext3 defaults 0 0

   
   
   


4. Backup the new config file to the EBS into its own separate directory, maybe you need it some time:
   
   

   mkdir /ebsmnt/configs rsync -a /etc/fstab /ebsmnt/configs/

   
   
   

Now let's tell MySQL to use the EBS volume to store its databases.

1. Stop MySQL:
   
   

   /etc/init.d/mysqld stop

   
   
   And to be safe:
   
   

   killall mysqld_safe

   
   
   


2. Move the existing database files. Since there isn't much yet except a couple of test databases, not much needs to be done. First let's make a separate dir for MySQL on the EBS volume:
   
   

   mkdir /ebsmnt/mysql

   
   
   Then:
   
   

   cd /ebsmnt/mysql mkdir lib log

   
   
   And start moving stuff:
   
   

   mv /var/lib/mysql /ebsmnt/mysql/lib/ mkdir /var/lib/mysql # Note that we need this dir for the mysql.sock file chown mysql:mysql /var/lib/mysql # Give it again the correct permissions mv /var/log/mysqld.log /ebsmnt/mysql/log/

   
   
   


3. Tell MySQL to look on the mounted EBS from now on. Edit /etc/my.cnf and change it as below. The '# Was: ' indicates wat was there originally:
   
   

   [mysqld] # Was: datadir=/var/lib/mysql datadir=/ebsmnt/mysql/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql # Default to using old password format for compatibility with mysql 3.x # clients (those using the mysqlclient10 compatibility package). old_passwords=1 [mysqld_safe] # Was: log-error=/var/log/mysqld.log log-error=/ebsmnt/mysql/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid

   
   
   Note that I put the log-error file also on the mount. The reason for this is that I want to have to logfile saved even when I shutdown an instance. If you don't care, you can leave it as it was.
   Another advantage I found out by practice is that when you've set the log-error like above, you can't detach the volume because MySQL is still accessing that logfile. So you can't accidentally detach, maybe leaving MySQL in an inconsistent state, which is a good thing (I mean not leaving MySQL in an inconsistent state).
   In that case you'll have to stop your instance w/o detaching it first (this makes sure MySQL shutsdown first). Update: as far as I can tell from /etc/rc.d, unmounting from fstab is done as the last thing, so doing a 'Terminate' of the instance to automatically let the shutdown sequence do the unmount and unattach of the volume should be safe.
   Note also that I didn't modify the mysql.sock socket file. Advantage is that I can keep using mysql and mysqladmin the way they are. If you would change the location of the socket file, say to /ebsmnt/mysql/lib/mysql/mysql.sock, then you will have to start mysql and mysqladmin with the '-S ' option.
   


4. Backup the new config file to the EBS, just to be safe:
   
   

   rsync -a /etc/my.cnf /mnt/configs/

   
   
   


5. Restart MySQL again:
   
   

   /etc/init.d/mysqld start

   
   
   You can check everything is going ok by tailing the logfile (on the mount of course):
   
   

   tail -f /ebsmnt/mysql/log/mysqld.log

   
   
   


6. To see later on that the data is still available after having terminated the AMI, create an example database:
   
   

   mysql -p -e 'CREATE DATABASE esb_test_database'

   
   

So now the database is setup on the EBS. If you want to create snapshots, you can do that via the AWS Console or follow the instructions in the mentioned article, which also includes automated snapshots and cleaning up everything that has been created in EBS when following the above steps (handy to know, because otherwise you'll be charged for using it until the end of time! hahahahaaa).

So now the AMI is how I want it to be. Let's store it on S3 to make it persistent.

Bundling the new Linux AMI

This part is based upon Bundling a Linux or UNIX AMI from the online EC2 Developer Guide. There's currently almost no AWS Management Console facilities to do this, so we'll have to use the AMI Tools on the host AMI.

1. Installing the AMI Tools: luckily they are already installed on the AMI we're using (how you'd think it got created? :-). Try this command to see that they are indeed installed (well of course that only proves this one is installed ;-):
   
   

   ec2-bundle-image --manual

   
   
   If you're using an AMI that doesn't have these tools installed, follow the steps in section 'Installing the AMI Tools' in the article.
   


2. Bundling an AMI Using the AMI Tools: Let's do the bundling in /tmp:
   
   

   cd /tmp

   
   
   Then to bundle execute:
   
   

   ec2-bundle-vol --prefix Fedora8-JEE5-JDK7-Tomcat55-MySQL50 -k <private_keyfile> -c <certificate_file> -u <user_id>

   
   
   Note that I specify a prefix. By default it's 'image', so you'll get 'image.manifest.xml'. When you look at all the AMIs out there, they should be more descriptive than that, so I used Fedora8-JEE5-JDK7-Tomcat55-MySQL50.
   The other parameters are explained in the article. The user_id you can find in the AMC under the 'Your Account --> Account Activity' menu.
   Your private_keyfile is the pk*.pem file, which you need to upload to the AMI (host). The same goes for the cert*.pem file. Note I put them in /mnt so they won't get put on the new AMI, which you don't want!
   An example pscp copy from Windows command prompt to get the files over would be:
   
   

   pscp -i Fedora8.ppk pk*.pem cert*.pem root@ec2-174-129-150-80.compute-1.amazonaws.com:/mnt/

   
   
   Fedora8.ppk is the filename of the key generated for putty with puttygen, as clearly described in the before-mentioned video on the AMC homepage.
   You can accept all the defaults at the prompts.
   The warnings:
   
   

   "NOTE: rsync with preservation of extended file attributes failed. Retrying rsync without attempting to preserve extended file attributes..."

   
   
   and
   
   

   "NOTE: rsync seemed successful but exited with error code 23. This probably means that your version of rsync was built against a kernel with HAVE_LUTIMES defined, although the current kernel was not built with this option enabled. The bundling process will thus ignore the error and continue bundling. If bundling completes successfully, your image should be perfectly usable. We, however, recommend that you install a version of rsync that handles this situation more elegantly. " are (apparently) no serious problem, so these can be ignored.

   
   
   Note that a file with the same name as the --prefix parameter gets created in the directory where you started the command.
   Notice from the manual of the above command, that it skips /mnt (amongst others), so in case you mounted it there, the command won't include the whole mounted EBS in the bundle (luckily)!
   Note also that it seems the script is smart enough to not include the mounted /ebsmnt data either. If you want to be really sure, umount it first before running the bundle command.
   Running the command takes about 5-10 minutes.
   The message
   
   

   "Unable to read instance meta-data for product-codes"

   
   
   at the end does not seem to be a serious problem, I haven't found any problems at least :-).
   

Uploading a Bundled AMI

Here you can just follow the steps in the article, thus:

ec2-upload-bundle -b <bucket> -m Fedora8-JEE5-JDK7-Tomcat55-MySQL50.manifest.xml -a <access_key> -s <secret_key>

The two keys should be under the menu 'Your Account --> Access Identifiers' in the AWS Mangement Console.
You can imagine being a directory in your S3 storage. Let's use "mybucket" as name.

Registering the AMI

There are two ways of registering. Via the AMC is described first. Then the old way via the Amazon EC2 API command line tools as in the article is described.

Via AWS Console

1. Go to the 'AMIs' page and click the 'Register New AMI' button.
   


2. In the popup enter the path to your above uploaded manifest file, including the bucket. So the whole path becomes:
   
   

   http://s3.amazonaws.com:80/mybucket/Fedora8-JEE5-JDK7-Tomcat55-MySQL50.manifest.xml

   
   
   Thus the popup would look something like this:
   
   
   


3. Hit 'Register' to register the AMI. Too late for this tutorial I noticed this 'Register New AMI' button, thus here my personal practical experience ends for this option. I used the command line tools as described below. But I guess the result will be the same: a registered AMI!
   

Via Amazon EC2 API command line tools

For this I used the steps described here in the AWS Developer Guide.
For this you need to download and install the Amazon EC2 API command line tools on your local machine and a Java Runtime, I just used the Java 5 JDK.
Then run:

ec2-register mybucket/Fedora8-JEE5-JDK7-Tomcat55-MySQL50.manifest.xml

where mybucket is the you specified in the above 'ec2-upload-bundle' command.
It returns the id of the created AMI, e.g.

IMAGE ami-aabb5cc3

Now check that you can find it in the AMIs 'Owned By Me' in the AMC by searching for the above AMI ID ami-aabb5cc3. And yes there it is:


If you want, you can make it public such that other users can see it etc. But I won't describe that here.

That's it! And as final step:

Let's see if the new AMI actually works

Let's see if it starts up, the EBS volume can be connected to it and the newly created test database is still there.
First terminate the running (modified) AMI. Note that you can't detach, since MySQL still uses the volume!
Start the new AMI. Now the problem is that you can't attach until Fedora OS is starting to boot. But /ebsmnt can't be mounted until it is attached! Thus MySQL can't find its data.
So what you can do is run the 'ec2-attach-volume' command until it succeeds, or check the instance status until it reaches a "specific intermediate state" and then run the ec2-attach-volume command. This way you "hope" that the attachment succeeds before the mounting. I never managed to get this working.
The safest solution seems to attach the volume when the AMI has reached status 'running', and then reboot the instance.
Of course you can put this all in scripts to automate it as much as possible.

Note that my AWS Mangement Console did not show my newly running AMI instance when trying to attach a volume via the AMC. There might be some delay in status updates(?). The command 'ec2-attach-volume' did work in that case. An example of that command on your local workstation:

ec2-attach-volume -d /dev/sdf -i i-ec8f0e85 vol-52d4303b

Also don't forget to use the same zone for the volume and the instance, otherwise you also might not see the running instance in the 'Attach' popup window.

Done

Did you notice BTW what a large amount of memory you get, even for a small AMI instance:

Mem: 1747764k total, 176724k used, 1571040k free, 12236k buffers Swap: 917496k total, 0k used, 917496k free, 64340k cached

Tip: for managing your AMI and other data that you (want to) store on Simple Storage Service S3, you can either use the more manual tools from Amazon or the great graphical Amazon S3 Firefox Organizer (S3Fox) Firefox plugin.

Best of this Week Summary 06 October - 12 October 2008

• One of the many nice editions of The A-Z of Programming Languages series, this time on C#. Yes, yes, that word is not supposed to be on this blog, but this article is still interesting if you're into programming languages. For example the challenges when designing a language. Check also the other editions, for example about: AWK, Forth, Modula-3, Python and Javascript.
  


• An Amazon S3 introduction/beginners guide for setting up hosting of images.
  


• Four cases of CSRF attacks desribed, including one were it was possible to transfer money from a bank account! Check also the mentioned paper on how to prevent them: Cross-Site Request Forgeries: Exploitation and Prevention. The paper explains how to prevent CSRF in your web-application/frameworks: don't let a GET modify anything and pass a random number in the cookie and each form POST. It also describes XSS and the same-origine policy.
  

Best of this Week Summary 18 August - 24 August 2008

• Patterns-based Evaluation of Open Source BPM Systems: jBPM, OpenWFE, and Enhydra Shark. Report's conclusion: "Overall one can conclude that the open source systems are geared more towards developers than business analysts. If one is proficient with Java, jBPM may be a good choice, although if not, choosing jBPM is less advisable. Similarly, whilst OpenWFE has a powerful language for workflow specification in terms of its support for the workflow patterns, we postulate that it will be difficult to understand by non-programmers. Finally, Endydra Shark’s minimalistic support for the workflow patterns may require complicated work-arounds for capturing nontrivial business scenarios."
  


• As part of their EC2 offering, Amazon introduced this week their Elastic Block Store, enabling you to mount an EBS and format it or setup a database on it. S3 is not really intended for database-like storage. Here's another introduction, including a link to an article on how to setup MySQL on EBS. Here's ESB explained, including some diagrams.
  


• Article on the succesful execution of a large project (20 man years, 100.000+ lines of code) with an Agile approach using Scrum, with developers from India and The Netherlands. Includes lessons learned. Technologies used: Java, Spring, Hibernate, WebLogic, Oracle, Swing (UI) and Flex (for the displays) and Bamboo (continuous integration).
  


• Handy short comparison between LWUIT (recently open sourced by Sun) and JavaFX Mobile. Here's another comparison from Sun.
  

Best of this Week Summary 28 July - 03 August 2008

• Introduction from google on OAuth and why you should/might want to use oAuth, including a few examples.
  


• A bit more than a week ago (July 20th to be exact), Amazon's S3 had a big outage of several hours, affecting many (startup) websites. Here's the explanation why, from Amazon. The short story: message corruption in server-to-server communication.
  


• In this blog I of course also take care of the starting software engineer :-), so here's ten concepts every software engineer should know.
  


• MySpace is now also supporting OpenID.
  


• Finally, Sun introduced the JavaFX preview SDK. Official announcement here. Below a screenshot of the demo application:
  
  
• In case you're not using it yet, definitely worth checking out this opensource LGPL static Java analyzing tool Findbugs.
  

Best of this Week Summary 21 April - 27 April 2008

• How Yahoo! is transforming itself into an open and social platform, allowing developers access to internal assets of Yahoo. A first example of this is SearchMonkey. Below a screenshot of their new architecture Yahoo Open Strategy.
  
  
  
• Two initiatives that could improve the chances of OpenID becoming more mainstream: IDSelector (also mentioned here) and Confident's RecognitionAUTH, which you can see implemented at MyVidoop (mentioned in a previous post).
  


• Amazon announced its Fulfillment Web Service API, which allows merchants to automatically store inventory in Amazon's warehouses and ship orders to customers. Again, Amazon is one of the frontrunners in this area...
  

Best of this Week Summary 7 April - 12 April 2008

• 12 great points you should know about WOA and REST. Note that these points are not all about technology only...
  


• Difference between WOA and SOA shortly discussed.
  


• New from Google: run your Python webapplication on their infrastructure with Google App Engine. Quite a contrast to Amazon's webservices approach S3, EC2 and SimpleDB.
  
• A status check on OpenID, including where you can get an OpenID (providers), resources and where you can use it (couple of major sites only).
  

Best of this Week Summary 11 February - 17 February 2008

• Jaxer is an interesting new approach to development that brings Javascript, DOM, HTML and CSS to the server! An example of its use is that it helps you to reuse your validation logic on client and server. Here's some hands-on experiences.
  
  
  
• TrustBearer is connecting the (hard) real world with the (software) virtual world for OpenID authentication via hardware token, smart card or biometric reader.
  


• Some experiences from using Amazon's EC2. Check also the comments. This week though EC2 and S3 were not performing as they should be. Amazon gave an explanation for the outage (in short: an accidental DDOS).
  

Best of this Week Summary 10 December - 15 December 2007

• Good summary of somebody using GWT for three months and the pros and cons found.
  


• Short intro to Amazon's new offering SimpleDB, which is its third offering besides S3 (Simple Storage Service) and EC2 (Elastic Compute Cloud). It provides "a simple web services interface to create and store multiple data sets, query your data easily, and return the results".
  


• Nice set of best programming practices for Spring.
  

Best of this Week Summary 6 Nov - 11 Nov 2007

• This is a good blog to get you started on JavaFX and related technologies. See this post on what's been covered until now.
  


• Madly interesting is this technical posting about Amazon's Dynamo, which is a their internal distributed storage system in which the data is stored and looked up via a key, with a put() and get() interface. Sounds quite similar to the put() and get() for a Hashtable in Java right? ;-) Actually Dynamo is built in Java, so I guess that's no coincidence! The posting gives you quite some details on Amazon's internal infrastructure, and introduces interesting new terms like that it is an "eventually consistent storage system". What is also cool is that each of Amazon's internal applications can setup their own SLA with Dynamo. This SLA defines the amount of delays and data discrepancy the application will tolerate from Dynamo. The fact that Amazon is opening up its services (with S3 and EC2), makes it a huge differentiator from companies like Google and Microsoft, which don't open up their systems (some Google GFS info you can find here). Related to this the new term that is being coined recently: HaaS (Hardware as a Service). No time to read the whole paper? A summary you can find here. Compare it with Hadoop and CouchDB.
  


• Related to my last week's post about OpenSocial, this week the (very) alpha version 0.5 of the Container API has been released.
  


• Note: I turned on moderation for comments this week because of a big spamming "effort"... Thanks whoever you are...