Spring @Scheduled using DynamoDB AWS X-Ray throws SegmentNotFoundException: failed to begin subsegment

Introduction

AWS X-Ray is designed to automatically intercept incoming web requests, see at this introduction.  And also here. 

But when you start your own thread (either via a Runnable or plain new Thread() or @Scheduled), X-Ray cannot initialise itself: there are no web requests to intercept for it. Then it throws an exception like this: 

Suppressing AWS X-Ray context missing exception (SegmentNotFoundException): Failed to begin subsegment named 'AmazonDynamoDBv2': segment cannot be found.

In the above example the distributed DynamoDB Lock Client was used, which uses DDB in its implementation for acquiring and releasing a lock.

Regular web requests were not throwing this X-Ray exception.

Investigation

Amazon explains that crucial bit of knowledge that web requests are "automagically" setting up the X-Ray recorder a bit here. 

But that was not fully explaining it with an example. E.g only adding 

AWSXRay.beginSubsegment("AmazonDynamoDBv2") 

(and ending it) but that didn't fix it. That then gave this exception:

Suppressing AWS X-Ray context missing exception (SubsegmentNotFoundException): Failed to end subsegment: subsegment cannot be found.

My suspicion here is that the lock-client already closed the exactly same named subsegment "AmazonDynamoDBv2".

Also, I'm not creating any worker thread myself, Spring is doing it for me.

Note that you at least can avoid exceptions be thrown by setting environment variable AWS_XRAY_CONTEXT_MISSING   to LOG_ERROR. That will only log the above exception.

Solution

Creating a 'parent' segment and setting the trace entity and the subsegment did the job:

Entity parentSegment = AWSXRay.beginSegment("beginSegmentForSomeScheduledTask");
AWSXRay.getGlobalRecorder().setTraceEntity(parentSegment);
AWSXRay.beginSubsegment("AmazonDynamoDBv2");

I did test only creating the subsegment and only creating and setting the parentSegment. But those raised the exception again.

I did not further investigate whether the "AmazonDynamoDBv2" name of the subsegment is essential.

And of course the matching closeSegment() and closeSubsegment() calls of course; not the reverse order: close the last one begun as first.

This thread pointed me in the right direction. This was a next option I would have tried next: setting up my own filter to run it earlier in the (filter) chain; though the @Scheduled task of course does not have a filter. Another workaround would have been to put the X-Ray logging at a very high level:

logging.level.com.amazonaws.xray = SEVERE

Also helped in explaining is this X-Ray reported issue.

Update: additionally, the error was also due to DynamoDB Lock Client! I did not specify withCreateHeartbeatBackgroundThread() when creating the lock client, but the exception of X-Ray showed that it was trying to send a heartbeat. After explicitly setting withCreateHeartbeatBackgroundThread(false) the exception (and error) regarding segment cannot found was fully fixed.

Recreate Vagrant box when original .box is gone from vagrantcloud

Introduction

Suppose you've setup a cool Vagrant image based upon a box from the Vagrant Hashicorp cloud. But suddenly one day the URL is gone! So the original box is gone. E.g this one, which gives a 404 page: https://atlas.hashicorp.com/box-cutter/boxes/ubuntu1404-desktop
Your 'vagrant up' still works, vagrant will just complain it can't find the box anymore, but the VirtualBox image will start up fine.

But you are still getting new team-members on board, and you want them to use the Vagrant setup you created; but the URL is gone so the base box can't be downloaded anymore. What to do?

Well, it turns out you can recreate the .box file which the above URL is pointing to.

This post describes how.

Steps

1. Locate where the Vagrant box from the URL is stored on your local machine. On Windows 7 it is C:/Users/ttlnews/.vagrant.d/boxes 
   At this location are all the base Vagrant boxes downloaded. So without any changes/provisioning made to them. The exact directory of the above ubuntu1404-desktop URL is: C:/Users/ttlnews/.vagrant.d/boxes/box-cutter-VAGRANTSLASH-ubuntu1404-desktop
   
   
2. Create a zip of the deepest directory in there. In my case it I 7zipped all the files in C:/Users/ttlnews/.vagrant.d/boxes/box-cutter-VAGRANTSLASH-ubuntu1404-desktop/1.0.11/virtualbox So not the directory, just the files, like the box.ovf, metadata.json, *.vmdk, Vagrantfile. The resulting file was: box-cutter-VAGRANTSLASH-ubuntu1404-desktop.box.7z
   
   This is what it should look like in the .7z file:
   
   
   
   
   
3. Rename the file to box-cutter-VAGRANTSLASH-ubuntu1404-desktop.box.
   
   
4. Put it in c:/tmp or wherever you want; make sure to change accordingly in the below steps.
   
   
5. In the directory where you have your (already existing because you have created it successfully before from the now-defunct box URL) Vagrantfile add the following line below config.vm.box:
   
   config.vm.box_url = "file:///c:/tmp/box-cutter-VAGRANTSLASH-ubuntu1404-desktop.box"
   
   Tip 1: when testing this out, it's safer to copy your Vagrantfile + whatever else you had to a new directory, so you won't mess up your working Vagrant setup.
   
   Tip 2: in that new setup from tip 1, also change the value of config.vm.box in the Vagrantfile, so you can clearly identify this new image.
   
   
6. Run 'vagrant up' and you should see it copy (extract) the files, start a VirtualBox and start provisioning the box!

Main source for this solution: https://docs.vagrantup.com/v2/boxes/format.html