Datadog: Malformed _X_AMZN_TRACE_ID value Root - also known as X-Amzn-Trace-Id

Introduction

Since 14 March 2023 suddenly my AWS lambdas started to log this error:

datadog: Malformed _X_AMZN_TRACE_ID value: Root=1-6411cb3d-e6a0db584029dba86a594b7e;Parent=8c34f5ad8f92d510;Sampled=0;Lineage=f627d632:0

Note that the lambda processing was finishing normally, this metrics logging to Datadog is happening apparently in the background.

Setup

• AWS lambda
• Datadog
• Gradle dependency library implementation "com.datadoghq:datadog-lambda-java:1.4.5"

Investigation

After a lot of searching, I found out that the datadog-lambda-java library was causing the issue, since that same day the issue was reported here in its Github repository.

Which also seems to point to the code that is the culprit: the length != 3 is assuming that the trace field will always consist of exactly 3 parts. But its specifications allow for more, so it seems AWS added another part.

The definition of the header can be found here and here where the examples still have 3 elements (parts separated by a ';'), but can now be 4.

Solution

A patch has been posted, but as the 3rd comment says, the library is deprecated anyway. Here is the upgrade guide.

 

UPDATE 24 March 2023: the patch has been applied and a new release has been made! See https://github.com/DataDog/datadog-lambda-java/pull/90 for the 1.4.10 release.