Datadog: Malformed _X_AMZN_TRACE_ID value Root - also known as X-Amzn-Trace-Id

Introduction

Since 14 March 2023 suddenly my AWS lambdas started to log this error:

datadog: Malformed _X_AMZN_TRACE_ID value: Root=1-6411cb3d-e6a0db584029dba86a594b7e;Parent=8c34f5ad8f92d510;Sampled=0;Lineage=f627d632:0

Note that the lambda processing was finishing normally, this metrics logging to Datadog is happening apparently in the background.

Setup

• AWS lambda
• Datadog
• Gradle dependency library implementation "com.datadoghq:datadog-lambda-java:1.4.5"

Investigation

After a lot of searching, I found out that the datadog-lambda-java library was causing the issue, since that same day the issue was reported here in its Github repository.

Which also seems to point to the code that is the culprit: the length != 3 is assuming that the trace field will always consist of exactly 3 parts. But its specifications allow for more, so it seems AWS added another part.

The definition of the header can be found here and here where the examples still have 3 elements (parts separated by a ';'), but can now be 4.

Solution

A patch has been posted, but as the 3rd comment says, the library is deprecated anyway. Here is the upgrade guide.

 

UPDATE 24 March 2023: the patch has been applied and a new release has been made! See https://github.com/DataDog/datadog-lambda-java/pull/90 for the 1.4.10 release.

 

 

 

Docker build with Git command running in CircleCI failing with: Fatal: No names found, cannot describe anything, invalid argument, for "-t, --tag" flag: invalid reference format

Introduction

Context: Docker, CircleCI, Github.

The Docker build command 

docker build -f .circleci/Dockerfile -t $AWS_ACCOUNT_ID.ecr.$AWS_DEFAULT_REGION.amazonaws.com/${CIRCLE_PROJECT_REPONAME}:`git describe --tags` -t $AWS_ACCOUNT_ID.ecr.$AWS_DEFAULT_REGION.amazonaws.com/${CIRCLE_PROJECT_REPONAME}:${CIRCLE_BUILD_NUM} -t $AWS_ACCOUNT_ID.ecr.$AWS_DEFAULT_REGION.amazonaws.com/${CIRCLE_PROJECT_REPONAME}:${CIRCLE_SHA1} .

was failing with this message:

fatal: No names found, cannot describe anything.

invalid argument "********************************************/my-project:" for "-t, --tag" flag: invalid reference format

See 'docker build --help'.

Exited with code exit status 125

Solution

You'd expect the Docker command maybe being syntactically incorrect. But the error message is referring to something else: it turns out the git describe --tags command gave the fatal message.

The cause was that there was no git-tag set at all on the Github project yet.   After manually creating a release (including a tag) on Github and running the above build command again, the docker build command succeeded.

Gitlab maven:3-jdk-8 Cannot get the revision information from the scm repository, cannot run program "git" in directory error=2, No such file or directory

Introduction

In a Gitlab project setup the most recent Docker image for maven was always retrieved from the internet before each run by having specified image: maven:3-jdk-8 in the gitlab-ci.yml. The image details can be found here.

Of course this is not a best-practice; your build can suddenly start failing at a certain point because an update to the image might have something changed internally causing things to fail.

What you want is controlled updates. That way you can anticipate on builds failing and plan the upgrades in your schedule.

The issue and workarounds/solutions

And indeed suddenly on March 29 2018 our builds started failing with this error:

[ERROR] Failed to execute goal org.codehaus.mojo:buildnumber-maven-plugin:1.4:create (useLastCommittedRevision) on project abc: Cannot get the revision information from the scm repository :
[ERROR] Exception while executing SCM command.: Error while executing command. Error while executing process. Cannot run program "git" (in directory "/builds/xyz"): error=2, No such file or directory

That message is quite unclear: is git missing? Or is the directory wrong? Or could the maven buildnumber plugin not find the SCM repository?

After lots of investigation it turned out the maven:3-jdk-8 image indeed had changed about 18 hours before.

And after running the maven command in a local version of that Docker image indeed the same error occured!  Awesome, the error was reproducable.

And after installing git again in the image with:

- apt-get update
- apt-get install git -y

the error disappeared!  But a new one appeared:

[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?

This also hadn't happened before. After some searching it turned out it might be the surefire and failsafe plugins being outdated.

So I updated them to 2.21.0 and indeed the build succeeded.

Here's the issue reported in the Docker Maven github. UPDATE: it is caused by an openjdk issue (on which the maven:3-jdk-8 is based upon.

This issue made us realize we really need an internal Docker repository. And so we implemented that :)

One disadvantage about Docker images is that you can't specify a commit hash to use. Yes you can specify a digest instead of a tag, but that is a unique UUID hashcode only. You can't see from that hashcode anymore the (related) tagname.

Vaadin error: A connector with id 7 is already registered!

This post describes a short analysis and possible solution when getting the error java.lang.RuntimeException: A connector with id 7 is already registered!

Setup

• Vaadin 7.0.4 with Spring using the SpringVaadinIntegration plugin 1.6.5 and Shiro 1.2.1
• The UI class has @Component en @Scope("prototype") as annotations
• All components extend CustomComponent
• All components but one component also have the @Scope("prototype") annotation
• One component, let's call that the HelpWindow, it extends Window, has no @Scope annotation, and thus is a Spring singleton. The reason for it being a singleton is that it seemed the only solution to always have a sub-window on top of another sub-window (using UI.addWindow()). At least one situation in the project made the sub-window appear beneath another sub-window sometimes...
• No @VaadinView used anywhere.
• Spring 3.1.2
• Tomcat 7 as application server.

Scenario

This is the reproducible scenario that causes the A connector with id 7 is already registered! error to appear: 

1. Log in for first time, so authentication is via Shiro: all fine, the HelpWindow is shown.
2. Log out.
3. Log in again. I can see the UI constructor and its init() getting invoked.
4. Exception in the logfile:
   
   java.lang.RuntimeException: A connector with id 7 is already registered!
   ....... stuff deleted.....
   SEVERE: 
   
   java.lang.RuntimeException: A connector with id 7 is already registered!
   at com.vaadin.ui.ConnectorTracker.registerConnector(ConnectorTracker.java:131)
   at com.vaadin.server.AbstractClientConnector.attach(AbstractClientConnector.java:599)
   at com.vaadin.ui.AbstractComponent.attach(AbstractComponent.java:554)
   at com.vaadin.ui.Label.attach(Label.java:430)
   at com.vaadin.server.AbstractClientConnector.setParent(AbstractClientConnector.java:586)
   at com.vaadin.ui.AbstractComponent.setParent(AbstractComponent.java:457)
   at com.vaadin.ui.Table.registerComponent(Table.java:2350)
   at com.vaadin.ui.Table.parseItemIdToCells(Table.java:2337)
   at com.vaadin.ui.Table.getVisibleCellsNoCache(Table.java:2147)
   at com.vaadin.ui.Table.refreshRenderedCells(Table.java:1668)
   at com.vaadin.ui.Table.getVisibleCells(Table.java:3921)
   at com.vaadin.ui.Table.beforeClientResponse(Table.java:3155)
   at com.vaadin.server.AbstractCommunicationManager.writeUidlResponse(AbstractCommunicationManager.java:799)
   at com.vaadin.server.AbstractCommunicationManager.paintAfterVariableChanges(AbstractCommunicationManager.java:728)
   at com.vaadin.server.AbstractCommunicationManager.handleUidlRequest(AbstractCommunicationManager.java:599)
   at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:315)
   at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:201)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
   at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
   at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
   at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
   at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
   at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
   at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
   at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
   at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
   at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
   at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
   at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
   at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:722)
   
   The exception occurs at the moment UI.addWindow(helpWindow) is invoked (not included in this exception-dump).
5. After this, even appending ?restartApplication to the URL won't save you anymore. Only a full restart of the application server (Tomcat) fixes the problem.
6. BUT: if after the log out I press ctrl-F5 (so full browser refresh) once, the problem still occurs after logging in again.
7. BUT: if after the log out I press ctrl-F5 twice, the problem does not occur! All works fine. 

Investigation and Solution

So very strange that Vaadin's ConnectorTracker thinks/sees the HelpWindow's connectorId already registered, because the user logged out and the UI's constructor and its init() method were invoked, so a completely new UI is created.

After pressing ctrl-F5 twice, I did see that the HelpWindow's connectorId is cleared (null).

Also changing the HelpWindow's scope to prototype fixed the problem, no exception; but prototype isn't an option for the project for above mentioned reason.

So something somewhere doesn't get fully cleared at logout time. Potentially the reasoning could be: since the HelpWindow is a singleton, Spring won't re-instantiate it (it's a singleton), Vaadin won't clear its connectorId either for some reason, so the ConnectorTracker (see the stacktrace) detects that it is already registered --> exception!

Several posts on the Vaadin forum indicated that the singleton vs prototype might be the problem. Like this one which indicates a potential problem right there. And in this one another caching-like problem is mentioned.

Also not "everything" getting cleared at logout was pointed out as a potential problem. So for that, the code invoked when logging out I changed from only doing currentUser.logout() and the setLocation() to:

// currentUser.logout() might also clear the Vaadin session, but better safe than sorry...

UI.getCurrent().getSession().close();

UI.getCurrent().getSession().getService().closeSession(VaadinSession.getCurrent());

UI.getCurrent().close();

Subject currentUser = SecurityUtils.getSubject(); // Shiro

currentUser.logout(); // Shiro

// It's apparently essential to do the redirect too

UI.getCurrent().getPage().setLocation(VaadinServlet.getCurrent().getServletContext().getContextPath());

For readability, the try/catch around each statement is omitted. And, as mentioned in the first comment, currentUser.logout() might already be sufficient to clear the session, did not further investigate this.

Together with a colleague another solution came to mind: try @Scope("request")! Using that as scope, within an HTTP request the HelpWindow bean exists only once, but a new one is created for each new HTTP request.

And yes, that did it! And it also keeps the 'sub-window on top of a sub-window' problem away.

Sufficient work-around for now!

PS: note that there might be something not 100% ok regarding scope in the SpringVaadinIntegration add-on; see this discussion on what the scope of the UI should be. The last comment (from the user pointing out this potential issue) also refers to using this post as a reference to integrating Spring with Vaadin....

Tip: talking about injection/CDI: integration Vaadin with JEE 6? This might be a good one to read.