Spring @Scheduled using DynamoDB AWS X-Ray throws SegmentNotFoundException: failed to begin subsegment

Introduction

AWS X-Ray is designed to automatically intercept incoming web requests, see at this introduction.  And also here. 

But when you start your own thread (either via a Runnable or plain new Thread() or @Scheduled), X-Ray cannot initialise itself: there are no web requests to intercept for it. Then it throws an exception like this: 

Suppressing AWS X-Ray context missing exception (SegmentNotFoundException): Failed to begin subsegment named 'AmazonDynamoDBv2': segment cannot be found.

In the above example the distributed DynamoDB Lock Client was used, which uses DDB in its implementation for acquiring and releasing a lock.

Regular web requests were not throwing this X-Ray exception.

Investigation

Amazon explains that crucial bit of knowledge that web requests are "automagically" setting up the X-Ray recorder a bit here. 

But that was not fully explaining it with an example. E.g only adding 

AWSXRay.beginSubsegment("AmazonDynamoDBv2") 

(and ending it) but that didn't fix it. That then gave this exception:

Suppressing AWS X-Ray context missing exception (SubsegmentNotFoundException): Failed to end subsegment: subsegment cannot be found.

My suspicion here is that the lock-client already closed the exactly same named subsegment "AmazonDynamoDBv2".

Also, I'm not creating any worker thread myself, Spring is doing it for me.

Note that you at least can avoid exceptions be thrown by setting environment variable AWS_XRAY_CONTEXT_MISSING   to LOG_ERROR. That will only log the above exception.

Solution

Creating a 'parent' segment and setting the trace entity and the subsegment did the job:

Entity parentSegment = AWSXRay.beginSegment("beginSegmentForSomeScheduledTask");
AWSXRay.getGlobalRecorder().setTraceEntity(parentSegment);
AWSXRay.beginSubsegment("AmazonDynamoDBv2");

I did test only creating the subsegment and only creating and setting the parentSegment. But those raised the exception again.

I did not further investigate whether the "AmazonDynamoDBv2" name of the subsegment is essential.

And of course the matching closeSegment() and closeSubsegment() calls of course; not the reverse order: close the last one begun as first.

This thread pointed me in the right direction. This was a next option I would have tried next: setting up my own filter to run it earlier in the (filter) chain; though the @Scheduled task of course does not have a filter. Another workaround would have been to put the X-Ray logging at a very high level:

logging.level.com.amazonaws.xray = SEVERE

Also helped in explaining is this X-Ray reported issue.

Update: additionally, the error was also due to DynamoDB Lock Client! I did not specify withCreateHeartbeatBackgroundThread() when creating the lock client, but the exception of X-Ray showed that it was trying to send a heartbeat. After explicitly setting withCreateHeartbeatBackgroundThread(false) the exception (and error) regarding segment cannot found was fully fixed.

PACT provider test with Serverless Java 11 Lambda and JUnit 5 (Jupiter)

Introduction

This blogpost shows how to create a Pact provider test using:

1. Pact 4.0.0
2. AWS Serverless Lambda
3. Java 11
4. JUnit 5 (Jupiter)

Example Provider Test

Below is the example test class. Note how a mock service is used to simulate the endpoint defined (also) in serverless.yml.

package com.ttlnews.pact.tests.provider;

import au.com.dius.pact.provider.junit.Provider;
import au.com.dius.pact.provider.junit.State;
import au.com.dius.pact.provider.junit.loader.PactBroker;
import au.com.dius.pact.provider.junit.loader.PactBrokerAuth;
import au.com.dius.pact.provider.junit5.HttpTestTarget;
import au.com.dius.pact.provider.junit5.PactVerificationContext;
import au.com.dius.pact.provider.junit5.PactVerificationInvocationContextProvider;
import com.amazonaws.services.lambda.runtime.Context;
import lombok.extern.slf4j.Slf4j;
import net.jcip.annotations.NotThreadSafe;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.TestTemplate;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockserver.integration.ClientAndServer;

import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.mockserver.integration.ClientAndServer.startClientAndServer;
import static org.mockserver.matchers.Times.exactly;
import static org.mockserver.model.Header.header;
import static org.mockserver.model.HttpRequest.request;
import static org.mockserver.model.HttpResponse.response;
import static org.mockserver.model.JsonBody.json;

/**
 * Java 11 serverless provider Pact contract provider implementation using Pact with Junit5 (Jupiter) for provider SERVICE_A.
 * 
 * Dependencies needed:
 *
     <dependency>
     <groupId>au.com.dius</groupId>
     <artifactId>pact-jvm-consumer-junit5</artifactId>
     <version>4.0.0</version>
     <scope>test</scope>
     </dependency>
    
     <dependency>
     <groupId>au.com.dius</groupId>
     <artifactId>pact-jvm-provider-junit5</artifactId>
     <version>4.0.0</version>
     <scope>test</scope>
     </dependency>
 *
 * 
 */
@Provider(SERVICE_A)
// Below environment variables must be set when running this test
@PactBroker(host = "${PACT_HOST}",
        authentication = @PactBrokerAuth(username = "${PACT_HOST_USERNAME}", password = "${PACT_HOST_PASSWD}"))
@NotThreadSafe // Pact contract tests can't seem to handle methods running in parallel, so prevent maven failsafe/surefire plugin to run Pact tests in parallel
@Slf4j
public class ServiceAProviderTest {

    private static final String SERVICE_A = "service-A";
    
    private Context lambdaContext;
    private SomeRepository someRepository;

    private ClientAndServer mockClientAndServer;

    @BeforeEach
    void before(PactVerificationContext context) {

        lambdaContext = mock(Context.class);
        someRepository = new InMemoryMockedRepo();

        mockClientAndServer = startClientAndServer(8888);

        context.setTarget(new HttpTestTarget("127.0.0.1", 8888));

    }

    @AfterEach
    void stopServer() {
        mockClientAndServer.stop();
    }

    // This triggers the defined contract test(s) at the host PACT_HOST
    @TestTemplate
    @ExtendWith(PactVerificationInvocationContextProvider.class)
    void pactVerificationTestTemplate(PactVerificationContext context) {
        context.verifyInteraction();
    }

    @State("Create a new resource A")
    public void shouldCreateResourceA() {

        log.info("Set up state");

        // Given
        // Maybe some more mocking of 'lambdaContext' needed, depending on your case
        
        // Lambda being tested
        CreateResourceALambda createResourceALambda = new CreateResourceA(someRepository);

        CreateResourceALambdaRequest createResourceALambdaRequest = CreateResourceALambdaRequest.builder()
                .someValue("25")
                .build();

        // When
        final LambdaResult lambdaResult = createResourceALambda.executeRequest(createResourceALambdaRequest, lambdaContext);
        final String body = lambdaResult.getBody();
        assertNotNull(body);

        // Prepare the mockserver to return what the lambda returns when it is invoked (set up above in the 'body' variable) 
        // Of course the path to this CreateResourceALambda is defined in the serverless.yml, but that we can't access now. So need to repeat that
        // endpoint here.
        mockClientAndServer.when(
                request()
                        .withMethod("POST")
                        .withPath("/resources/")
                        .withHeaders(

                                header("x-request-trace-id"),  // Any value is fine
                                header("Authorization"),  // Any value is fine
                                header("Content-Type", "application/json")

                        )
                        .withBody(json("{someValue: '25'}")) // Indeed need to use this strange JSON format
                ,
                exactly(1))
                .respond(
                        // Response as defined by the matching Pact consumer test; in this case found at host PACT_HOST
                        response()
                                .withStatusCode(201)
                                .withHeaders(
                                        header("Content-type", "application/json; charset=utf-8"),
                                        header("Authorization") // Any value is fine
                                )
                                .withBody(body)
                );
    }
}

Migrating from your Pact tests from JUnit4 to Junit5 can be found here.

Note to self: use https://www.opinionatedgeek.com/codecs/htmlencoder to encode code, open the HTML view, and then wrap the code in <pre> open en close tag.