Unknown application error occurred Runtime.Unknown - Startup issue AWS Serverless Lambda

Introduction

Trying to invoke a deployed AWS Serverless Lambda on AWS, I was getting this error CloudWatch when trying to invoke the lambda via an SQS event, published by another service in my landscape:

 

2023-08-15T15:20:44.047+02:00 START RequestId: ab924ff5-236c-5b09-8a29-12a0b9447e41 Version: $LATEST
2023-08-15T15:20:45.223+02:00 Unknown application error occurred
  Runtime.Unknown
  Unknown application error occurred Runtime.Unknown
2023-08-15T15:20:45.223+02:00 END RequestId: ab924ff5-236c-5b09-8a29-12a0b9447e41
 

That's all. No more details. Nothing appeared in Datadog to which my CW logging is forwarded to. But the lambda ran fine when running it locally in IntelliJ using the SAM AWS Toolkit, with me logged in with my IAM role.

Adding logging or a try/catch wouldn't do anything, since this error appears already before the lambda even gets invoked.

 

Setup:

• AWS Serverless Lambda
• IAM
• IntelliJ AWS Toolkit
• Kotlin 1.8.10
• CloudWatch
• Datadog
• AWS Parameter Store
• KMS
• SSM
• SQS
   

 

Solution

Then I tried to trigger the lambda via the AWS console by manually creating the SQS event and sending it on the SQS queue the lambda is listening to. There I did get the details of the error shown:

{
  "errorMessage": "User: arn:aws:sts::100004:assumed-role/my-lambda-role-acc/my-lambda is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:eu-west-1:100004:parameter/abc/apikey because no identity-based policy allows the ssm:GetParameter action (Service: AWSSimpleSystemsManagement; Status Code: 400; Error Code: AccessDeniedException; Request ID: 657c62f2-3527-42e0-8ee4-xxxxxxxx; Proxy: null)",
  "errorType": "com.amazonaws.services.simplesystemsmanagement.model.AWSSimpleSystemsManagementException"
} 

 

See this screenshot: 

 

 

The reason it worked locally is probably because there I'm logged in with a different IAM account (with more permissions) than when the lambda is deployed in the AWS cloud.

Then after fixing that by adding the path abc/apikey to the key as resource, I got this error:
{
  "errorMessage": "User: arn:aws:sts::100004:assumed-role/my-lambda-role-acc/my-lambda
is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:eu-west-1:100004:key/ff841b70-5038-6f0b-8621-xxxxxx because no identity-based policy
allows the kms:Decrypt action (Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException; Request ID: aaa5a8d0-d26e-5051-7ac0-xxxxxxxx; Proxy: null)
(Service: AWSSimpleSystemsManagement; Status Code: 400; Error Code: AccessDeniedException; Request ID: f807b8d7-826e-4d4c-9b5c-xxxxxxx; Proxy: null)",
  "errorType": "com.amazonaws.services.simplesystemsmanagement.model.AWSSimpleSystemsManagementException"
}

So the KMS decrypt is not allowed (no permission for) on that specific AWS Parameter Store entry abc/apikey.

The fix here was to add the action for the correct resource, see the items in bold:

  statement {
    sid    = "AllowReadingParameterStoreParameters"
    effect = "Allow"

    actions = [
      "ssm:DescribeParameters",
      "ssm:GetParameter",
      "kms:Decrypt"          
    ]

    resources = [
      "arn:aws:ssm:eu-west-1:100004:parameter/abc/apikey",
      "arn:aws:kms:eu-west-1:100004:key/*" 
    ]
  }

Note that the error gave away already a little on how to name that resource. Be aware that this way you potentially give more Decrypt access than you want...

Misc:

Other tips to try if you ever have this Runtime.Unknown error (but did not try): Instrumenting Java code in AWS Lambda.
And some more generic tips for troubleshooting during/before invocation.
And while executing: https://docs.aws.amazon.com/lambda/latest/dg/troubleshooting-execution.html

 

AWS SAM CLI FileNotFoundError: WinError 3: The system cannot find the path specified .class class Kotlin 1.7 Windows 10

Introduction

The AWS SAM CLI command 

sam.cmd build MyFunction --template C:\techie\workspace\my-function\local\template.yaml --build-dir C:\techie\workspace\my-function\local\.aws-sam\build --debug 

fails in an IntelliJ commandline terminal due to this 

FileNotFoundError: [WinError 3] The system cannot find the path specified 

error.

Setup

- Windows 10 Pro laptop

- IntelliJ 2022

- Kotlin 1.7

- Java 11 at least for compilation

- Serverless lambda written in Kotlin

- AWS SAM CLI, version 1.67.0

- AWS Toolkit plugin for IntelliJ 

Investigation

First I tried to install SAM AWS CLI using HomeBrew (formerly Brew) in WSL 1 (ubuntu) under Windows 10 using these steps: 

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-homebrew.html

But that failed during Homebrew installation. Probably upgrading to WSL 2 would fix that. But then I also realized: IntelliJ then doesn't know about that at all, since SAM CLI is then installed in WSL, not in Windows.

It kept on saying after running brew postinstall --verbose --debug gcc: 

==> Installing aws-sam-cli from aws/tap

Error: An exception occurred within a child process:

  Errno::EFAULT: Bad address - /home/linuxbrew/.linuxbrew/bin/gcc-12

And also:

Warning: The post-install step did not complete successfully

You can try again using:

  brew postinstall gcc

Also trying brew postinstall --verbose --debug gcc didn't succeed.  This error mentioned here was also applicable: https://github.com/orgs/Homebrew/discussions/4052

I also didn't dare wsl --update because other configurations I already had set up might fail after that. Guess I will do that at a more quiet moment :)

So then I went for the manual installation in Windows, as found here: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html

In IntelliJ you then have to set the path to the executable to where you installed it:

So IntelliJ will use the Windows AWS SAM CLI, not the one in the terminal (WSL 1).

Than I ran my command, first outside IntelliJ to be able to control the parameters more easily:

C:\Users\techie>C:\Amazon\AWSSAMCLI\bin\sam.cmd build MyFunction --template C:\techie\workspace\my-function\local\template.yaml --build-dir C:\techie\workspace\my-function\local\.aws-sam\build --debug

But that gave this error:

FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\techie\\AppData\\Local\\Temp\\tmpmjdhug40\\7c98ad184709dded6b1c874ece2a0edea9c55b0a\\build\\classes\\kotlin\\test\\com\\mycompany\\myfunction\\domain\\MyServiceTest$should register valid make this a long text to exceed this successfully$2.class'

First I thought it was due to spaces in the test-methodname. But replacing them with an underscore didn't work either. Or maybe case-insensitive-ness; but my test-methodname is all lowercase.

I tried many things to exclude the whole test-class from the process, because why is it even included during the sam build command? 

Options I tried were:

1. Setting GRADLE_OPTS before running the command -x test, similar to the MAVEN_OPTS example here: https://github.com/aws/aws-sam-cli/issues/1105#issuecomment-777703158
2. Excluding the test-file or even just all tests in build.gradle, like:
   
   

   jar {

     sourceSets {

           main {

               java {

                   exclude '**/TestExcludeClass.java'

               }

   
   

               kotlin {

                   exclude '**/TestExcludeKotlinClass.kt'

               }

           }

       }

   }

   Note that excluding everything with 'exclude '**/*.kt'  did make the sam build fail, so the changes were taken into account.
3. In build.gradle add: excludeTestsMatching "com.mycompany.myfunction.lambda.MyServiceTest"
4. test.onlyIf { ! Boolean.getBoolean(skipTests) } and then specifying as GRADLE_OPTS="-DskipTests=true"

It seems the initial step of the sam build (JavaGradleWorkflow:JavaGradleCopyArtifacts) just takes all .class files anyway, even test classes.

I tried turning off Telemetry That did have no affect on the error.

Solution

Then I found this comment: https://github.com/aws/aws-sam-cli/issues/4031#issuecomment-1173730737

Could it be that, that the path is just too long? Typical Windows limit so that could definitely be it.

And yes after applying below command in PowerShell as an Administrator

New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" `-Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force

it worked!

Detailed explanation: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=powershell#enable-long-paths-in-windows-10-version-1607-and-later

The command ran fine! This is then the output:

Build Succeeded

Built Artifacts  : ..\..\techie\workspace\my-function\local\.aws-sam\build

Built Template   : ..\..\techie\workspace\my-function\local\.aws-sam\build\template.yaml

Commands you can use next

=========================

[*] Validate SAM template: sam validate

[*] Invoke Function: sam local invoke -t ..\..\techie\workspace\my-function\local\.aws-sam\build\template.yaml

[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch

[*] Deploy: sam deploy --guided --template-file ..\..\techie\workspace\my-function\local\.aws-sam\build\template.yaml

PACT provider test with Serverless Java 11 Lambda and JUnit 5 (Jupiter)

Introduction

This blogpost shows how to create a Pact provider test using:

1. Pact 4.0.0
2. AWS Serverless Lambda
3. Java 11
4. JUnit 5 (Jupiter)

Example Provider Test

Below is the example test class. Note how a mock service is used to simulate the endpoint defined (also) in serverless.yml.

package com.ttlnews.pact.tests.provider;

import au.com.dius.pact.provider.junit.Provider;
import au.com.dius.pact.provider.junit.State;
import au.com.dius.pact.provider.junit.loader.PactBroker;
import au.com.dius.pact.provider.junit.loader.PactBrokerAuth;
import au.com.dius.pact.provider.junit5.HttpTestTarget;
import au.com.dius.pact.provider.junit5.PactVerificationContext;
import au.com.dius.pact.provider.junit5.PactVerificationInvocationContextProvider;
import com.amazonaws.services.lambda.runtime.Context;
import lombok.extern.slf4j.Slf4j;
import net.jcip.annotations.NotThreadSafe;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.TestTemplate;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockserver.integration.ClientAndServer;

import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.mockserver.integration.ClientAndServer.startClientAndServer;
import static org.mockserver.matchers.Times.exactly;
import static org.mockserver.model.Header.header;
import static org.mockserver.model.HttpRequest.request;
import static org.mockserver.model.HttpResponse.response;
import static org.mockserver.model.JsonBody.json;

/**
 * Java 11 serverless provider Pact contract provider implementation using Pact with Junit5 (Jupiter) for provider SERVICE_A.
 * 
 * Dependencies needed:
 *
     <dependency>
     <groupId>au.com.dius</groupId>
     <artifactId>pact-jvm-consumer-junit5</artifactId>
     <version>4.0.0</version>
     <scope>test</scope>
     </dependency>
    
     <dependency>
     <groupId>au.com.dius</groupId>
     <artifactId>pact-jvm-provider-junit5</artifactId>
     <version>4.0.0</version>
     <scope>test</scope>
     </dependency>
 *
 * 
 */
@Provider(SERVICE_A)
// Below environment variables must be set when running this test
@PactBroker(host = "${PACT_HOST}",
        authentication = @PactBrokerAuth(username = "${PACT_HOST_USERNAME}", password = "${PACT_HOST_PASSWD}"))
@NotThreadSafe // Pact contract tests can't seem to handle methods running in parallel, so prevent maven failsafe/surefire plugin to run Pact tests in parallel
@Slf4j
public class ServiceAProviderTest {

    private static final String SERVICE_A = "service-A";
    
    private Context lambdaContext;
    private SomeRepository someRepository;

    private ClientAndServer mockClientAndServer;

    @BeforeEach
    void before(PactVerificationContext context) {

        lambdaContext = mock(Context.class);
        someRepository = new InMemoryMockedRepo();

        mockClientAndServer = startClientAndServer(8888);

        context.setTarget(new HttpTestTarget("127.0.0.1", 8888));

    }

    @AfterEach
    void stopServer() {
        mockClientAndServer.stop();
    }

    // This triggers the defined contract test(s) at the host PACT_HOST
    @TestTemplate
    @ExtendWith(PactVerificationInvocationContextProvider.class)
    void pactVerificationTestTemplate(PactVerificationContext context) {
        context.verifyInteraction();
    }

    @State("Create a new resource A")
    public void shouldCreateResourceA() {

        log.info("Set up state");

        // Given
        // Maybe some more mocking of 'lambdaContext' needed, depending on your case
        
        // Lambda being tested
        CreateResourceALambda createResourceALambda = new CreateResourceA(someRepository);

        CreateResourceALambdaRequest createResourceALambdaRequest = CreateResourceALambdaRequest.builder()
                .someValue("25")
                .build();

        // When
        final LambdaResult lambdaResult = createResourceALambda.executeRequest(createResourceALambdaRequest, lambdaContext);
        final String body = lambdaResult.getBody();
        assertNotNull(body);

        // Prepare the mockserver to return what the lambda returns when it is invoked (set up above in the 'body' variable) 
        // Of course the path to this CreateResourceALambda is defined in the serverless.yml, but that we can't access now. So need to repeat that
        // endpoint here.
        mockClientAndServer.when(
                request()
                        .withMethod("POST")
                        .withPath("/resources/")
                        .withHeaders(

                                header("x-request-trace-id"),  // Any value is fine
                                header("Authorization"),  // Any value is fine
                                header("Content-Type", "application/json")

                        )
                        .withBody(json("{someValue: '25'}")) // Indeed need to use this strange JSON format
                ,
                exactly(1))
                .respond(
                        // Response as defined by the matching Pact consumer test; in this case found at host PACT_HOST
                        response()
                                .withStatusCode(201)
                                .withHeaders(
                                        header("Content-type", "application/json; charset=utf-8"),
                                        header("Authorization") // Any value is fine
                                )
                                .withBody(body)
                );
    }
}

Migrating from your Pact tests from JUnit4 to Junit5 can be found here.

Note to self: use https://www.opinionatedgeek.com/codecs/htmlencoder to encode code, open the HTML view, and then wrap the code in <pre> open en close tag.

Lessons learned using AWS Kinesis to process business events and commands (messages)

Introduction

When using AWS Kinesis as the means of communicating business events, several challenges arise. For the basic key concepts of Kinesis see here. 

Of course the "normal" use case to use Kinesis is described as here in the section "When should I use Amazon Kinesis Data Streams, and when should I use Amazon SQS?" Another comparison between Kinesis and SQS can be found here.

Especially built for Kafka-like stream-processing of millions of events, where usually the producers (e.g think IoT devices) are much faster than the consumer applications.

Usually business events are not generated millions per minute or second. Events from IoT devices are usually not considered business events.

Any next time I would not recommend using Kinesis as a transport mechanism for business events; the main reason for me is its intended use-case is not matching that type of use. Plus the many technical challenges I had to solve, for most of which any "regular" messaging tool like RabbitMQ would suffice.

Additionally, usually one wants business events to arrive almost instantly - or at least as fast as possible. Kinesis is fast but e.g RabbitMQ is often sufficient as transport for business events.

And yes RabbitMQ is now also available as a managed solution - hosted at AWS but managed not by AWS itself. For example with CloudAMQP.

In the below discussion a message can be an event or a command (though most often you'd want the commands to be synchronous, since they should only be delivered to one service anyway)

High level Kinesis architecture overview

Below is a high level overview of Kinesis' architecture: 

There can be multiple streams, and within each stream are one or more shards. Messages in a shard are guaranteed to be delivered at the consumer in the order they were published onto the shard.

The partition-key used while publishing determines into what shard the message is published. Messages with the same partition key get always published into the same shard.

A design decision one has to make is: do I make one stream were all services publish their messages, or do I want a stream per XYZ? Where XYZ could be for example a Bounded Context.

No good reason was found to split into multiple streams, so I decided to go for one Kinesis stream all services publish on.

Lessons learned

When using Kinesis to handle business events between micro-services several challenges were to overcome. Below the lessons learned are described.

More than five different applications processing the same shard at the same time

Kinesis limits documentation  states that each shard within a stream supports up to five read transactions per second. And if you need more than that, it is recommended to increase the number of shards.

If you don't comply to that limit, you'll see ReadProvisionedThroughputExceeded or ProvisionedThroughputExceededException exceptions appear in your logs. This can easily be reached if you have large documents (e.g 1MB) in your DynamoDB too; even with the AWS Console you then can't even view such a large document in your browser!

Increasing the number of shards will spread the published messages across more shards and thus reduce the number of read transactions per second per shard.

BUT: in a microservices architecture you can easily have 10 or more services. And all these services need to process all messages on all shards.

So with for example 10 services, each of those services (container or lambda) will need to poll each shard regularly in some form. And the more services, the more chance that more than 5 services poll any given shard per second... Causing the above ReadProvisionedThroughputExceeded exception.

Thus: increasing the number of shards won't help, because still each service will need to poll each shard.

One workaround for this could be:

• have each service (container, lambda) publish to one central stream. Note that multiple streams in the end will also not help you, since you'll reach that 5 reads per second limit soon as the number of services increases to 5 or higher
• have a stream per service that needs to consume messages
• have a smart lambda that reads from the central stream and re-publishes each message on each of the per-service-streams using the same partitionkey the publisher used
• in case of error in that smart lambda: the lambda should store the failed events somewhere, e.g DynamoDB for later investigation. But when such an error occurs, that means what is received by the consumer is not in the same order anymore as the order the publisher published the messages (the consumer misses a message! And might get it in later when the investigation decides to republish it on the central stream)
• note that this introduces at least one second delay because of the smart lambda being allowed to poll once a second. And then if the consuming service is also a lambda, another second delay is introduced.

The above solution makes sure each service-stream has only one type of application consuming and any read limits can be fixed by increasing the number of shards (because only one service will be reading from it anyway).

Note Enhanced Fanout was not released yet then, which seems to also solve the described issue.

Options tried by changing the configurations: 

1. KCL: increasing withIdleTimeBetweenReadsInMillis  and withIdleTimeBetweenCallsInMillis: only helps in a limited way plus reduces throughput.
2. KCL: same for withMaxrecords(). Even if there are no records, all services will still from time to time have to poll the shard...
3. Increasing AWS provisioned maximum could of course help but what's the final maximum there? It will probably need to be increased for every X new services.

Increasing throughput

When the consumer needs to be really fast in processing messages from a Kinesis stream shard, one can spin off a new thread per record in the batch. But several things have to be taken into account when doing that:

• checkpointing is still at record (or batch) level. So you still have to in some way checkpoint only if you know all message before that moment are processed fine too. So you'll need some thread-orchestration to determine to what record to checkpoint
• if you care about the order, then a thread per record causes processing-ordering issues; so in that case you'll have to first group all records together that apply to the same entity, then start a thread to process these as a whole 

No direct Kinesis consumer

A totally different approach is to have your container services not be bothered with Kinesis at all, and put Lambdas in front of them, which invoke the services via regular REST calls. This alleviates the consumers completely from processing messages.

An example framework that provides this is Zalando's Nakadi. Though implemented for Kafka, still a viable option. This architecture is definitely something to consider.
A more advanced solution could be to have the lambdas put the messages in an SQS queue. This provides even more decoupling and if your service is not available, the lambdas can still deliver the messages into the SQS; care has to be taken of how to support multiple consumers reading from the same SQS queue. Of course then you almost must start to wonder why you are using Kinesis when you are putting a regular queue behind it...

Kinesis Client Libray (KCL) challenges

For the Java consumers the recommended option by AWS is to use the Kinesis Client Library, KCL for short. A short high level introduction can be found here.

This library takes away a lot of standard challenges in a distributed solution. See below's KPL link to implementing efficient producers for an explanation.

But still quite a few challenges exist. E.g how does it handle multiple instances running a Worker? Should you checkpoint per batch or per record?

Are batches provided to the worker one at at time, sequentially or in parallel? How to handle failed messages?

Should workers have a unique ID?

Here's a summary of lessons learned:

• a shard will be consumed by a record processor thread (not necessarily always the same thread!) of one single worker only at any given moment. And thus batches will never be handed over to processRecords() in parallel; first batch 1, then batch 2 etc.   consumeShard() is protected via synchronized.
• starting multiple workers on same host won't improve throughput; one on each host seems most effective, otherwise Workers start maybe also stealing leases too many times/too much
• give each worker a unique ID per host so you can see which host has what lease
• seems best to pass in your own Executor to the KCL when using Spring, so Spring knows about that threadpool. 

Note that checkpointing too much (or querying your own DynamoDB database while processing each message) can easily cause a ProvisionedThroughputExceededException.

Therefore you also don't want to do that too often. In the end this is the high level algorithm (pseudo code) that one can use to cover all the above concerns:

In processRecords(records) {

   lastSuccessfulRecord = null;

   for (each record in records) {

      try {

         parse record into domain object;

         process(domain object);

         lastSuccessfulRecord = record;

      } catch (ex) {

         checkpointUpToIncluding(lastSuccessfulRecord);

      }

   checkpointBatch(); // All records in the batch successfully processed

}
Another question for next implementation will be: should the service really be doing smart checkpointing (i.e only checkpoint when record is processed successfully; if not successful, keep processing but at next restart of service start from the last checkpoint)? Because the "normal" messaging implementation of putting a message on a Dead Letter Queue might be a better solution. Though in that solution you'll have to think about edge scenarios, like when a message A to modify entity E is put on the DLQ, but a following message B related to entity E is processed successfully; should message A be replayed (published) on the main stream again? Or should as soon as one message for E is put onto the DLQ, all messages after that for E also be put on the DLQ?

Kinesis Producer Library (KPL) challenges

For the Java producers the recommended option by AWS is to use the Kinesis Producer Library, KPL for short. For key concepts see here.

See here for a thorough explanation of its advantages with example code.

The only challenge there was: how large should the batch size be?  It seems to depend mostly on how well your consumers can handle batches.

In the end I set it to values between 10 and 100.

Note that the standard AWS Kinesis SDK does not implement handling batches. That means if you use the KPL with batchsize 10, then when these batches of records are received by e.g Javascript lambdas using the standards SDK, they by default can't process them correctly because batches are not supported; unless you write code to unwrap the batch.

As a temporary workaround there the batchsize could be set to size 1 of course, which then makes sure each consumer gets to process one message at a time and not in a batch.

Hope these lessons learned help somebody while implementing Kinesis.