tag:blogger.com,1999:blog-84748714069588441402024-03-18T03:03:59.390+00:00Tech Team Lead NewsThe best articles and links to interesting posts for technical team leaders building sophisticated websites, applications and mobile apps.
Think about: software architecture, hardware architecture, design, programming, frameworks, scalability, performance, quality assurance, security, resolving issues, fixing bugs and Android.Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.comBlogger269125tag:blogger.com,1999:blog-8474871406958844140.post-27027837008024580492024-02-14T10:48:00.003+00:002024-02-14T10:54:59.281+00:00Kotlin: how to mock a static Companion method using MockK<h1 style="text-align: left;">Introduction</h1><p style="text-align: left;">How do you mock a static companion method in Kotlin using MockK?</p><p style="text-align: left;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZ7dLWwSL9Ge1_pMUmU7HFWuF9tWbMoXHmukz77oF1iqr5FOl_MrKSwxlaB4RTeRJ5S7ig40YGWIsqsaag4QNvd-tayXYazGFKP1ycZ6ejwoaKVPlJDor_b4mt-v1ZiaAhEj3FEZakwmxD3YJc8JXXKtER6lRXKNkCrx3Nyw2ddzDK5qhLVr5ExMX7YFE/s837/logo.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="152" data-original-width="837" height="73" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZ7dLWwSL9Ge1_pMUmU7HFWuF9tWbMoXHmukz77oF1iqr5FOl_MrKSwxlaB4RTeRJ5S7ig40YGWIsqsaag4QNvd-tayXYazGFKP1ycZ6ejwoaKVPlJDor_b4mt-v1ZiaAhEj3FEZakwmxD3YJc8JXXKtER6lRXKNkCrx3Nyw2ddzDK5qhLVr5ExMX7YFE/w400-h73/logo.PNG" width="400" /></a></div><p></p><p style="text-align: left;">I wanted to mock this (static) method in a companion object in Kotlin in class MyClass:<br /><br /><i> companion object {<br /> fun isNegativeAndFromSavingsAccount(amount: BigDecimal, accountType: accountType) = amount < BigDecimal.ZERO && accountType == AccountType.SAVINGS<br /> }<br /></i><br /> Trying it with a regular 'every' like this doesn't work:<br /><br /><i> every { Declaration.isNegativeAndFromSavingsAccount(any(), any()) } returns false</i><br /><br />Note it does compile fine!<br />But when running the test, you'll get this error: <br /><br /><i> io.mockk.MockKException: Failed matching mocking signature for left matchers: [any(), any()]<br /> at io.mockk.impl.recording.SignatureMatcherDetector.detect(SignatureMatcherDetector.kt:97)<br /></i></p><p style="text-align: left;">Setup:</p><ul style="text-align: left;"><li style="text-align: left;">Kotlin 1.9</li><li style="text-align: left;"><a href="https://mockk.io/">MockK</a> 1.13.7<br /></li></ul><h1 style="text-align: left;">Solution</h1><p style="text-align: left;">This is the way it does work:<br /><br /><i>import io.mockk.mockkObject<br /><br /> mockkObject(Declaration.Companion) {<br /> every { MyClass.</i><i>isNegativeAndFromSavingsAccount</i><i>(any(), any()) } returns false<br /> }<br /></i><br />Note this did <i>not</i> work, got the same error:<br /><br /><i> mockkObject(Declaration::class)<br /> every { MyClass.</i><i>isNegativeAndFromSavingsAccount</i><i>(any(), any()) } returns false<br /></i><br />I found several posts, but none of them gave a clear answer and/or were using some older version of MockK. E.g: <a href="https://github.com/mockk/mockk/issues/61">https://github.com/mockk/mockk/issues/61</a><br />and <a href="https://stackoverflow.com/questions/51792130/kotlin-unit-testing-how-to-mock-component-of-companion-object">this StackOverflow post</a>. <br /><br />Some more examples and variants of solutions can be found <a href="https://github.com/mockk/mockk/issues/136">here</a>, e.g when using <i>@JvmStatic</i>.<br /><br /></p>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-37440108732742859702023-12-26T13:26:00.000+00:002023-12-26T13:26:01.952+00:00OWASP DependencyCheck returns 403 Forbidden accessing NVD API using API key<h1 style="text-align: left;">Introduction</h1><p style="text-align: left;">Recently, the NVD (<a href="https://nvd.nist.gov/">National Vulnerability Database</a>) which the <a href="https://github.com/jeremylong/DependencyCheck">Owasp dependency check plugin</a> uses to get its data from to check for vulnerabilities, has <a href="https://nvd.nist.gov/general/news/API-Key-Announcement">introduced the use of an API key</a>. That's for them to better control access and throttling - imagine how many companies and organizations are using that API, each time a dependency check build is performed. Especially those that don't cache the NVD database and at each run retrieve it again. And be aware: "<i>... previous versions of dependency-check
utilize the NVD data feeds which will be deprecated on Dec 15th, 2023. Versions
earlier then 9.0.0 are no longer supported and could fail to work after Dec 15th, 2023.</i>"</p><p style="text-align: left;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAwuWlKYzAAXt6c6z4JGQ5TCdtmIXqim6EuPN3Ii574eXws5-nUtipH_0g84po6c_lck3vNOpE-VQI_nAuwOQ33OiG1dvN8LMcMgqlBSM_BdHi7PjeCnSP1p1UwpgP-BzMAQs8JxO_ulTgB4xZQzmN2dgkF6oM4pmom2USSrYAwo7v6B_hBRL60hNCQek/s5102/owasp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1701" data-original-width="5102" height="107" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAwuWlKYzAAXt6c6z4JGQ5TCdtmIXqim6EuPN3Ii574eXws5-nUtipH_0g84po6c_lck3vNOpE-VQI_nAuwOQ33OiG1dvN8LMcMgqlBSM_BdHi7PjeCnSP1p1UwpgP-BzMAQs8JxO_ulTgB4xZQzmN2dgkF6oM4pmom2USSrYAwo7v6B_hBRL60hNCQek/s320/owasp.png" width="320" /></a></div><p></p><p style="text-align: left;">But this introduction doesn't go without some hiccups. For example it is possible to still get HTTP 403 Forbidden responses, even while you have a valid key. Here's my research while trying to fix it. <br /></p><p style="text-align: left;">Setup:</p><ul style="text-align: left;"><li style="text-align: left;">Gradle 7.x</li><li style="text-align: left;"><a href="https://mvnrepository.com/artifact/org.owasp/dependency-check-gradle/9.0.6">Dependency Check v9.0.6</a> (issue applies at least for versions > 8.4.3)<br /></li><li style="text-align: left;">Configuration:<br /><br /><i>dependencyCheck {<br /> failBuildOnCVSS = 6<br /> failOnError = true<br /> suppressionFile = '/bamboo/owasp/suppressions.xml'<br /> nvd.apiKey = '</i><i><yourkey></i><i>'<br />}<br /></i><br />You can also set it dynamically via an environment variable like this:<br /><br /><i>dependencyCheck {<br /></i><i> nvd {<br /> apiKey = System.getProperty("ENV_NVD_API_KEY")<br /> }<br /></i>}<br /><br /></li><li style="text-align: left;">Via commandline you can invoke it like this: <br /><br /><i>./gradlew dependencyCheckAggregate -DENV_NVD_API_KEY=<yourkey></i><br /></li></ul><p> </p><h1 style="text-align: left;">Solution <br /></h1><p>First you should check if your API key is valid by execution this command:</p><p><i>curl -H "Accept: application/json" -H "apiKey: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -v https://services.nvd.nist.gov/rest/json/cves/2.0\?cpeName\=cpe:2.3:o:microsoft:windows_10:1607:\*:\*:\*:\*:\*:\*:\*<br /></i> </p><p>(where xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx is your NVD API key)</p><p>That should return JSON (and not a 404). Now you know your API key is valid.<br /><i> </i></p><p><a href="https://github.com/jeremylong/DependencyCheck/issues/6195">Some have some success</a> with setting the delay longer: <i><br /><br /> nvd {<br /> apiKey = System.getProperty("ENV_NVD_API_KEY")<br /> delay = 6000 // milliseconds, default is 2000 with API key, 8000 without<br /> }<br /><br /></i>Commandline version: <br /><br /><i>--nvdDelay 6000<br /> </i></p><p>You can also increase the <i>validForHours </i>option, but that doesn't work if during you construct completely new Docker containers each build - you lose that history.</p><p>All NVD options you can pass to DependencyCheck are found <a href="https://jeremylong.github.io/DependencyCheck/dependency-check-gradle/configuration.html">here</a>. <br /></p><p>But currently (27 December 2023) all the above efforts don't always fix the problem of the 403. Sometimes it works for a while, but then not again. If you build many projects in your company at about the same time, you still have a chance of getting throttled of course. <br /></p><p>The best solution is to <a href="https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#caching-the-nvd-cve-data">create a local cache so you are less dependent on NVID API calls</a> (and thus the throttling).</p><p> </p><h2 style="text-align: left;">Other causes mentioned</h2><ul style="text-align: left;"><li>Being behind a proxy with your whole company, see <a href="https://github.com/jeremylong/DependencyCheck/issues/6127">https://github.com/jeremylong/DependencyCheck/issues/6127</a> and "If you have multiple builds happening at the same time - using the same API key you could hit the NVD rate limiting threshold. Ideally, in an environment with multiple builds you would implement some sort of caching strategy". See: <a href="https://github.com/jeremylong/DependencyCheck/issues/6195">https://github.com/jeremylong/DependencyCheck/issues/6195</a></li><li>Use --data argument to control cache location.</li><li>It appears the NVD has put a temporary block on all requests that use a virtualMatchString of "cpe:2.3:a" - outstanding issue.<br /></li></ul><p style="text-align: left;"><br /></p><h2 style="text-align: left;">Misc</h2><ul style="text-align: left;"><li>Check the <a href="https://nvd.nist.gov/general/news/API-Key-Announcement">best practices from NVD</a> themselves<br /></li></ul><p style="text-align: left;"> </p>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-82671238859218614382023-11-04T14:18:00.002+00:002023-11-04T14:21:18.349+00:00MapStruct library: setting a field with an expression to a ZonedDateTime<h1 style="text-align: left;">Introduction</h1><p style="text-align: left;">Setup:</p><p style="text-align: left;">- <a href="https://mapstruct.org/">Mapstruct</a> 1.5.5<br />- Kotlin 1.8<br />- IntelliJ 2023.2.3<br /> </p><p style="text-align: left;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6ccYGOfbBLI3VQL3WDzxmdQud7tV0D98fGlzRtsbZa0ZejN7A-JGwXmNLczH_PEI58osZQQD-DPRKhAtrwDmJsIqhvUzRe-lfB56dnM3z0QKZEcqotiZm6Va_mrmvd8tpOClhS-S1VKX9SqlRwF-hRru1neowhfHLaI56UWLMw3_xiqKjXBCye1TJWWs/s517/mapstructlogo.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="182" data-original-width="517" height="141" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6ccYGOfbBLI3VQL3WDzxmdQud7tV0D98fGlzRtsbZa0ZejN7A-JGwXmNLczH_PEI58osZQQD-DPRKhAtrwDmJsIqhvUzRe-lfB56dnM3z0QKZEcqotiZm6Va_mrmvd8tpOClhS-S1VKX9SqlRwF-hRru1neowhfHLaI56UWLMw3_xiqKjXBCye1TJWWs/w400-h141/mapstructlogo.PNG" width="400" /></a></div> <p></p><p style="text-align: left;">For a mapping between two classes, the source class did not have the mandatory (non-null, Kotlin!) target class field <span style="font-family: courier;">created</span>. And I wanted to fill it with the current ZonedDateTime class, with the timezone of <span style="font-family: courier;">Amsterdam/Europe</span>. And that specific ZoneId is a constant in my Application.kt class named <span style="font-family: courier;">MY_DEFAULT_TIME_ZONE</span>.</p><h1 style="text-align: left;">Solution<br /></h1><p style="text-align: left;">So I looked at the <span style="font-family: courier;">expression</span> field in @Mapping found <a href="https://mapstruct.org/documentation/dev/reference/html/#expressions">here</a>.<br />I got this solution working quite fast with: </p><p style="text-align: left;"><i>@Mapping(target = "created", expression = "java(ZonedDateTime.now())"). </i></p><p style="text-align: left;">But as you see, the ZoneId constant is still missing. <br /></p><h1 style="text-align: left;">Potential other solutions <br /></h1><p style="text-align: left;"></p><p style="text-align: left;">I had to try quite a few things to get that working, because the class ZoneId was not getting recognized in the generated implementation MapStruct mapper.<br />In the end this worked: <br /></p><p style="text-align: left;"><i>@Mapper(componentModel = MappingConstants.ComponentModel.SPRING, imports = [ZoneId::class, Application::class])<br />interface MyMapper {<br /><br /> @Mapping(target = "created", expression = "java(ZonedDateTime.now(my.package.Application.MY_DEFAULT_TIME_ZONE))")<br /> fun myDtoToResponseDto(myDto: MyDto): ResponseDto<br /> ...<br />}<br /></i>Note the <span style="font-family: courier;">imports</span> field to have the class ZoneId and the constant available (imported) in the implementation class, generated by MapStruct.<br /></p><p style="text-align: left;">In the Application.kt you then have to make the <span style="font-family: courier;">MY_DEFAULT_TIME_ZONE</span> constant available to Java, since that's what MapStruct uses as language:<br /><br /><i>Application.kt<br />{<br /> companion object {<br /><br /> @JvmField<br /> val MY_DEFAULT_TIME_ZONE: ZoneId = ZoneId.of("Europe/Amsterdam")<br /> ...<br />}<br /></i></p><p style="text-align: left;">I also tried this: </p><p style="text-align: left;"><i>@Mapping(target = "created", source = ".", qualifiedByName = ["getValue"]) </i></p><p style="text-align: left;">with a function:<br /><br /><i>@Named(value = "getValue")<br />fun getValue(myDto: MyDto): ZonedDateTime {<br /> return ZonedDateTime.now(MY_DEFAULT_TIME_ZONE)<br />}</i> <br /></p><p style="text-align: left;">The advantage of this solution is that you can use Kotlin code and you don't have to wait and see if your <span style="font-family: courier;">expression</span> has the correct syntax and will compile.<br />But then I got this error: <i>ZonedDateTime does not have an accessible constructor</i>. I also tried to wrap the field <span style="font-family: courier;">created</span> in a small class, but that didn't work either (could be me :)<br />See <a href="https://www.journeytoawebapp.com/posts/kotlin-mapstruct-advanced">this</a> and <a href="https://www.tutorialspoint.com/mapstruct/mapstruct_using_expression.htm">this</a> for more details on how that should work.<br /><i><br /></i>I also tried with the @<span style="font-family: courier;">JvmDefault</span> annotation, but that is deprecated + it requires you to use the <span style="font-family: courier;">-Xjvm-default</span> property, which I couldn't get to work in IntelliJ with Gradle. <br />And it is not always guaranteed to work, see <a href="https://stackoverflow.com/questions/57924875/mapstruct-kotlin-and-named-annotation">here</a> and <a href="https://github.com/mapstruct/mapstruct/issues/1730">here</a> and <a href="https://github.com/mapstruct/mapstruct-examples/issues/55">here</a>:<br /><br />I'm definitely still a beginner in using MapStruct. So probably one of the other methods could work too... Any tips are welcome :)<br /><br /></p>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-33476750709697956072023-09-27T12:46:00.002+01:002023-09-27T13:03:47.841+01:00SpringDoc OpenAPI Swagger generated Swagger API shows incorrect class with same name<h1 style="text-align: left;">Introduction</h1><p style="text-align: left;">When you have multiple classes with the same name in your classpath, <a href="https://springdoc.org/">SpringDoc</a> with Swagger API annotations potentially picks the wrong class with the same name when generating the Swagger UI documentation.</p><p style="text-align: left;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6o_CkO9c6NQ2D8rYesHXpHGky4xkm31GgBzHpWGLUUQccoV-VlNjX4SlFXhSYdewBAG8PJry12gZVLz-D_aSAdpVXGEgD00-9Es7uHZjjCGjj2u22jTG2DMk6vwAdJMIxoCAWg_1-EgMT6CS6sS40dBsABYg1TZ6UwWQXB9UrITsol6nforX0X8absfw/s1536/springboot-swagger.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="243" data-original-width="1536" height="51" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6o_CkO9c6NQ2D8rYesHXpHGky4xkm31GgBzHpWGLUUQccoV-VlNjX4SlFXhSYdewBAG8PJry12gZVLz-D_aSAdpVXGEgD00-9Es7uHZjjCGjj2u22jTG2DMk6vwAdJMIxoCAWg_1-EgMT6CS6sS40dBsABYg1TZ6UwWQXB9UrITsol6nforX0X8absfw/s320/springboot-swagger.png" width="320" /></a></div><br />Suppose you have these classes:<p></p><ul style="text-align: left;"><li style="text-align: left;">org.example.BookDto</li><li style="text-align: left;">org.example.domain.BookDto<br /> </li></ul><p style="text-align: left;">And you specified your endpoint like this, where you want to have it use <i>org.example.BookDto</i>: </p><p style="text-align: left;"> <i>@Operation(summary = "Get a list of books for a given shop")<br /> @ApiResponses(<br /> value = [<br /> ApiResponse(<br /> responseCode = "200",<br /> description = "A list of books",<br /> content = [Content(mediaType = "application/json",<br /> array = ArraySchema(schema = Schema(implementation = BookDto::class)))]<br /> )<br /> ]<br /> )<br /> @GetMapping("/books/{shopId}")<br /> fun getBooksByShopId(<br /> @Parameter(description = "Shop to search for")<br /> @PathVariable shopId: Long<br /> ): List<BookDto> {<br /> return bookService.getBooksByShopId(shopId)<br /> .map { BooksMapper.mapDto(it) }<br /> }</i></p><p style="text-align: left;">Then whatever it finds first on the classpath will be visible in <a href="https://localhost:8080/swagger-ui.html">https://localhost:8080/swagger-ui.html</a>. Not necessarily the class you meant, it might pick <i>org.example.domain.BookDto</i>. </p><p style="text-align: left;">Setup:</p><ul style="text-align: left;"><li style="text-align: left;">Spring Boot 3</li><li style="text-align: left;">Kotlin 1.8</li><li style="text-align: left;">Springdoc OpenAPI 2.2.0<br /> </li></ul><h1 style="text-align: left;">Solution</h1><div style="text-align: left;"><p style="text-align: left;">Several solutions exist:</p><h3 style="text-align: left;">Solution 1</h3><p style="text-align: left;">Specify in your application.yml:</p><p style="text-align: left;"><i>springdoc:<br /> use-fqn: true</i><br /> </p><p style="text-align: left;">Disadvantage: the Swagger documentation in the <i>swagger-ui.html</i> endpoint has then the fully specified package classpath + classname in it. Looks ugly. <br /></p><h3 style="text-align: left;">Solution 2</h3><p style="text-align: left;">Setting it in the @Bean configuration:</p><p style="text-align: left;"><i>import io.swagger.v3.core.jackson.TypeNameResolver<br /> @Bean<br /> fun openAPI(): OpenAPI? {<br /><br /> <b>TypeNameResolver.std.setUseFqn(true)</b><br /> return OpenAPI()<br /> .addServersItem(Server().url("/"))<br /> .info(<br /> Info().title("Books Microservice")<br /> .description("The Books Microservice")<br /> .version("v1")<br /> )<br /> .externalDocs(<br /> ExternalDocumentation()<br /> .description("Books Microservice documentation")<br /> .url("https://github.com/myproject/README.md")<br /> )<br /> }</i></p><p style="text-align: left;">Disadvantage: also in this solution the Swagger documentation in the <i>swagger-ui.html</i> endpoint has then the fully specified package classpath + classname in it. Looks ugly.<i> </i></p><h3 style="text-align: left;">Solution 3</h3><p style="text-align: left;">You can create your own ModelConverters, but that is much more work. Examples here: <a href=" https://github.com/swagger-api/swagger-core/wiki/Swagger-2.X---Extensions#extending-core-resolver"> https://github.com/swagger-api/swagger-core/wiki/Swagger-2.X---Extensions#extending-core-resolver</a> and <a href="https://groups.google.com/g/swagger-swaggersocket/c/kKM546QXGY0">https://groups.google.com/g/swagger-swaggersocket/c/kKM546QXGY0</a><br /></p><p style="text-align: left;"></p><p style="text-align: left;"></p><h3 style="text-align: left;">Solution 4</h3><p style="text-align: left;">Make sure for each endpoint you specify the response class with full class package path:</p><p style="text-align: left;"> <i>@Operation(summary = "Get a list of books for a given shop")<br /> @ApiResponses(<br /> value = [<br /> ApiResponse(<br /> responseCode = "200",<br /> description = "A list of books",<br /> content = [Content(mediaType = "application/json",<br /> array = ArraySchema(schema = Schema(implementation = </i><b>org.example.</b><i><b>BookDto::class</b>)))]<br /> )<br /> ]<br /> )<br /> @GetMapping("/books/{shopId}")<br /> fun getBooksByShopId(<br /> @Parameter(description = "Shop to search for")<br /> @PathVariable shopId: Long<br /> ): List<BookDto> {<br /> return bookService.getBooksByShopId(shopId)<br /> .map { BooksMapper.mapDto(it) }<br /> }</i></p></div><p style="text-align: left;"> See the bold Schema implementation value for what changed.<br /></p><br /><p style="text-align: left;"> <br /></p><p style="text-align: left;"> </p>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-39544306415882717712023-08-23T10:00:00.001+01:002023-08-23T10:00:00.146+01:00Unknown application error occurred Runtime.Unknown - Startup issue AWS Serverless Lambda<h1 style="text-align: left;">Introduction</h1><div style="text-align: left;"><h2 style="text-align: left;"></h2></div><div style="text-align: left;">Trying to invoke a deployed AWS Serverless Lambda on AWS, I was getting this error CloudWatch when trying to invoke the lambda via an SQS event, published by another service in my landscape:</div><div style="text-align: left;"> </div><div style="text-align: left;"><i>2023-08-15T15:20:44.047+02:00 START RequestId: ab924ff5-236c-5b09-8a29-12a0b9447e41 Version: $LATEST<br />2023-08-15T15:20:45.223+02:00 Unknown application error occurred<br /> Runtime.Unknown<br /> Unknown application error occurred Runtime.Unknown<br />2023-08-15T15:20:45.223+02:00 END RequestId: ab924ff5-236c-5b09-8a29-12a0b9447e41</i><br /> </div><div style="text-align: left;"></div><div style="text-align: left;"><br /></div><div style="text-align: left;">That's all. No more details. Nothing appeared in Datadog to which my CW logging is forwarded to. But the lambda ran fine when running it locally in IntelliJ using the SAM <a href="https://aws.amazon.com/intellij/">AWS Toolkit</a>, with me logged in with my IAM role.<br /></div><div style="text-align: left;">Adding logging or a try/catch wouldn't do anything, since this error appears already before the lambda even gets invoked.</div><div style="text-align: left;"> </div><div style="text-align: left;">Setup:</div><div style="text-align: left;"><ul style="text-align: left;"><li>AWS Serverless Lambda</li><li>IAM</li><li>IntelliJ AWS Toolkit</li><li>Kotlin 1.8.10</li><li>CloudWatch</li><li>Datadog</li><li>AWS Parameter Store</li><li>KMS</li><li>SSM</li><li>SQS<br /> </li></ul></div><div style="text-align: left;"> <div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwJtutrG301hDJaZ-CHOAum1hE81nuP-QG02jsHb1C2q16Qxlg-nbTJJUOCbHrDNHjO_iw2uckiorjuDK05-lo-L5_hSZWrPB9ue5qWlAilAVHPG1EWx8jM1YQnH3RgdI4gx82iCeBObOyVNNPqwJtiwk6pzCnIWOSHF1ND_OtNTIbgYFq5QIOEXipqu0/s2048/aws-lambda.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1343" data-original-width="2048" height="210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwJtutrG301hDJaZ-CHOAum1hE81nuP-QG02jsHb1C2q16Qxlg-nbTJJUOCbHrDNHjO_iw2uckiorjuDK05-lo-L5_hSZWrPB9ue5qWlAilAVHPG1EWx8jM1YQnH3RgdI4gx82iCeBObOyVNNPqwJtiwk6pzCnIWOSHF1ND_OtNTIbgYFq5QIOEXipqu0/s320/aws-lambda.png" width="320" /></a></div><br /></div><div style="text-align: left;"><h1 style="text-align: left;">Solution</h1><div style="text-align: left;">Then I tried to trigger the lambda via the AWS console by manually creating the SQS event and sending it on the SQS queue the lambda is listening to. There I <i>did </i>get the details of the error shown:<br /><br /><i>{<br /> "errorMessage": "User: arn:aws:sts::100004:assumed-role/my-lambda-role-acc/my-lambda is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:eu-west-1:</i><i>100004</i><i>:parameter/abc/apikey because no identity-based policy allows the ssm:GetParameter action (Service: AWSSimpleSystemsManagement; Status Code: 400; Error Code: AccessDeniedException; Request ID: 657c62f2-3527-42e0-8ee4-xxxxxxxx; Proxy: null)",<br /> "errorType": "com.amazonaws.services.simplesystemsmanagement.model.AWSSimpleSystemsManagementException"<br />} </i></div><div style="text-align: left;"><i> </i><br /></div></div><div style="text-align: left;"></div><div style="text-align: left;"></div><div style="text-align: left;"></div><div style="text-align: left;"></div><div style="text-align: left;"></div><div style="text-align: left;">See this screenshot: </div><div style="text-align: left;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbfgswaJKy3mqBKZRupUHi32bSGmYMqm4r8W81DulfYhB0Lj_UFfJ_RWUhPn3i5Ix8PBcdle3zZzjGCFgYk5NXv8ffI8dFeh9BcPvNYlJ3P-SmJ5_hIAG-JIEjBY08vqKLFxyO1WD_BaVdidiepGWnUoWJekNCql1Awdxw3bky3t9y14FwSez0gNG7jkQ/s1841/aws-console-output-does-show-the-error%20-%20clean.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="897" data-original-width="1841" height="156" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbfgswaJKy3mqBKZRupUHi32bSGmYMqm4r8W81DulfYhB0Lj_UFfJ_RWUhPn3i5Ix8PBcdle3zZzjGCFgYk5NXv8ffI8dFeh9BcPvNYlJ3P-SmJ5_hIAG-JIEjBY08vqKLFxyO1WD_BaVdidiepGWnUoWJekNCql1Awdxw3bky3t9y14FwSez0gNG7jkQ/s320/aws-console-output-does-show-the-error%20-%20clean.PNG" width="320" /></a></div><br /> </div><div style="text-align: left;"> </div><div style="text-align: left;"></div><div style="text-align: left;">The reason it worked locally is probably because there I'm logged in with a different IAM account (with more permissions) than when the lambda is deployed in the AWS cloud.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">Then after fixing that by adding the path <i>abc/apikey</i> to the key as resource, I got this error:<br /><i>{<br /> "errorMessage": "User: arn:aws:sts::</i><i>100004</i><i>:assumed-role/</i><i>my-lambda-role-acc/my-lambda</i><i> <br />is not authorized to perform: kms:Decrypt on resource: arn:aws:kms:eu-west-1:</i><i></i><i>100004</i><i></i><i>:key/ff841b70-5038-6f0b-8621-xxxxxx because no identity-based policy <br />allows the kms:Decrypt action (Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException; Request ID: aaa5a8d0-d26e-5051-7ac0-xxxxxxxx; Proxy: null) <br />(Service: AWSSimpleSystemsManagement; Status Code: 400; Error Code: AccessDeniedException; Request ID: f807b8d7-826e-4d4c-9b5c-xxxxxxx; Proxy: null)",<br /> "errorType": "com.amazonaws.services.simplesystemsmanagement.model.AWSSimpleSystemsManagementException"<br />}</i><br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;">So the KMS decrypt is not allowed (no permission for) on that specific AWS Parameter Store entry <i>abc/apikey</i>.</div><div style="text-align: left;"><br />The fix here was to add the action for the correct resource, see the items in <b>bold</b>:</div><div style="text-align: left;"><br /><i> statement {<br /> sid = "AllowReadingParameterStoreParameters"<br /> effect = "Allow"<br /><br /> actions = [<br /> "ssm:DescribeParameters",<br /> "ssm:GetParameter",<br /> "<b>kms:Decrypt</b>" <br /> ]<br /><br /> resources = [<br /></i><i> "arn:aws:ssm:</i><i>eu-west-1</i><i>:</i><i>100004</i><i>:parameter/abc/apikey",<br /> "<b>arn:aws:kms:</b></i><b><i>eu-west-1</i><i>:</i><i>100004</i></b><i><b>:key/*</b>" <br /> ]<br /> }</i><br /></div><div style="text-align: left;">Note that the error gave away already a little on how to name that resource. Be aware that this way you potentially give more Decrypt access than you want...<br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;">Misc:<br /></div><div style="text-align: left;">Other tips to try if you ever have this Runtime.Unknown error (but did not try): <a href="https://docs.aws.amazon.com/lambda/latest/dg/java-tracing.html">Instrumenting Java code in AWS Lambda</a>.<br />And some more generic tips for<a href="https://docs.aws.amazon.com/lambda/latest/dg/troubleshooting-invocation.html"> troubleshooting during/before invocation</a>.<br />And while executing: <a href="https://docs.aws.amazon.com/lambda/latest/dg/troubleshooting-execution.html">https://docs.aws.amazon.com/lambda/latest/dg/troubleshooting-execution.html</a></div><div style="text-align: left;"> <br /></div>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-18287058599006722992023-07-21T16:45:00.003+01:002023-07-22T12:42:34.706+01:00Too many open files in AWS Lambda serverless troubleshooting<h1 style="text-align: left;">Introduction</h1><div>One of my Kotlin lambdas was throwing <i>Too many open files </i>exceptions and thus logging errors, but only nightly, when it got bursts of SQS messages to process. </div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLmhG8IBhklnW-LJgepZRpnZRVBBxRoU56M5rTseKAIoAcTQrYlr5B-3SwjxPjncU5NwAt9G1JSFW83XxYzod3teflbftxi-bywPgfjcOd_OqlD2GPVuvHGFZtIThOiRIQqyBKzJbeRf_otyXuIKI44XvJltil7M4rCrPxgJ1lq7Uax0_jN6Cinw4Wh3o/s1600/book.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1066" data-original-width="1600" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLmhG8IBhklnW-LJgepZRpnZRVBBxRoU56M5rTseKAIoAcTQrYlr5B-3SwjxPjncU5NwAt9G1JSFW83XxYzod3teflbftxi-bywPgfjcOd_OqlD2GPVuvHGFZtIThOiRIQqyBKzJbeRf_otyXuIKI44XvJltil7M4rCrPxgJ1lq7Uax0_jN6Cinw4Wh3o/s320/book.jpg" width="320" /></a></div><div><br /></div><div><br /></div><div>To find out what was causing these errors, I followed these steps:</div><div><ol style="text-align: left;"><li>Read up on what can be the causes</li><li>Determine what/where in the code is not closing the file descriptors</li><li>Find a solution to the issue</li><li>Then fix the issue</li></ol><div>After reading up, it turned out that it can be actual files not getting closed, but open connections also use file descriptors, so they count for the total number of open File Descriptors (FDs).</div></div><div><br /></div><div>I also found out that for AWS Lambda Serverless, the maximum open file descriptors is <a href="https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html">fixed to 1024</a>. Normally in Linux systems you can modify that limit e.g with the <a href="https://linuxhint.com/linux_ulimit_command/">ulimit</a> command, but <i>not </i>in the lambdas execution runtimes. Thus a quick fix of increasing the open files limit wasn't possible.</div><div><br /></div><div>Important to know too when analyzing this problem is that "<i>... Lambda doesn't send more than one invocation at a time to the same container. The container is not used for a second invocation until the first one finishes. If a second request arrives while a first one is running, the second one will run in a different container. When Lambda finishes processing the first request, this execution environment can then process additional requests for the same function. In general, each instance of your execution environment can handle at most 10 requests per second. This limit applies to synchronous on-demand functions, as well as functions that use provisioned concurrency. In you're unfamiliar with this limit, you may be confused as to why such functions could experience throttling in certain scenarios.</i>" Partially from: <a href="https://docs.aws.amazon.com/lambda/latest/dg/lambda-concurrency.html">https://docs.aws.amazon.com/lambda/latest/dg/lambda-concurrency.html</a> My addition: note that in the above mentioned 10 requests per second performance, still those 10 requests are handled sequentially!!</div><div><br /></div><div>Note that <i>no </i>events were lost in the end when the exceptions occurred; AWS lambda recovered by itself by providing the events again from the queue, since these were not processed due to the exception. It also scaled up the number of instances significantly, probably due to the burst <i>and</i> it detecting that messages were not getting processed sufficiently quick.</div><div><br /></div><div>To the determine the <i>cause </i>of the open file descriptors, I tried several options to find out <i>how many and which</i> files are opened by <i>what</i>:</div><div><ol style="text-align: left;"><li>Try by using a Java MXBean</li><li>Try the <a href=" https://github.com/jenkinsci/lib-file-leak-detector ">File Leak Detector</a> library</li><li>Try via Linux commands</li><li>Force Garbage Collections</li><li>Examine the code for potential spots where files and connections are reopened over and over again</li></ol><div>Not tried but could be an option to explore: track the network connections created, e.g get the open connections count from OkHttpClient. Something like this: <i>OkHttpClient().newBuilder().build()..dispatcher().runningCallsCount()</i></div></div><div><br /></div><div>Setup</div><div><ul>
<li><p style="margin-bottom: 0in;">AWS lambda</p>
</li><li><p style="margin-bottom: 0in;">Kotlin 1.8.10</p>
</li><li><p style="margin-bottom: 0in;">Java 17</p></li><li><p style="margin-bottom: 0in;">Retrofit2</p></li><li><p style="margin-bottom: 0in;">IntelliJ</p></li><li><p style="margin-bottom: 0in;">Gradle</p></li></ul></div><div><br /></div><h2 style="text-align: left;">Java MXBean open files detection</h2><div>This option only supports showing <i>the amount </i>of open file descriptors. Not which part(s) of the lambda have a given file descriptor in use.</div><div><pre class="western"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><span style="color: #0033b3;">val </span><span style="color: black;">os</span><span style="color: #080808;">: </span><span style="color: black;">OperatingSystemMXBean </span><span style="color: #080808;">= </span><span style="color: black;">ManagementFactory</span><span style="color: #080808;">.getOperatingSystemMXBean()</span></span></span>
<span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><span style="color: #0033b3;">if </span><span style="color: #080808;">(</span><span style="color: black;">os </span><span style="color: #0033b3;">is </span><span style="color: black;">UnixOperatingSystemMXBean</span><span style="color: #080808;">) {</span></span></span>
<span style="color: #080808;"> </span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">logger</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.info(</span></span></span><span style="color: #067d17;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">"Number of open fd: " </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">+ (</span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">os </span></span></span><span style="color: #0033b3;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">as </span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">UnixOperatingSystemMXBean</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">).</span></span></span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><i>openFileDescriptorCount</i></span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">)</span></span></span>
<span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">}</span></span></span></pre><p style="margin-bottom: 0in;">
Found via:
<a href="https://stackoverflow.com/questions/16360720/how-to-find-out-number-of-files-currently-open-by-java-application"><span style="color: navy;"><span lang="zxx"><u>https://stackoverflow.com/questions/16360720/how-to-find-out-number-of-files-currently-open-by-java-application</u></span></span></a></p>
<p style="margin-bottom: 0in;">Note the call will fail at the
moment the <i>Too many files</i> error starts to happen, because logging and
many other calls require a file descriptor; and the MXBean itself probably too.... So all you can see is that number of open file descriptors increase and increase up to the exception.</p></div><div><h2 style="text-align: left;">File Leak Detector open files detection</h2><div>I used v1.13 since v1.15 was not available on Maven Central. </div><div>First you have to get this library on the command line when starting the Lambda. But after supplying the java agent to the
command line of the AWS lambda like this:</div>
<p style="margin-bottom: 0in;"><span face="ui-monospace, SFMono-Regular" style="color: #1f2328; font-size: 9pt;">java -javaagent:lib/file-leak-detector.jar</span></p><p style="margin-bottom: 0in;">the error during startup was:</p>
<p style="margin-bottom: 0in;"><i>Failed to find Premain-Class
manifest attribute in lib/file-leak-detector-1.13.jar</i></p>
<p style="margin-bottom: 0in;"><i>Error occurred during initialization
of VM</i></p>
<p style="margin-bottom: 0in;"><i>agent library failed to init:
instrument</i></p>
<p style="margin-bottom: 0in;">That error shows up because the
<span style="color: #232629;"><span face="apple-system, BlinkMacSystemFont"><span style="font-size: 11pt;">MANIFEST.MF</span></span></span>
file is missing the <code class="western">Premain-Class </code>entry, which tells the runtime what the
main method is to start the agent.</p></div><div><br /></div><div>I tried some other paths to
verify the path was correct; if the path is incorrect you get a
message like “<i>Error opening zip file or JAR manifest missing</i>”.</div><div><br /></div><div>(note that I already had a <i>-javaagent</i>
argument on the command line for Datadog. Both added caused the
deployment to fail with a timeout; didn't further investigate why, I just
removed that Datadog -javaagent for now)</div><div><br /></div><div>And indeed when I looked inside the
MANIFEST.MF of the <span style="color: #1f2328;"><span face="ui-monospace, SFMono-Regular"><span style="font-size: 9pt;">file-leak-detector-v1.13.jar</span><span>,
no such entry</span><span style="font-size: 9pt;">. </span></span></span>I then downloaded the source code of the library from Github and noticed another jar file getting created: <i>file-leak-detector-1.16-SNAPSHOT-jar-with-dependencies.jar</i></div><div><p style="margin-bottom: 0in;">(note here I switched to v1.16-snapshot just to have the latest)<br /><br />And in there, the Premium-Class is set!</p><pre class="western"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><span style="color: #0033b3;">Premain-Class</span><span style="color: #080808;">: </span><span style="color: black;">org.kohsuke.file_leak_detector.AgentMain</span></span></span></pre><p style="margin-bottom: 0in;">I then decided to add the new jar
locally to the build of the lambda, for testing purposes, as described here:
<a href="https://stackoverflow.com/questions/20700053/how-to-add-local-jar-file-dependency-to-build-gradle-file"><span style="color: navy;"><span lang="zxx"><u>https://stackoverflow.com/questions/20700053/how-to-add-local-jar-file-dependency-to-build-gradle-file</u></span></span></a></p><p style="margin-bottom: 0in;">
</p><p style="margin-bottom: 0in;">The jar was put in the 'libs' directory
which I created in the root directory of the (IntelliJ) Gradle
project. Gradle depencency: <i>implementation files('libs/file-leak-detector-1.16-SNAPSHOT-jar-with-dependencies.jar')</i></p><p style="margin-bottom: 0in;">After that, the File Leak Detector
started up fine, as can be seen from these messages:</p><p style="margin-bottom: 0in;"><i>File leak detector installed</i></p><p style="margin-bottom: 0in;"><i>Could not load field socket from
SocketImpl: java.lang.NoSuchFieldException: socket</i></p><p style="margin-bottom: 0in;"><i>Could not load field serverSocket
from SocketImpl: java.lang.NoSuchFieldException: serverSocket</i></p><p style="margin-bottom: 0in;">Note the last two messages are due to
Java17+ not allowing this anymore, you can find more details about
this when searching for those exact error messages in the File Leak Detector
source code.</p><p style="margin-bottom: 0in;">I then did have <i>SocketExceptions </i>appear at the nightly runs too like
“<i>Caused by: java.net.SocketException: Too many open files</i>” so I
couldn't tell too much yet. It seems the lib-file-leak-detector is
then not dumping the open files, probably because the above mentioned Java 17+ issue. Or something else went wrong, at least I
couldn't see any dumps in AWS CloudWatch though.</p><p style="margin-bottom: 0in;">So I set up my own listener from the
library, so I could then dump the open files whenever I wanted. It is possible, but no full example is given; the
*Demo.java examples give some ideas away. Here's what I used:</p><pre class="western" style="background: rgb(255, 255, 255);"><pre class="western"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><span style="color: #871094;">logger</span><span style="color: #080808;">.info(</span><span style="color: #067d17;">"Is leak detector agent installed = " </span><span style="color: #080808;">+ </span><span style="color: black;">Listener</span><span style="color: #080808;">.isAgentInstalled()</span><span style="color: #067d17;">"</span><span style="color: #080808;">)</span></span></span>
<span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><span style="color: #0033b3;">if </span><span style="color: #080808;">(</span><span style="color: black;">Listener</span><span style="color: #080808;">.isAgentInstalled()</span><span style="color: #080808;">) {</span></span></span>
<span style="color: #080808;"> </span><span style="color: #0033b3;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">try </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">{</span></span></span>
<span style="color: #080808;"> </span><span style="color: #0033b3;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">val </span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">b </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">= ByteArrayOutputStream()</span></span></span>
<span style="color: #080808;"> </span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">Listener</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.dump(</span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">b</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">)</span></span></span>
<span style="color: #080808;"> </span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">logger</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.info(</span></span></span><span style="color: #067d17;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">"The current open files Listener dump = </span></span></span><span style="color: #0037a6;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">$</span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">b</span></span></span><span style="color: #067d17;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">"</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">)</span></span></span>
<span style="color: #080808;"> </span><span style="color: #0033b3;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">val </span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">currentOpenFiles </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">= </span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">Listener</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.getCurrentOpenFiles()</span></span></span>
<span style="color: #080808;"> </span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">logger</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.info(</span></span></span><span style="color: #067d17;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">"The current open files Listener list size = </span></span></span><span style="color: #0037a6;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">${</span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">currentOpenFiles</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.</span></span></span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">size</span></span></span><span style="color: #0037a6;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">}</span></span></span><span style="color: #067d17;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">"</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">)</span></span></span>
<span style="color: #080808;"> </span></pre><pre class="western"><span style="color: #0033b3;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"> var </span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">jarFilesCounter </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">= </span></span></span><span style="color: #1750eb;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">0</span></span></span>
<span style="color: #1750eb;"> </span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">currentOpenFiles</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.</span></span></span><span style="color: #00627a;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><i>forEach </i></span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><b>{</b></span></span></span>
<span style="color: #080808;"> </span><span style="color: #0033b3;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">when </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">(</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><b>it</b></span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">) {</span></span></span>
<span style="color: #080808;"> </span><span style="color: #0033b3;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">is </span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">Listener</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.</span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">FileRecord </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">-> {</span></span></span>
<span style="color: #080808;"> </span><span style="color: #0033b3;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">if </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">(!</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><b>it</b></span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.</span></span></span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">file</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.</span></span></span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><i>name</i></span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.</span></span></span><span style="color: #00627a;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><i>endsWith</i></span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">(</span></span></span><span style="color: #067d17;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">"jar"</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">)) {</span></span></span>
<span style="color: #080808;"> </span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">logger</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.info(</span></span></span><span style="color: #067d17;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">"File named " </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">+ </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><b>it</b></span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.</span></span></span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">file </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">+ </span></span></span><span style="color: #067d17;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">" is opened by thread:" </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">+ </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><b>it</b></span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.</span></span></span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">threadName</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">)</span></span></span>
<span style="color: #080808;"> </span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">} </span></span></span><span style="color: #0033b3;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">else </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">{</span></span></span>
<span style="color: #080808;"> </span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">jarFilesCounter</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">++</span></span></span>
<span style="color: #080808;"> <span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">}</span></span></span>
<span style="color: #080808;"> <span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">}</span></span></span>
<span style="color: #080808;"> </span><span style="color: #0033b3;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">else </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">-> </span></span></span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">logger</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.info(</span></span></span><span style="color: #067d17;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">"Found record by Listener is not a file record, skipping"</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">)</span></span></span>
<span style="color: #080808;"> <span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">}</span></span></span>
<span style="color: #080808;"> <span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><b>}</b></span></span></span>
<span style="color: #080808;"> </span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">logger</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.info(</span></span></span><span style="color: #067d17;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">"Of the open files, </span></span></span><span style="color: #0037a6;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">$</span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">jarFilesCounter</span></span></span><span style="color: #067d17;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"> are .jar files, those were skipped in the logging of the list of files currently open"</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">)</span></span></span>
<span style="color: #080808;"> </span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">b</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.close()</span></span></span><br /></pre><pre class="western"><span style="color: #080808;"> </span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">} </span></span></span><span style="color: #0033b3;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">catch </span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">(</span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">ex</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">: </span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">Exception</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">) {</span></span></span>
<span style="color: #080808;"> </span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">logger</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.error(</span></span></span><span style="color: #067d17;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">"Dump of current open files failed with exception: "</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">, </span></span></span><span style="color: black;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">ex</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">)</span></span></span>
<span style="color: #080808;"> <span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">}</span></span></span>
<span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;"><span style="color: #080808;">} </span><span style="color: #0033b3;">else </span><span style="color: #080808;">{</span></span></span>
<span style="color: #080808;"> </span><span style="color: #871094;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">logger</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">.info(</span></span></span><span style="color: #067d17;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">"Leak detector agent is not installed, so not dumping open files"</span></span></span><span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">)</span></span></span>
<span style="color: #080808;"><span style="font-family: JetBrains Mono, monospace;"><span style="font-size: 9pt;">}</span></span></span></pre></pre>Note I skipped the jars during logging, which I noticed count for a <i>lot </i>of the open files listed.<br /><br /></div><div>The <i>Listener.dump()</i> lists <i>all </i>open files, instead of showing how many times a given file is opened. I couldn't find anything mentioning the library does support this; would be a very useful feature.<br /><br />I noticed the open files count was always lower than when using the MXBean. My guess is that the MXBean <i>does </i>count the open Socket connections too. And thus is much more precise. <br /><br /><h2>Linux commands open files detection</h2><div>There are two ways in Kotlin (and Java) to execute a command on the command line:</div><div><br /></div><div><i><span> </span>p = Runtime.getRuntime().exec(command)</i></div><div><br /></div><div>and</div><div><br /></div><div><div><i><span> </span>val pb = ProcessBuilder(command, arguments)</i></div><div><i><span> </span>val startedProcess = pb.start()</i></div></div><p style="margin-bottom: 0in;">I tried both ways. My goal was to use the '<i>lsof</i>' command, but that was not available in the lambda runtime. Then I tried to get the user of the process. And the process ID of the lambda itself. Then via <i>/procs/fd</i> one could find what files are kept open by a give PID.</p><p style="margin-bottom: 0in;">These commands worked:</p><p style="margin-bottom: 0in;"><i> val pb = ProcessBuilder("sh", "-c", "echo $$")</i></p><p style="margin-bottom: 0in;"><i> val startedProcess = pb.start()</i></p><div><i><br /></i></div><div><div><i> val pb = ProcessBuilder("sh", "-c", "ls -al")</i></div><div><i> val startedProcess = pb.start()</i></div></div><div><i><br /></i></div><div><div><i> val pb = ProcessBuilder("sh", "-c", "ls -al /proc")</i></div><div><i> val startedProcess = pb.start()</i></div></div><div><i><br /></i></div><div><i><span> </span>p = Runtime.getRuntime().exec("id -u $userName")</i></div><div><br /></div><div><br /></div><div>These didn't work:</div><div><br /></div><div><i><span> </span>p = Runtime.getRuntime().exec("/proc/self/status")</i></div><div><i> p = Runtime.getRuntime().exec("echo PID = $$")</i></div><div><i> p = Runtime.getRuntime().exec("$$")</i></div><div><i> val pb = ProcessBuilder("echo", "$$")</i></div><div><i><span> </span>val pb = ProcessBuilder("$$")</i></div><div><i><span> </span>p = Runtime.getRuntime().exec(command) // Exited with value 2, so probably invalid command</i></div><div><div><br /></div></div><div>When I got the PID, I tried this to get the open FDs by this PID in different ways, but that failed:</div><div><br /></div><div><i><span> </span>val pb = ProcessBuilder("sh", "-c", "ls -al /proc/$pidFound/fd")</i></div><div><br /></div><div>After listing all files in the current directory via this command:</div><div><br /></div><div><i><span> </span>val pb = ProcessBuilder("sh", "-c", "ls -al /proc")</i></div><div><br /></div><div>I saw that none of the numbers (PIDs) listed there were matching the found PID! At this point I stopped further exploring this option, since then I wouldn't find the open files in <i>/proc/$pidFound/fd</i> anyway....</div><div><br /></div><div><h2>Force GC</h2>A theory why the <i>Too many open files</i> error is appearing was that the Java runtime doesn't get enough time to clean up (garbage collect) the opened file descriptors. </div><div>So to test this theory, I forced a Garbage Collect after each 50 invocations of the lambda instance. Of course calling <i>System.gc() </i>doesn't fully guarantee it will happen right at that moment, e.g when the runtime is too busy it will happen later.</div><div>To cater for that I also added a <i>Thread.sleep() </i>call. Yes this solution normally is a potential performance killer, but an option to verify the theory. This is the code I used:</div><div><br /></div><div><div><i> nrOfInvocationsOfThisLambdaInstance++</i></div><div><i> if (nrOfInvocationsOfThisLambdaInstance % 50L == 0L) {</i></div><div><i> logger.info("$nrOfInvocationsOfThisLambdaInstance % 50 == 0, so going to garbage collect")</i></div><div><i> try {</i></div><div><i> System.gc()</i></div><div><i> } catch (e: Throwable) {</i></div><div><i> logger.error("Unable to garbage collect due to: ${e.message}. Full details in the exception.", e)</i></div><div><i> }</i></div><div><i> logger.info("$nrOfInvocationsOfThisLambdaInstance % 50 == 0, so going to runFinalization")</i></div><div><i> try {</i></div><div><i> System.runFinalization()</i></div><div><i> } catch (e: Throwable) {</i></div><div><i> logger.error("Unable to runFinalization due to: ${e.message}. Full details in the exception.", e)</i></div><div><i> }</i></div><div><i><br /></i></div><div><i> logger.info("Going to sleep() so it hopefully gets time to GC...")</i></div><div><i> try {</i></div><div><i> Thread.sleep(5000)</i></div><div><i> } catch (i: InterruptedException) {</i></div><div><i> // ignore</i></div><div><i> }</i></div><div><i> }</i></div></div><div><br /></div><div>And indeed, the <i>Too many open</i> files error was gone! But this couldn't really be the final acceptable solution of course.</div><div><h2>Examine the code for keeping files open</h2></div><div>See below.</div><div><br /></div></div><h1 style="text-align: left;">Solution</h1><div>So at this point I only had some idea of how many open files there were at a certain point. I saw the number using the MXBean solution go up to about 1023 and then the <i>Too many open files</i> error started to appear.</div><div>In the code I <i>did</i> find a spot where it was opening and reading a configuration file on each incoming request! After moving that code into an object class (or <i>init</i>{} block, or <i>val</i> variable at class level), the <i>Too many open files</i> error started to appear already much later (as in: the number of open files count went up much slower and the exception occurred less).</div><div>So I was moving into the right direction! Also, the errors were all <i>SocketException</i>s now. After investigating the code some more and more, I noticed the <i>OkHttpClient</i> was getting created <i>each time an HTTP request to an external third party was made (</i>which is relatively slow of course<i>)</i>. After also moving this part into an object class, the error was completely gone!</div><div><br /></div><div><b>Conclusion</b>: the tools gave some more insights on what was going on, and I learned quite few things on how/where/when file descriptors are used in lambdas, but in the end the problem was found during plain old code examination :)</div><div><br /></div>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-32095501439179495712023-04-21T15:46:00.002+01:002023-04-21T15:46:26.286+01:00OWASP Dependency Check plugin suppressions.xml examples<h1 style="text-align: left;">Introduction</h1><div>One of the features of the <a href="https://owasp.org/www-project-dependency-check/">OWASP</a> <a href="https://github.com/jeremylong/DependencyCheck">dependency check plugin</a> is to be able to suppress reported vulnerabilities, for example because they are false-positives for your configuration, or no new version is available yet, so you want to suppress the alert for a certain period of time.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLBQafPIGdH3F9NrlqruFyr3RPsgARlpsm2EeUAHa7LmveQ7Js7vbFk1rCIE3mDBtkp-CvCQczyiOs4yoxjpyOQxXTXstHV_96G9EAU3q3ekaZdZgGC0G6BIUn_qtiITalhjBI3XsZDv1HDU--Noj-Hrr6zAmgoXy5tBQ5jJQfPMEN4WHAVLlbClQ9/s376/owasp-logo.webp" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="350" data-original-width="376" height="298" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLBQafPIGdH3F9NrlqruFyr3RPsgARlpsm2EeUAHa7LmveQ7Js7vbFk1rCIE3mDBtkp-CvCQczyiOs4yoxjpyOQxXTXstHV_96G9EAU3q3ekaZdZgGC0G6BIUn_qtiITalhjBI3XsZDv1HDU--Noj-Hrr6zAmgoXy5tBQ5jJQfPMEN4WHAVLlbClQ9/s320/owasp-logo.webp" width="320" /></a></div><div><br /></div><div><br /></div><div>Those suppressions you specify in the suppressions.xml file. The format is specified <a href="https://jeremylong.github.io/DependencyCheck/general/suppression.html">here</a>.</div><div><br /></div><div>But not all possibilities of suppressing have examples. Especially those where you just want to exclude a whole set of packages, e.g. everything of the <a href="https://spring.io/">Spring framework</a> starting with 'spring-', like 'spring-webflux', 'spring-web' etc for a given version.</div><div><br /></div><div>After some trial and here I came up with some more additional useful examples.</div><div><br /></div><h1 style="text-align: left;">Solution</h1><h2 style="text-align: left;">Setup</h2><div><ul style="text-align: left;"><li>Kotlin 1.8.10</li><li>Gradle</li><li>Spring Boot 2.7.9</li><li>failBuildOnCVSS set to 7</li><li>OWASP plugin versions tested: 7.2.1, 8.0.0</li></ul><h3 style="text-align: left;">Examples</h3></div><div><div><h4 style="text-align: left;">Reported vulnerabilities as HIGH</h4><div><ul><li>logback-core-1.3.0.jar</li><li>logback-classic-1.3.0.jar</li></ul><div>Suppressions:</div></div></div><ul style="text-align: left;"><li><i><packageUrl regex="true">^pkg:maven/ch\.qos\.logback/logback-core@1.3.*$</packageUrl><br /></i>Will not show logback-core anymore in the report as HIGH.<br /><br /></li><li><i><packageUrl regex="true">^pkg:maven/ch\.qos\.logback/logback.*@1.3.*$</packageUrl></i><br />Will not report neither logback-core nor logback-classic anymore as vulnerabilities.</li></ul><div>Full example of the suppression:</div><div><br /></div><div><div><i> <suppress until="2023-10-01Z"></i></div><div><i> <notes><![CDATA[</i></div><div><i> No new version exists yet for any version after this version.</i></div><div><i> ]]></notes></i></div><div><i> <packageUrl regex="true">^pkg:maven/ch\.qos\.logback/logback.*@1.3.*$</packageUrl></i></div><div><i> <vulnerabilityName>CVE-2021-42550</vulnerabilityName></i></div><div><i> </suppress></i></div></div><div><br /></div><div><br /></div></div><div><div><h4 style="text-align: left;">Reported vulnerabilities as HIGH</h4><div><ul><li>spring-webflux-5.3.25.jar</li><li>spring-messaging-5.3.25.jar</li></ul><div>Suppressions:</div></div></div><ul></ul></div><div><ul style="text-align: left;"><li><i><packageUrl regex="true">^pkg:maven/org\.springframework/spring-.*@5.3.25$</packageUrl></i><br />Will not report neither of the two as vulnerabilities anymore.</li></ul><div><div>Notice the "<i>.*</i>" used!</div><div><br /></div><div>Full example of the suppression:<br /><br /></div><div><div><i> <suppress until="2023-10-01Z"></i></div><div><i> <notes><![CDATA[</i></div><div><i> No new version exists yet for any version after 5.3.26, which has the same issue.</i></div><div><i> ]]></notes></i></div><div><i> <packageUrl regex="true">^pkg:maven/org\.springframework/spring-.*@5.3.25$</packageUrl></i></div><div><i> <vulnerabilityName>CVE-2023-20860</vulnerabilityName></i></div><div><i> <vulnerabilityName>CVE-2016-1000027</vulnerabilityName></i></div><div><i> </suppress></i></div></div></div></div><div><br /></div><div><br /></div><div>And here some examples that <i>don't</i> work:</div><div><ul style="text-align: left;"><li><i><packageUrl regex="true">^pkg:maven/ch\.qos\.logback/logback-*@1.3.*$</packageUrl></i><br />Shows both logback-core and logback-classic again in the report.<br /><br /></li><li><i><packageUrl regex="true">^pkg:maven/ch\.qos\.logback/logback*@1.3.*$</packageUrl><br /></i>Shows both logback-core and logback-classic again in the report.</li></ul></div><div>Another thing to know from <a href="https://github.com/jeremylong/DependencyCheck/issues/5658">here</a>: the vulnerabilities report can show an issue as MEDIUM, while the vulnerability reports as a 8.5 in the CVSSv2 ranking, while the CVSSv3 rates it at 6.6. So the report seems to take only the CVSSv3 value into account for the Highest Severity level.</div><div><br /></div><div><br /></div>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-41221397378516263202023-03-16T12:24:00.006+00:002023-03-24T13:08:38.949+00:00Datadog: Malformed _X_AMZN_TRACE_ID value Root - also known as X-Amzn-Trace-Id<h1 style="text-align: left;">Introduction</h1><div>Since 14 March 2023 suddenly my AWS lambdas started to log this error:</div><div><br /></div><div><div><i>datadog: Malformed _X_AMZN_TRACE_ID value: Root=1-6411cb3d-e6a0db584029dba86a594b7e;Parent=8c34f5ad8f92d510;Sampled=0;Lineage=f627d632:0</i></div></div><div><br /></div><div>Note that the lambda processing was finishing normally, this metrics logging to Datadog is happening apparently in the background.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXJi-CHzt3ASfACwSDnM40WNRs6qgKggWJstjNMmapWWwFI5o7Rkte-jyfiRDVoJyW4Uu2WAQe11zgGx9rot-gMb9PCsYgcFRSlXpaQrpgPeXD503uadgY3h6jxJCez_ZFelaFoEmCY3FqqOxMvAZdD2UDF1xK490-hMtqWWfywwDZu3_UZMF4E4LY/s1229/footsteps.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="360" data-original-width="1229" height="118" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXJi-CHzt3ASfACwSDnM40WNRs6qgKggWJstjNMmapWWwFI5o7Rkte-jyfiRDVoJyW4Uu2WAQe11zgGx9rot-gMb9PCsYgcFRSlXpaQrpgPeXD503uadgY3h6jxJCez_ZFelaFoEmCY3FqqOxMvAZdD2UDF1xK490-hMtqWWfywwDZu3_UZMF4E4LY/w400-h118/footsteps.jpg" width="400" /></a></div><div><br /></div><h2 style="text-align: left;">Setup</h2><div><ul style="text-align: left;"><li>AWS lambda</li><li><a href="https://www.datadoghq.com/">Datadog</a></li><li>Gradle dependency library implementation "<a href="https://github.com/DataDog/datadog-lambda-java">com.datadoghq:datadog-lambda-java:1.4.5</a>"</li></ul><h1 style="text-align: left;">Investigation</h1><div>After a lot of searching, I found out that the datadog-lambda-java library was causing the issue, since that same day the issue was reported <a href="https://github.com/DataDog/datadog-lambda-java/issues/89">here</a> in its Github repository.</div><div>Which also seems to point to the code that is the culprit: the <i>length != 3</i> is assuming that the trace field will always consist of exactly 3 parts. But its specifications allow for more, so it seems AWS added another part.</div><div><div>The definition of the header can be found <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-request-tracing.html">here</a> and <a href=" https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-tracingheader">here</a> where the examples still have 3 elements (parts separated by a ';'), but can now be 4.</div></div><div><br /></div><h1 style="text-align: left;">Solution</h1><div><div>A patch has been posted, but as the 3rd comment says, the library is deprecated anyway. Here is the <a href=" https://github.com/DataDog/datadog-lambda-java/issues/89#issuecomment-1470156639">upgrade guide</a>.</div><div> </div><div><b>UPDATE </b>24 March 2023: the patch has been applied and a new release has been made! See <a href="https://github.com/DataDog/datadog-lambda-java/pull/90">https://github.com/DataDog/datadog-lambda-java/pull/90</a> for the <a href=" https://github.com/DataDog/datadog-lambda-java/releases/tag/v1.4.10">1.4.10 release</a>.<br /><br /></div><div> </div><div> </div><div> <br /></div></div><div><br /></div></div>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-45853800837042166772023-02-15T12:56:00.012+00:002023-04-21T15:59:01.619+01:00AWS SAM CLI FileNotFoundError: WinError 3: The system cannot find the path specified .class class Kotlin 1.7 Windows 10<h1 style="text-align: left;">Introduction</h1><p>The AWS SAM CLI command </p><p><i>sam.cmd build MyFunction --template C:\techie\workspace\my-function\local\template.yaml --build-dir C:\techie\workspace\my-function\local\.aws-sam\build --debug </i></p><p>fails in an IntelliJ commandline terminal due to this<i> </i></p><p><i>FileNotFoundError: [WinError 3] The system cannot find the path specified </i></p><p>error<i>.</i></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioLnAnr975jo2X7ifp_JzAr3EKkS3Or6XbOPsNT9gbJZQd6gmrYdTXZjklnDTv52To5Nxt1KVzemjE5bRAQnZW1DXV7ZdgVSL1RcKqkibRl44ClCksX6lYU2kf14x5qxxavbSuI0SxMd3UqQ7-cyX4yfu4NbSXUbMj0lWqxNuwCDLCvVY5kwvnbC_M/s600/aws-cli.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="315" data-original-width="600" height="210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioLnAnr975jo2X7ifp_JzAr3EKkS3Or6XbOPsNT9gbJZQd6gmrYdTXZjklnDTv52To5Nxt1KVzemjE5bRAQnZW1DXV7ZdgVSL1RcKqkibRl44ClCksX6lYU2kf14x5qxxavbSuI0SxMd3UqQ7-cyX4yfu4NbSXUbMj0lWqxNuwCDLCvVY5kwvnbC_M/w400-h210/aws-cli.png" width="400" /></a></div><br /><i><br /></i><p></p><h3 style="text-align: left;">Setup</h3><p>- Windows 10 Pro laptop</p><p>- IntelliJ 2022</p><p>- Kotlin 1.7</p><p>- Java 11 at least for compilation</p><p>- Serverless lambda written in Kotlin</p><p>- <a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html">AWS SAM CLI</a>, version 1.67.0</p><p>- <a href="https://docs.aws.amazon.com/toolkit-for-jetbrains/latest/userguide/welcome.html">AWS Toolkit plugin for IntelliJ</a> </p><h1 style="text-align: left;">Investigation</h1><p>First I tried to install SAM AWS CLI using HomeBrew (formerly Brew) in WSL 1 (ubuntu) under Windows 10 using these steps: </p><p><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-homebrew.html">https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-homebrew.html</a></p><p>But that failed during Homebrew installation. Probably upgrading to WSL 2 would fix that. But then I also realized: IntelliJ then doesn't know about that at all, since SAM CLI is then installed in WSL, not in Windows.</p><p>It kept on saying after running <i>brew postinstall --verbose --debug gcc</i>: </p><p><i>==> Installing aws-sam-cli from aws/tap</i></p><p><i>Error: An exception occurred within a child process:</i></p><p><i> Errno::EFAULT: Bad address - /home/linuxbrew/.linuxbrew/bin/gcc-12</i></p><p>And also:</p><p><i>Warning: The post-install step did not complete successfully</i></p><p><i>You can try again using:</i></p><p><i> brew postinstall gcc</i></p><p>Also trying <i>brew postinstall --verbose --debug gcc</i> didn't succeed. This error mentioned here was also applicable: <a href="https://github.com/orgs/Homebrew/discussions/4052">https://github.com/orgs/Homebrew/discussions/4052</a></p><p>I also didn't dare <i>wsl --update</i> because other configurations I already had set up might fail after that. Guess I will do that at a more quiet moment :)</p><p>So then I went for the manual installation in Windows, as found here: <a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html">https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html</a></p><p>In IntelliJ you then have to set the path to the executable to where you installed it:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvhlOewHpE-kWUxD_wIvOuecajgK-JLZRA8VFn8mNAqLM3DaaKZXjVOx3TYJG1H5xM3j0Nz7PblIGzQT8fcJ0dE3_jiJCJJfFPtVroR9d58GAOoU5bgtiSgcsN3pYDTCiFgS-UWPPBwzOM1VqQZFv23zu3SujQ_G4yopbnN5X6IP7Y6inCRLptxwZU/s1205/blogpost-file-not-found-aws-sam-cli-windows.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="885" data-original-width="1205" height="294" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvhlOewHpE-kWUxD_wIvOuecajgK-JLZRA8VFn8mNAqLM3DaaKZXjVOx3TYJG1H5xM3j0Nz7PblIGzQT8fcJ0dE3_jiJCJJfFPtVroR9d58GAOoU5bgtiSgcsN3pYDTCiFgS-UWPPBwzOM1VqQZFv23zu3SujQ_G4yopbnN5X6IP7Y6inCRLptxwZU/w400-h294/blogpost-file-not-found-aws-sam-cli-windows.PNG" width="400" /></a></div><br /><p>So IntelliJ will use the Windows AWS SAM CLI, not the one in the terminal (WSL 1).</p><p>Than I ran my command, first outside IntelliJ to be able to control the parameters more easily:</p><p><i>C:\Users\techie>C:\Amazon\AWSSAMCLI\bin\sam.cmd build MyFunction --template C:\techie\workspace\my-function\local\template.yaml --build-dir C:\techie\workspace\my-function\local\.aws-sam\build --debug</i></p><p>But that gave this error:</p><p><i>FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\techie\\AppData\\Local\\Temp\\tmpmjdhug40\\7c98ad184709dded6b1c874ece2a0edea9c55b0a\\build\\classes\\kotlin\\test\\com\\mycompany\\myfunction\\domain\\MyServiceTest$should register valid make this a long text to exceed this successfully$2.class'</i></p><p>First I thought it was due to spaces in the test-methodname. But replacing them with an underscore didn't work either. Or maybe case-insensitive-ness; but my test-methodname is all lowercase.</p><p>I tried many things to exclude the whole test-class from the process, because why is it even included during the <i>sam build</i> command? </p><p>Options I tried were:</p><p></p><ol style="text-align: left;"><li>Setting GRADLE_OPTS before running the command <i>-x test</i>, similar to the MAVEN_OPTS example here: <a href="https://github.com/aws/aws-sam-cli/issues/1105#issuecomment-777703158">https://github.com/aws/aws-sam-cli/issues/1105#issuecomment-777703158</a></li><li>Excluding the test-file or even just all tests in build.gradle, like:<br /><br /><p><i>jar {</i></p><p><i> sourceSets {</i></p><p><i> main {</i></p><p><i> java {</i></p><p><i> exclude '**/TestExcludeClass.java'</i></p><p><i> }</i></p><p><i><br /></i></p><p><i> kotlin {</i></p><p><i> exclude '**/TestExcludeKotlinClass.kt'</i></p><p><i> }</i></p><p><i> }</i></p><p><i> }</i></p><p><i>}</i></p>Note that excluding everything with 'exclude '**/*.kt' did make the sam build fail, so the changes were taken into account.</li><li>In build.gradle add: <i>excludeTestsMatching "com.mycompany.myfunction.lambda.MyServiceTest"</i></li><li><i>test.onlyIf { ! Boolean.getBoolean(skipTests) }</i> and then specifying as <i>GRADLE_OPTS="-DskipTests=true"</i></li></ol><p></p><p>It seems the initial step of the sam build (<i>JavaGradleWorkflow:JavaGradleCopyArtifacts</i>) just takes all .class files anyway, even test classes.</p><p>I tried turning off <a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html">Telemetry</a><i> </i>That did have no affect on the error.</p><h1 style="text-align: left;">Solution</h1><p>Then I found this comment: <a href="https://github.com/aws/aws-sam-cli/issues/4031#issuecomment-1173730737">https://github.com/aws/aws-sam-cli/issues/4031#issuecomment-1173730737</a></p><p>Could it be that, that the path is just too long? Typical Windows limit so that could definitely be it.</p><p>And yes after applying below command in PowerShell as an Administrator</p><p><i>New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" `-Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force</i></p><p>it worked!</p><p>Detailed explanation: <a href="https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=powershell#enable-long-paths-in-windows-10-version-1607-and-later">https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=powershell#enable-long-paths-in-windows-10-version-1607-and-later</a></p><p>The command ran fine! This is then the output:</p><p><i>Build Succeeded</i></p><p><i>Built Artifacts : ..\..\techie\workspace\my-function\local\.aws-sam\build</i></p><p><i>Built Template : ..\..\techie\workspace\my-function\local\.aws-sam\build\template.yaml</i></p><p><i>Commands you can use next</i></p><p><i>=========================</i></p><p><i>[*] Validate SAM template: sam validate</i></p><p><i>[*] Invoke Function: sam local invoke -t ..\..\techie\workspace\my-function\local\.aws-sam\build\template.yaml</i></p><p><i>[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch</i></p><p><i>[*] Deploy: sam deploy --guided --template-file ..\..\techie\workspace\my-function\local\.aws-sam\build\template.yaml</i></p><p><br /></p><p><br /></p>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-16595666948806560542022-12-21T11:00:00.001+00:002022-12-21T11:00:00.202+00:00Where to find .gitattributes on Windows 10/11 using Git, IntelliJ 2022 and WSL Ubuntu to fix CRLF (\r\n) command not found: convert to LF line endings on checkout<h1 style="text-align: left;">Introduction</h1><p>Problem: when checking out a project in IntelliJ in Windows, all files are checked out with Window's newline CRLF (\r\n).</p><p>But if you then open a terminal in IntelliJ which runs WSL (Ubuntu) and you want to run a bash script like this shell script, you'll get this error:</p><p><i>#!/bin/sh</i></p><p><i>set -e</i></p><p><i>[unrelated stuff deleted]</i></p><p>It will fail with: </p><p><i>./deploy.sh: line 2: $'\r': command not found</i></p><p><i>: invalid optione 3: set: -</i></p><p><i>set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]</i></p><p>Those errors are caused by the shell script having CRLF instead of just LF, which Ubuntu/Mac OS expects.</p><p>Sidenote: I made softlink from <i>/bin/sh</i> to bash in my system because dash does not support several features, see <a href="https://askubuntu.com/questions/854279/sh-0-illegal-option">here</a>. </p><p><br /></p><p>So I tried setting IntelliJ's Code Style to use \n for newlines and line endings and line separator, also for new projects like mentioned <a href="https://www.jetbrains.com/help/idea/configuring-line-endings-and-line-separators.html">here</a>.</p><p>But still after creating a new project from Git, all files including the above script was set to CRLF. You can see this at the bottom of the screen in this screenshot:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7V8H47XP_dLpt8FV81n21g7X1JOlKa5xn-Y6n9LeDz8cZlD5v1wTlthuJiuKMZ0Q4YkL7ZEiNl7ZZ_GiSX2XJ-o3ThbUMQskHr7F8VFr_1MLnlU2UgLQvhgmxQjoDI8wJLkWy7IekSjDPrZrHEQppZT673O-jNtVjMU9OzhD8niLiJYxtwZCXZ46C/s299/blogpost-gitattributes-default-git-cline.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="137" data-original-width="299" height="183" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7V8H47XP_dLpt8FV81n21g7X1JOlKa5xn-Y6n9LeDz8cZlD5v1wTlthuJiuKMZ0Q4YkL7ZEiNl7ZZ_GiSX2XJ-o3ThbUMQskHr7F8VFr_1MLnlU2UgLQvhgmxQjoDI8wJLkWy7IekSjDPrZrHEQppZT673O-jNtVjMU9OzhD8niLiJYxtwZCXZ46C/w400-h183/blogpost-gitattributes-default-git-cline.PNG" width="400" /></a></div><br /><div><div>I also realised that I don't want <b>all</b> files to have just LF, because I noticed doing that manually, on my commit of those files, I had to commit these too, as the newline changed. But I didn't want to bother other teammembers on Mac Books with this unnecessary change. Similar to what happens when you do a <i>dos2unix</i> on the file.</div><div>So the next to try was to get Git on my machine to handle it correctly. And thus hopefully IntelliJ too.</div><div>The .gitattributes file seemed a very good candidate: a suggestion was to change the git config 'core.autocrlf'. But that meant all the local files were getting changed still if I understood correctly, which I don't want.</div></div><div><br /></div><h1 style="text-align: left;">Solution</h1><div><div>It cost a lot of effort to find out where the gitattributes file is currently; and I wanted to change it for all my Git commands in the future, not just for one project.</div><div>What I wanted to change it to is mentioned <a href="https://askubuntu.com/questions/966488/how-do-i-fix-r-command-not-found-errors-running-bash-scripts-in-wsl">here</a>:</div><div><br /></div><div><i># Convert to LF line endings on checkout.</i></div><div><i>*.sh text eol=lf</i></div><div><i>**/*.sh eol=lf</i></div><div><br /></div><div>Those lines specify to change all end of lines (newlines) of files ending with .sh to be checked out with a LF (\n).</div><div>I also added the 3rd line myself, to specify subdirectories, but maybe that was not needed.</div><div><br /></div><div>It was hard to find the correct location for the gitattributes (or .gitattributes <i>file</i> which made it all more confusing), <a href="https://git-scm.com/docs/gitattributes">this official Git text</a> wasn't super-clear on it: </div><div>This guy also had <a href="https://stackoverflow.com/questions/32847697/windows-specific-git-configuration-settings-where-are-they-set">a challenge on finding their location</a>.</div><div><br /></div><div>Finally I found my gitattributes file on Windows here:</div><div><br /></div><div><i>C:\git\etc\gitattributes</i></div><div><br /></div><div>C:\git is actually where I installed the Git client. And it is the one IntelliJ uses (but not the one the terminal shell in IntelliJ uses!) </div><div><br /></div><div>And there were already quite a few settings in there like:</div><div><br /></div><div><i>*.RTF<span style="white-space: pre;"> </span>diff=astextplain</i></div><div><i>*.doc<span style="white-space: pre;"> </span>diff=astextplain</i></div><div><i>*.DOC<span style="white-space: pre;"> </span>diff=astextplain</i></div><div><br /></div><div>I just appended these:</div><div><br /></div><div><i>**/*.sh eol=lf</i></div><div><i>*.sh eol=lf</i></div><div><br /></div><div>Restarted IntelliJ to be sure. And yes, after a complete full new checkout of my Git project, Java and Kotlin files were still having CRLF line endings, and my shell script had the LF ending.</div></div>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-14982091018631709332022-12-18T14:32:00.001+00:002022-12-18T14:32:27.897+00:00Docker build with Git command running in CircleCI failing with: Fatal: No names found, cannot describe anything, invalid argument, for "-t, --tag" flag: invalid reference format<h1 style="text-align: left;">Introduction</h1><p>Context: <a href="https://www.docker.com/">Docker</a>, <a href="https://circleci.com/">CircleCI</a>, <a href="https://github.com/">Github</a>.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9hUbct-RZg0RfEQR_HBjIXUfYifHkmfReyo-f4xXW1UychP97e14VWk4n2JXfUCpFA-yfF8FSpEytMXE1L-S3hUKLzttrNCMyI9c78wPPUTWypC4TPjnpe5OB_QsxwPNkzwUWIr_h9tv5OVXllhweW2otb82S2Twzkw7DKPWVOjW5NrPhTWxYscEb/s800/blogpost-docker-build-git-error-message.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="323" data-original-width="800" height="161" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9hUbct-RZg0RfEQR_HBjIXUfYifHkmfReyo-f4xXW1UychP97e14VWk4n2JXfUCpFA-yfF8FSpEytMXE1L-S3hUKLzttrNCMyI9c78wPPUTWypC4TPjnpe5OB_QsxwPNkzwUWIr_h9tv5OVXllhweW2otb82S2Twzkw7DKPWVOjW5NrPhTWxYscEb/w400-h161/blogpost-docker-build-git-error-message.png" width="400" /></a></div><p>The Docker build command </p><p><i>docker build -f .circleci/Dockerfile -t $AWS_ACCOUNT_ID.ecr.$AWS_DEFAULT_REGION.amazonaws.com/${CIRCLE_PROJECT_REPONAME}:`git describe --tags` -t $AWS_ACCOUNT_ID.ecr.$AWS_DEFAULT_REGION.amazonaws.com/${CIRCLE_PROJECT_REPONAME}:${CIRCLE_BUILD_NUM} -t $AWS_ACCOUNT_ID.ecr.$AWS_DEFAULT_REGION.amazonaws.com/${CIRCLE_PROJECT_REPONAME}:${CIRCLE_SHA1} .</i></p><p>was failing with this message:</p><p><i>fatal: No names found, cannot describe anything.</i></p><p><i>invalid argument "********************************************/my-project:" for "-t, --tag" flag: invalid reference format</i></p><p><i>See 'docker build --help'.</i></p><p><i>Exited with code exit status 125</i></p><h1 style="text-align: left;">Solution</h1><div>You'd expect the Docker command maybe being syntactically incorrect. But the error message is referring to something else: it turns out the <i>git describe --tags</i> command gave the fatal message.</div><div><div>The cause was that there was no git-tag set at all on the Github project yet. After manually creating a release (including a tag) on Github and running the above build command again, the <i>docker build</i> command succeeded.</div></div><div><br /></div>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-41677546592431476762022-11-30T10:56:00.035+00:002022-11-30T10:56:00.226+00:00Flyway FlywayValidateException Validate failed: Migrations have failed validation. Detected failed migration to version 1 solution, please remove any half-completed changes then run repair to fix the schema history<h1 style="text-align: left;">Introduction</h1><div><div>Setup:</div><div><ul style="text-align: left;"><li>Spring boot 2.7.4</li><li>Kotlin 1.7.20</li></ul></div></div><div><br /></div><div><div>Flyway dependencies:</div><div><br /></div><div><i>plugins {</i></div><div><i> id "org.flywaydb.flyway" version "9.6.0"</i></div><div><i>}</i></div><div><i><br /></i></div><div><i>project(":application") {</i></div><div><i> dependencies {</i></div><div><i><br /></i></div><div><i> dependencies {</i></div><div><i><br /></i></div><div><i> implementation "org.flywaydb:flyway-core:9.6.0"</i></div><div><i> implementation "org.flywaydb:flyway-mysql:9.6.0"</i></div><div><i> }</i></div><div><i>}</i></div></div><div><br /></div><div><div>With only this <a href="https://flywaydb.org/">Flyway</a> script V1__initial_script present in the Spring Boot application:</div><div><br /></div><div><i>create table order</i></div><div><i>(</i></div><div><i> id varchar(255) primary key,</i></div><div><i> orderId bigint(20) not null</i></div><div><i>);</i></div><div><i><br /></i></div><div><i>create index order_id_idx on order(orderId);</i></div><div><i><br /></i></div><div><i>create table order_entry</i></div><div><i>(</i></div><div><i> id varchar(255) primary key,</i></div><div><i> orderid <span> </span>bigint(20) not null,</i></div><div><i> desc varchar(255) not null,</i></div><div><i> UNIQUE (oderId),</i></div><div><i> constraint fk_orderId foreign key (orderId) references order (id)</i></div><div><i>);</i></div><div><br /></div><div>gave this error:</div></div><div><br /></div><div><div><i>Invocation of init method failed; nested exception is org.flywaydb.core.api.exception.FlywayValidateException: Validate failed: Migrations have failed validation</i></div><div><i>Detected failed migration to version 1 (initialize schema).</i></div><div><i>Please remove any half-completed changes then run repair to fix the schema history.</i></div><div><i>Need more flexibility with validation rules? Learn more: https://rd.gt/3AbJUZE</i></div></div><div><br /></div><div>See also the below screenshot:</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFB4KW9jkVn4Wz4MLD5tamE8t73yC334_mLSVMii7A80G-gdkNyeCOaxzOBUEFpV1RCVpYWx-ihJy_tnc5fUdTLeKlL5x23pVto-4MNS-pfbtoQ0JGev8lCtox2lPavcbhp1WgYiRfxggScGmFC6Q4xZ172OqHXRSl8KPlameyL7_HHWJasiz9Jf8s/s1405/blogpost-flyway-unclear-error-message-edited.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="421" data-original-width="1405" height="192" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFB4KW9jkVn4Wz4MLD5tamE8t73yC334_mLSVMii7A80G-gdkNyeCOaxzOBUEFpV1RCVpYWx-ihJy_tnc5fUdTLeKlL5x23pVto-4MNS-pfbtoQ0JGev8lCtox2lPavcbhp1WgYiRfxggScGmFC6Q4xZ172OqHXRSl8KPlameyL7_HHWJasiz9Jf8s/w640-h192/blogpost-flyway-unclear-error-message-edited.PNG" width="640" /></a></div><br /><div>So the error is very vague. I also knew for sure that the schema script was the very first Flyway script to be run for that database, so it could not be the case that I edited the script after it already had run before succesfully.</div><h1 style="text-align: left;">Solution</h1><div><div>The <a href="https://flywaydb.org/documentation/usage/errorcodes/">error code documentation</a> doesn't tell much either. No details at all. </div><div><br /></div><div>But when I ran the contents of the above script on the SQL command line in <a href="https://dbeaver.io/">DBeaver</a> you get the exact details of the problem:</div></div><div><div><br /></div><div><i>org.jkiss.dbeaver.model.sql.DBSQLException: SQL Error [3780] [HY000]: Referencing column 'orderId' and referenced column 'id' in foreign key constraint 'fk_orderId' are incompatible.</i></div><div><i><span style="white-space: pre;"> </span>at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:133)</i></div><div><i><span style="white-space: pre;"> </span>...</i></div><div><br /></div><div>So Flyway just doesn't log the details of the error at all. I also tried the Flyway validate in a bean, to see if I could inspect the exception hopefully returned in the <i>validateWithResult </i>variable:</div></div><div><div><br /></div><div><i> @Bean</i></div><div><i> fun myFlyway(dataSource: DataSource): String {</i></div><div><i> val flyway = Flyway.configure().dataSource(dataSource).load()</i></div><div><i> val validateWithResult = flyway.validateWithResult()</i></div><div><i> logger.info("Flyway validate = ${validateWithResult}");</i></div><div><i> }</i></div><div><br /></div><div>But that only showed again the high level error:</div><div><br /></div><div><i>Migrations have failed validation</i></div><div><i>Error code: FAILED_VERSIONED_MIGRATION</i></div><div><i>Error message: Detected failed migration to version 1 (initialize schema). Please remove any half-completed changes then run repair to fix the schema history.</i></div><div><br /></div><div><div>I found out that older Flyway versions used to have <a href="https://stackoverflow.com/questions/65845380/existing-flyway-migrations-fail-validation-after-upgrade-to-spring-boot-2-4-2">a bug not showing the whole exception</a>,</div><div>But even in this most version 9.6.0 the full exception is not logged.</div></div></div><div><br /></div><div><div>I also tried configuring Flyway with outputting JSON as mentioned <a href="https://github.com/flyway/flyway/issues/2987">here</a> and <a href="https://flywaydb.org/documentation/usage/commandline/#machine-readable-output">here</a>:</div><div><br /></div><div><i>flyway {</i></div><div><i> outputType="json"</i></div><div><i>}</i></div><div><br /></div><div>But outputType can't be used here. Didn't further investigate how to specify that parameter, the <i>@Bean</i> has no option for it. Maybe it is possible via application.yml...</div><div><br /></div><div>Update: it actually seems only to be available for the command-line option: <a href="https://documentation.red-gate.com/fd/command-line-flyway-validate-167936471.html">https://documentation.red-gate.com/fd/command-line-flyway-validate-167936471.html</a> and <a href="https://flywaydb.org/documentation/usage/gradle/validate">https://flywaydb.org/documentation/usage/gradle/validate</a></div><div>Though it seems it should be possible via <a href="https://flywaydb.org/documentation/configuration/parameters/">configuration files and environment variables</a>.</div></div><div><br /></div><div>Thus in the end the solution for me was to run the script manually against the database to see what the exact error is. Running <i>repair</i> is also not the correct suggestion, because the script just contained some syntax/semantic errors. It looks like a Flyway bug that the exact error details aren't logged by default, and I couldn't get it to do that either.</div><div><br /></div><div><br /></div><div><br /></div><div><br /></div><div><br /></div><div><br /></div><div><br /></div><div><br /></div><div><br /></div><div><br /></div>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-72663208223235185322022-11-16T11:00:00.001+00:002022-11-16T11:00:00.199+00:00Spring JDBC and MySql using UUIDs in Java and VARCHAR(36) in database incorrect string value solution<h1 style="text-align: left;">Introduction <br /></h1><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjX8C2-a16VRrcDarIXL7TqcJd_3JVPmSksesu_4CjccDVDoNjgospZx30LfcPliKf9BjhfBc1R14q88aY-C9SNQriliZDeG82I2F0rDngO_kuFPhI0duQG3KiHhbDavIL5lLqIOCnTM9Ob4iEAmzHOrWu7YdvbwIJ2N_ZDgTbTQGJVnHo_quPEXG64/s800/blogpost-mysql-uuid-image.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="450" data-original-width="800" height="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjX8C2-a16VRrcDarIXL7TqcJd_3JVPmSksesu_4CjccDVDoNjgospZx30LfcPliKf9BjhfBc1R14q88aY-C9SNQriliZDeG82I2F0rDngO_kuFPhI0duQG3KiHhbDavIL5lLqIOCnTM9Ob4iEAmzHOrWu7YdvbwIJ2N_ZDgTbTQGJVnHo_quPEXG64/s320/blogpost-mysql-uuid-image.png" width="320" /></a></div><p></p><p>Using <a href="https://www.h2database.com/html/main.html">H2</a> as initial embedded database for a Spring Boot application worked fine. H2 is very forgiving in many situations and of course only tries to emulate the real target database, in this case <a href="https://dev.mysql.com/doc/relnotes/mysql/8.0/en/">MySql 8.0</a>.<br />So after connecting my Spring Boot application to MySql, this error started to appear when inserting a row in a table with a java.util.UUID property as 'id' field: <br /><br /><i> java.sql.SQLException: Incorrect string value: '\xAC\xED\x00\x05sr...' for column 'id' at row 1</i><br /><br />A quick internet search showed that potentially my character set and collate setting for the tables were using 3 bytes instead of 4 for UTF-8 storage.<br />But the database, tables and columns all had utf8mb4 specified as CHARACTER SET and COLLATE, since I'm using MySql 8.0. So that 3 vs 4 bytes for UTF-8 issue did not apply for me. </p><h1 style="text-align: left;">Solution<br /></h1><p>Then I found this blog <a href="https://petrepopescu.tech/2021/01/how-to-use-string-uuid-in-hibernate-with-mysql/">https://petrepopescu.tech/2021/01/how-to-use-string-uuid-in-hibernate-with-mysql/</a> that at least when using Hibernate, it doesn't know how to convert UUIDs to strings (varchars), and you need to specify a Hibernate-provided converter. But the Hibernate annotation of course did not work for <a href="https://spring.io/projects/spring-data-jdbc">Spring Data JDBC</a>.<br /><br />Luckily there is a way to <a href="https://docs.spring.io/spring-data/jdbc/docs/current/reference/html/#jdbc.entity-persistence.custom-converters">write your own converters for Spring JDBC datatypes</a>. <br /><br />Implementing that fixed that initial error message. Note the example in the above post is missing the <i>@Configuration</i> annotation on the <i>MyJdbcConfiguration </i>class.<br /><br />But then the error happened again during this custom JdbcTemplate select query:<br /><br /> <i>String query = "SELECT DISTINCT r.id, user_id FROM recipe r WHERE user_id = ?";<br /> List<Object> parameterValues = new ArrayList<>();<br /> parameterValues.add(userId);<br /> Object[] parameterValuesArray = parameterValues.toArray();<br /> jdbcTemplate.query(query, parameterValuesArray, new JdbcRecipeRowMapper());</i><br /><br />This was the output of that query, including the parameters used in the query:<br /><br /><i>Executing prepared SQL statement [SELECT DISTINCT r.id, user_id, r.name, vegetarian, number_of_servings, instructions, r.created_at, r.updated_at FROM recipe r INNER JOIN ingredient i ON r.id = i.recipe_id WHERE user_id = ? AND vegetarian = ? AND number_of_servings = ? AND instructions LIKE ? AND i.name IN (?) ]<br />2022-09-14 15:46:38.062 TRACE 16148 --- [nio-7000-exec-2] o.s.jdbc.core.StatementCreatorUtils : Setting SQL statement parameter value: column index 1, parameter value [e26b2a0a-3d2c-442f-8fa1-f26336d5a9d3], value class [java.util.UUID], SQL type unknown<br />2022-09-14 15:46:38.068 TRACE 16148 --- [nio-7000-exec-2] o.s.jdbc.core.StatementCreatorUtils : Setting SQL statement parameter value: column index 2, parameter value [true], value class [java.lang.Boolean], SQL type unknown<br />2022-09-14 15:46:38.068 TRACE 16148 --- [nio-7000-exec-2] o.s.jdbc.core.StatementCreatorUtils : Setting SQL statement parameter value: column index 3, parameter value [3], value class [java.lang.String], SQL type unknown<br />2022-09-14 15:46:38.068 TRACE 16148 --- [nio-7000-exec-2] o.s.jdbc.core.StatementCreatorUtils : Setting SQL statement parameter value: column index 4, parameter value [%Step%], value class [java.lang.String], SQL type unknown<br />2022-09-14 15:46:38.068 TRACE 16148 --- [nio-7000-exec-2] o.s.jdbc.core.StatementCreatorUtils : Setting SQL statement parameter value: column index 5, parameter value [spinach], value class [java.lang.String], SQL type unknown<br />2022-09-14 16:06:37.952 DEBUG 19748 --- [nio-7000-exec-2] o.s.jdbc.core.JdbcTemplate : SQLWarning ignored: SQL state 'HY000', error code '1366', message [Incorrect string value: '\xAC\xED\x00\x05sr...' for column 'user_id' at row 1]<br /></i><br />So that <b>only shows a warning message at DEBUG level</b>, not even as an error, so at first I missed it completely! The query just returned 0 results.<br /><br /> <i> SQLWarning ignored: SQL state 'HY000', error code '1366', message [Incorrect string value: '\xAC\xED\x00\x05sr...' for column 'user_id' at row 1]</i><br /><br />Makes sense though that this custom query has the same issue, since my string-based query of course does not use the converters that I configured earlier.<br />So I had to change the third line to explicitly convert the value to a string, so Spring JDBC will pass it on as a string:<br /><br /><i> parameterValues.add(userId.toString());</i><br /><br />Note: maybe implementing a placeholder interface for each repository like this would also have fixed it for the Spring generated methods/queries like save(), remove() etc, e.g: <i>interface RecipeRepository extends Repository<Recipe, UUID></i>. Did not try that out.<br /><br />Other related links used to get to the above solution:</p><ul style="text-align: left;"><li><a href="https://www.vertica.com/docs/11.1.x/HTML/Content/Authoring/ConnectingToVertica/ClientJDBC/UUIDValues.htm">https://www.vertica.com/docs/11.1.x/HTML/Content/Authoring/ConnectingToVertica/ClientJDBC/UUIDValues.htm</a> <br /></li><li><a href="https://www.anycodings.com/1questions/1947192/how-do-i-use-a-uuid-in-a-jdbc-template">https://www.anycodings.com/1questions/1947192/how-do-i-use-a-uuid-in-a-jdbc-template</a><br /></li><li><a href="https://stackoverflow.com/questions/10957238/incorrect-string-value-when-trying-to-insert-utf-8-into-mysql-via-jdbc">https://stackoverflow.com/questions/10957238/incorrect-string-value-when-trying-to-insert-utf-8-into-mysql-via-jdbc</a></li></ul><p style="text-align: left;"><br /></p>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-76832131492097613032022-10-13T17:04:00.004+01:002022-10-13T17:09:50.038+01:00Migrating Java 17 Spring Boot 2.7.3 application to Kotlin 1.7.20<h1 style="text-align: left;">Introduction</h1><div>This blogpost describes the challenges encountered when migrating a Java 17 Spring Boot 2.7.3 application to Kotlin 1.7.20. </div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhx5TaUrvjVBk9rTzOT5VZ0rYFxKKFdDQYnwaUr8291Y7XKHHZavDYUi0of_U4B4Y5ocLL4mgOUjJZ8F8RMtyvLaB9y02yPlTEIbFRm5AfTuW0c2JKZbcLh555REptT5tb0dYCy7aEM19Af-wmRQAtQzY_go0BgSJYd4zRttyvt_8gWUoJCzW5T4ZLG/s314/java-to-kotlin.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="161" data-original-width="314" height="205" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhx5TaUrvjVBk9rTzOT5VZ0rYFxKKFdDQYnwaUr8291Y7XKHHZavDYUi0of_U4B4Y5ocLL4mgOUjJZ8F8RMtyvLaB9y02yPlTEIbFRm5AfTuW0c2JKZbcLh555REptT5tb0dYCy7aEM19Af-wmRQAtQzY_go0BgSJYd4zRttyvt_8gWUoJCzW5T4ZLG/w400-h205/java-to-kotlin.jpg" width="400" /></a></div><div>Other libraries/tools used in the project:</div><div><pre style="background-color: white; color: #080808; font-family: "JetBrains Mono", monospace; font-size: 9.8pt;"><span style="color: #0033b3;">- </span>Swagger (OpenAPI 3.0.3)<br /><span style="color: #0033b3;">- </span>Spring boot 2.7.3<br /><span style="color: #0033b3;">- </span>Liquibase<br /><span style="color: #0033b3;">- </span>H2 in mem + file based<br /><span style="color: #0033b3;">- </span>JUnit5 with Mockito and Mockito-Kotlin<br /><span style="color: #0033b3;">- </span>MySql 8.0<br /><span style="color: #0033b3;">- </span>Actuator<br /><span style="color: #0033b3;">- </span>Maven 3</pre></div><div>Tip: <i>after</i> migration of most of the .java files manually, I found <a href="https://spring.io/guides/tutorials/spring-boot-kotlin/">this Spring tutorial</a> which helps to avoid having to add 'open' to <i>@Component</i>, <i>@Service</i> etc annotated classes.</div><div><div>It includes the <i><a href="https://kotlinlang.org/docs/reference/compiler-plugins.html#spring-support">kotlin-spring-plugin</a></i> and <i><a href="https://kotlinlang.org/docs/reference/compiler-plugins.html#jpa-support">Kotlin JPA plugin</a></i> to support Kotlin features better, including support for JSR-305 annotations + Spring nullability annotations. Usually you'd also want to include <i><a href="https://github.com/FasterXML/jackson-module-kotlin">jackson-module-kotlin</a></i> for serialization and deserialization of Kotlin classes.</div><div>Note that my application uses generated Java classes from an OpenAPI 3 Swagger yaml file, which are returned in the REST API, so therefore not needed.</div><div><br /></div><div>Total code reduction after migration:</div><div><b>Java: 4718</b></div><div><b>Kotlin: 2860</b></div><div><br /></div><div>So about 44% reduction in code. Not bad.</div></div><div><br /></div><h1 style="text-align: left;">Steps</h1><div><h2 style="text-align: left;">Convert POJOs</h2><div>As first I converted a simple POJO which in my case had <i>@Data</i> and <i>@Builder</i> Lombok annotations.</div><div>Open that POJO Java file. Hit <i>ctrl-alt-shift-k</i>. Or, find it in the IntelliJ 'actions' search panel:</div><div><br /></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXOTE6YNGMnBEyEyA2lSRAyIUaQq2I5vNPNCj0H2kxv1Ei7vWEMxu4Y7A6mK11cd1V9wsoOo-v-uL1YKy_XmZ05G3zSWdi16BbmJ1Wye5rb_wY3C3EZpZdGP9u43SxBhd9NBf88_KbS_RBKuEnJpSOFcMeYi1bY-i2qGGuAxIdjNhqUGeRCvDwMY8L/s988/01-convert-file.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="341" data-original-width="988" height="138" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXOTE6YNGMnBEyEyA2lSRAyIUaQq2I5vNPNCj0H2kxv1Ei7vWEMxu4Y7A6mK11cd1V9wsoOo-v-uL1YKy_XmZ05G3zSWdi16BbmJ1Wye5rb_wY3C3EZpZdGP9u43SxBhd9NBf88_KbS_RBKuEnJpSOFcMeYi1bY-i2qGGuAxIdjNhqUGeRCvDwMY8L/w400-h138/01-convert-file.png" width="400" /></a></div><br /><div><br /></div><div>Then make sure to rebuild to project by enforcing Maven to rebuild if it didn't do that.</div><div>Then I had to redo it on the POJO class.</div><div><br /></div><div>I was a bit surprised what came out:</div><div><br /></div><div><i><span> </span>@Builder</i></div><div><i><span> </span>@Data</i></div><div><i><span> </span>class Ingredient {</i></div><div><i> <span> </span>private val name: String = null</i></div><div><i> <span> </span>private val recipeId: UUID = null</i></div><div><i> <span> </span>private val createdAt: LocalDateTime = null</i></div><div><i> <span> </span>private val updatedAt: LocalDateTime = null</i></div><div><i><span> </span>}</i></div><div><br /></div><div>I would have expected the Lombok annotations to be gone. But on the other hand IntelliJ can't know how to fix them I guess (see below for more on Lombok migration).</div><div>And maybe because of the <i>@Data</i> it made all fields private...</div><div><br /></div><div>I did do expect more like this:</div><div><br /></div><div><i><span> </span>@Builder</i></div><div><i><span> </span>@Data</i></div><div><i><span> </span>class Ingredient(val name: String, recipeId: UUID, createdAt: LocalDateTime, </i></div><div><i><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>updatedAt: LocalDateTime)</i></div><div><br /></div><div>But it seems the converter is not that smart (yet?): <a href="https://stackoverflow.com/questions/70813859/how-to-let-java-to-kotlin-converter-understand-lombok-and-convert-to-data-class">https://stackoverflow.com/questions/70813859/how-to-let-java-to-kotlin-converter-understand-lombok-and-convert-to-data-class</a></div><div>But even when I remove the Lombok annotations, still the fields are created as private fields, not as part of the primary constructor... Maybe because of the other annotations on some of the fields, like <i>@Id</i> and <i>@Version</i>?</div><div><br /></div><div>I manually converted it some more, into this:</div><div><br /></div><div><i><span> </span>data class Ingredient(val name: String, val recipeId: UUID, val createdAt: LocalDateTime, val <span> </span><span> </span></i></div><div><i><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>updatedAt: LocalDateTime)</i></div><div><br /></div><div>Then I converted all uses of the Builder in the Java class to the regular (primary) constructor of the Kotlin data class. E.g:</div><div><br /></div><div><i><span> </span>new Ingredient(ingredient, recipeId, createdAt, createdAt)</i></div><div><br /></div><div>Doing this for all POJOs would be quite some work. And later on, you want to convert those Java instance creations to Kotlin constructors anyway, with named parameters.</div><div>So I didn't do this for all classes, I started to skip this step of replacing builders with constructors.</div><div><br /></div><div>Now first let's try to rebuild the project with '<i>mvn clean install</i>' for example.</div><div><br /></div><div>That gave an error:<i> </i>the newly created Kotlin Ingredient class (symbol) could not be found. Note that IntelliJ was able to find all dependencies just fine.</div><div>The answer to that can be found <a href="https://stackoverflow.com/questions/43980110/using-kotlin-class-in-java-cannot-find-symbol">here</a>. </div><div>I applied the solution where you move your .kt file into its new<i> src/main/kotlin/x/y/z</i> package. Make sure to mark that <i>src/main/kotlin</i> directory as a source directory in IntelliJ.</div><div><br /></div><div>But still an error: </div><div><br /></div><div><i><span> </span>Cannot find symbol (Ingredient)</i>. </div><div><br /></div><div>So that didn't fix it, so applied the accepted solution, so to make sure the compilation order is Kotlin then Java.</div><div>After this change in the <i>pom.xml</i>, IntelliJ couldn't find the Spring Boot application class anymore. '<i>mvn clean install</i>' ran up to the tests, but many failed due to:</div><div><br /></div><div><span> </span><i>java.lang.NoClassDefFoundError: kotlin/reflect/full/KClasses</i> </div><div><br /></div><div>See next section below for how those were fixed.</div><div>I changed also in the Kotlin plugin in the pom.xml the JVM target to: <i><jvmTarget>1.17</jvmTarget></i></div><div>After that, IntelliJ compiled fine again.</div><div><br /></div><div>Then trying to run the application with '<i>mvn spring-boot:run</i>' gave this error:</div><div><br /></div><div><i><span> </span>Compilation failure</i></div><div><i><span> </span>Unknown JVM target version: 1.17</i></div><div><i><span> </span>Supported versions: 1.6, 1.8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18</i></div><div><br /></div><div>So the <i><jvmTarget></i> needs to be 17 (not 1.17) apparently. And then it almost started, but I got the same error as when running the tests:</div><div><br /></div><div><i><span> </span>java.lang.ClassNotFoundException: kotlin.reflect.full.KClasses</i></div><div><br /></div><div>Note also this warning showed up during the build, that needs to be checked and fixed, because it refers to Kotlin JDK8 and we use Java 17:</div><div><br /></div><div><i><span> </span>[INFO] Scanning for projects...</i></div><div><i><span> </span>[WARNING] </i></div><div><i><span> </span>[WARNING] Some problems were encountered while building the effective model for com.project:kotlinrecipes:jar:1.0.0</i></div><div><i><span> </span>[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.jetbrains.kotlin:kotlin-stdlib-jdk8:jar -> duplicate declaration of version ${kotlin.version} @ line 159, column 15</i></div><div><i><span> </span>[WARNING] </i></div><div><i><span> </span>[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.</i></div><div><br /></div><div><br /></div><h2 style="text-align: left;">Fixing the Java tests Part 1</h2><div>Fixing the tests with the error <i>java.lang.ClassNotFoundException: kotlin.reflect.full.KClasses</i></div><div><br /></div><div>Adding this dependency, as also mentioned <a href="https://kotlinlang.org/docs/maven.html#compile-kotlin-only-source-code">here</a>, fixed it:</div><div><br /></div><div><i><span style="white-space: pre;"> </span><dependency></i></div><div><i><span style="white-space: pre;"> </span><groupId>org.jetbrains.kotlin</groupId></i></div><div><i><span style="white-space: pre;"> </span><artifactId>kotlin-reflect</artifactId></i></div><div><i><span style="white-space: pre;"> </span><version>${kotlin.version}</version></i></div><div><i><span style="white-space: pre;"> </span></dependency></i></div><div><br /></div><div>Outstanding question for this stdlib dependency for me was, why is Kotlin stdlib using Java 8?</div><div><br /></div><div><span style="white-space: pre;"> </span><dependency></div><div><span style="white-space: pre;"> </span><groupId>org.jetbrains.kotlin</groupId></div><div><span style="white-space: pre;"> </span><artifactId>kotlin-stdlib-jdk8</artifactId> // Can't this be Java 17?</div><div><span style="white-space: pre;"> </span><version>${kotlin.version}</version></div><div><span style="white-space: pre;"> </span></dependency></div><div><br /></div><div>This seems to be the answer: <a href="https://discuss.kotlinlang.org/t/what-are-plans-for-kotlin-runtimes-after-kotlin-stdlib-jdk8/20786">Kotlin can run with any higher version of the JVM</a> </div><div>And: the Kotlin standard library <i>kotlin-stdlib</i> <b>targets</b> Java 6 and above. There are extended versions of the standard library that add support for some of the features of JDK 7 and JDK 8.</div><div>And when you include kotlin-stdlib-jdk8, <a href="https://mbonnin.medium.com/the-different-kotlin-stdlibs-explained-83d7c6bf293">it will pull kotlin-stdlib-jdk7 and kotlin-stdlib</a>. </div><div>And also note: <a href="https://stackoverflow.com/questions/65731542/why-is-there-no-kotlin-stdlib-jdk11">https://stackoverflow.com/questions/65731542/why-is-there-no-kotlin-stdlib-jdk11</a>. So basically, all fine since Kotlin just doesn't use any API from Java's JDK higher than 8.</div><div><br /></div><div>After this, the application started fine, connected to the MySql Docker instance and the REST endpoints worked all fine.</div><div><br /></div><h2 style="text-align: left;">Migrating the Spring Boot application class</h2><div>The IntelliJ converter worked fine. The <i>@Bean</i> annotation in that class was migrated correctly too. Constants were correctly put in a companion object {} block.</div><div>But when starting the application this message showed up:</div><div><br /></div><div><i> org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: @Configuration class 'RecipesApplication' may not be final. Remove the final modifier to continue.</i></div><div><br /></div><div>Strange: it says the class might not be final, remove the final modifier... Sounds contradicting!</div><div>I made the class 'open' (because the default is public final) and then it worked.</div><div><br /></div><div>Then for the <i>@Bean</i> I got:</div><div><br /></div><div><i> org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: @Bean method 'encoder' must not be private or final; change the method's modifiers to continue</i></div><div><i><br /></i></div><div>I made the method 'encoder' also 'open' and then it worked.</div><div><br /></div><div>If you have other issues, <a href="https://www.luminis.eu/blog/development-en/migrating-a-spring-boot-service-from-java-11-to-kotlin/">check this post</a> for more tips. </div><div><br /></div><h2 style="text-align: left;">Migrating a Spring Boot @RestController</h2><div>I applied the IntelliJ converter. Its result was pretty good. Inheritance from the OpenAPI 3 generated Java code was correctly applied. </div><div>I had to add the Kotlin logging to replace the @Slf4j static logger, see <a href="https://www.baeldung.com/kotlin/kotlin-logging-library">here</a>.</div><div>Note the default static 'log' field generated by @Slf4j is after applying that named <i>private val logger = KotlinLogging.logger {}</i>. But you can name it as you want of course.</div><div><br /></div><div>When running the application, again the controller also had to be made open, to allow it to be subclassed (as can be seen in the tip in the introduction section, Spring and some other frameworks require classes to be open (extendable)).</div><div>After this change, the controller worked fine.</div><div><br /></div><div>I also modified the generated code a bit by adding a @NotNull annotation (for something which had already a '?' so was incorrect anyway I realized afterwards), but then at runtime only I sadly got this error:</div><div><br /></div><div><i> javax.validation.ConstraintDeclarationException: HV000151: A method overriding another method must not redefine the parameter constraint configuration, but method UserController#loginUser(JwtRequest) redefines the configuration of UsersApi#loginUser(JwtRequest).</i></div><div><br /></div><div>Removing the incorrectly added @NotNull fixed that problem; and it is unnecessary in combination with the '?' too anyway.</div><div><br /></div><h2 style="text-align: left;">Migrating @Component service</h2><div>Got this error after adding the logger:</div><div><br /></div><div><i> java.lang.NullPointerException: Cannot invoke "mu.KLogger.info(String)" because "this.logger" is null</i></div><div><br /></div><div>Strange, the <i>@RestController</i> did not have that issue. Though that one is overriding a (generated OpenAPI 3) class.</div><div>This <a href="https://stackoverflow.com/questions/37431817/null-pointer-exception-in-spring-proxy-class-and-kotlin">question</a> triggered me, so I added the 'open' keyword to the methods in the <i>@Component</i> class, to make it non-final so Spring has access to it with its proxies too.</div><div><br /></div><div>That worked.</div><div><br /></div><h2 style="text-align: left;">Converting Spring @Repository</h2><div>For interfaces, the question was: a <i>findByUsername(username)</i> can return null when it doesn't find the user. How to best define that in Kotlin? Allow null to be returned (but at least the caller then has to handle the null possibility)? Or use Optional in that case? Or is there another better solution?</div><div>No clear best answer to me, e.g: <i>https://discuss.kotlinlang.org/t/how-to-deal-with-database-null-return-according-kotlin-null-safety-feature/2546</i></div><div>I went for having the repo return '?'. But another repo already was using Optional, so left that there too. Will have to decide on consistency here...</div><div>See this post on <a href="https://elizarov.medium.com/null-is-your-friend-not-a-mistake-b63ff1751dd5">how null can be seen positively</a> and also the String.toIntOrNull() function extension built into Kotlin! :) Based on that, I went for allowing repository functions to return 0.</div><div> </div><h2 style="text-align: left;">Converting @MappedCollection(idColumn = "RECIPE_ID")</h2><div>Only had to make sure using the arrayOf() and using a MutableSet because you usually want to add elements to the child (collection):</div><div><br /></div><div><div><i> @OneToMany(cascade = arrayOf(CascadeType.ALL), orphanRemoval = true, </i></div><div><i><span> </span><span> </span><span> </span><span> </span><span> </span>targetEntity = Ingredient::class)</i></div><div><i> @JoinColumn(name = "recipe_id")</i></div><div><i> var ingredients: MutableSet<Ingredient> = HashSet<Ingredient>(),</i></div></div><div><br /></div><h2 style="text-align: left;">Use @NotNull or not</h2><div>Is it useful to have the <i>@NotNull </i>annotation, while it is only done at runtime? And doesn't Kotlin also throw an exception when you pass in a null value to a parameter that is already non-nullable by default (i.e it doesn't have the '?' appended to its type)?</div><div>Seems like double because Kotlin <a href="https://stackoverflow.com/questions/41993706/is-notnull-needed-on-kotlin ">inserts the @NotNull into the code</a>? </div><div>Maybe you'd put it in when Java code is calling your Kotlin code, to make it more explicit - and the Java code can then validate on it?</div><div><br /></div><h2 style="text-align: left;">Converting @Configuration class</h2><div>A class annotated with that has also to be open: </div><div><br /></div><div><i><span> </span>@Configuration</i></div><div><i><span> </span>internal open class JdbcConfig : AbstractJdbcConfiguration() {...}</i></div><div><br /></div><div>I noticed the Kotlin converter from IntelliJ didn't like always comments at the end of a line of Java code. Sometimes the '()' of a method call were then put on the wrong line.</div><div><br /></div><h2 style="text-align: left;">Fixing the Java tests Part 2</h2><div>While still as Java code, some failed with this:</div><div><br /></div><div><div style="font-style: italic;"><span> </span>org.mockito.exceptions.base.MockitoException: </div></div><div><i><span> </span>Cannot mock/spy class com.project.kotlinrecipes.user.infra.JwtUserDetailsServiceImpl</i></div><div><i><span> </span>Mockito cannot mock/spy because :</i></div><div><i><span> </span> - final class</i></div><div><br /></div><div>So that was easy, I made those classes 'open'.</div><div><br /></div><div>I also had to make methods 'open' used in <i>Mockito.when()</i> matchers, because otherwise it complains: </div><div><br /></div><div><span> </span>org.mockito.exceptions.misusing.InvalidUseOfMatchersException: </div><div><i><span> </span>Invalid use of argument matchers!</i></div><div><i><span> </span>0 matchers expected, 1 recorded:</i></div><div><br /></div><div>But also verify() started to fail:</div><div><br /></div><div><i> verify(jwtTokenUtil, Mockito.times(0)).validateToken(isA(String.class), isA(UserDetails.class));</i></div><div><i><span> </span>java.lang.NullPointerException: Parameter specified as non-null is null: method</i></div><div><i><span> </span>com.project.kotlinrecipes.infra.security.JwtTokenUtil.validateToken, parameter userDetails</i></div><div><br /></div><div>That also meant added the 'open' keyword to that method.</div><div><br /></div><h2 style="text-align: left;">Converting JUnit 5 tests with Mockito to Kotlin</h2><div>IntelliJ's auto-converter works pretty good. Except that it converts</div><div><br /></div><div><i> private JdbcIngredientRowMapper jdbcIngredientRowMapper;</i></div><div><i><br /></i></div><div><i> @BeforeEach</i></div><div><i> public void setUp() {</i></div><div><i> jdbcIngredientRowMapper = new JdbcIngredientRowMapper();</i></div><div><i> }</i></div><div><br /></div><div>to:</div><div><br /></div><div><i> private var jdbcIngredientRowMapper: JdbcIngredientRowMapper? = null</i></div><div><i><br /></i></div><div><i> @BeforeEach</i></div><div><i> fun setUp() {</i></div><div><i> jdbcIngredientRowMapper = JdbcIngredientRowMapper()</i></div><div><i> }</i></div><div><br /></div><div>But it can be made nullsafe by changing it to:</div><div><br /></div><div><i> private lateinit var jdbcIngredientRowMapper: JdbcIngredientRowMapper</i></div><div><i> </i></div><div><i> @BeforeEach</i></div><div><i> fun setUp() {</i></div><div><i> jdbcIngredientRowMapper = JdbcIngredientRowMapper()</i></div><div><i> }</i></div><div><br /></div><div>I also introduced the `test description` notation, e.g:</div><div><br /></div><div><i> @Test</i></div><div><i> fun `Should map row`() {}</i></div><div><br /></div><div>I use in some tests:</div><div><br /></div><div><i> @ParameterizedTest</i></div><div><i> @MethodSource("filterNullPermutations")</i></div><div><br /></div><div>That <i>filterNullPermutations </i>has to be a static method. IntelliJ made it a companion object with that method 'private'. I added <i>@JvmStatic</i> to make it accessible for the <i>@MethodSource</i>.</div><div><br /></div><div>I had to add <a href="https://github.com/mockito/mockito-kotlin to better handle ">mockito-kotlin library</a> for better interoperability for this case: </div><div><i>ArgumentMatchers.isA(class)</i> for Kotlin methods that don't allow null values be put in (which <i>isA()</i> and <i>any() </i>for example can return).</div><div>And by adding this library I could now also use '<i>whenever</i>' instead of '<i>`when`'</i>.</div><div><br /></div><div><i>isA(MyClass.class)</i> in Java had to be converted to: </div><div><br /></div><div> <i>whenever(jwtTokenUtil.generateToken(isA<UserDetails>))).thenReturn(BEARER_TOKEN_VALUE)</i></div><div><i><br /></i></div><div>Or even shorter:</div><div><br /></div><div> <i>whenever(jwtTokenUtil.generateToken(isA())).thenReturn(BEARER_TOKEN_VALUE)</i></div><div><i><br /></i></div><div>using the mockito-kotlin library, which creates an instance (instead of the regular mockito which can return null). See <a href="https://github.com/mockito/mockito-kotlin/wiki/Mocking-and-verifying">here</a> for more explanation. </div><div><br /></div><div>Replaced all <i>mock()</i> with Kotlin style: <i>val mockBookService : BookService = mock()</i></div><div><br /></div><div>The generated code did not work for this <i>TestRestTemplate.exchange()</i> call in Java:</div><div><br /></div><div><i> // Set up find parameters</i></div><div><i> Map<String, String> uriVariables = new HashMap<>();</i></div><div><i> uriVariables.put("vegetarian", "true");</i></div><div><i> uriVariables.put("numberOfServings", "3");</i></div><div><i> uriVariables.put("includedIngredients", "onion");</i></div><div><i> uriVariables.put("excludedIngredients", "fish");</i></div><div><i> uriVariables.put("instructions", "First");</i></div><div><i><br /></i></div><div><i> HttpHeaders headers = new HttpHeaders();</i></div><div><i> headers.set(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE);</i></div><div><i> HttpEntity<?> entity = new HttpEntity<>(headers);</i></div><div><i><br /></i></div><div><i> // When</i></div><div><i> ResponseEntity<List<RecipeResponse>> foundRecipesEntity = testRestTemplate.exchange("/recipes/findByFilter?vegetarian={vegetarian}&numberOfServings={numberOfServings}&includedIngredients={includedIngredients}&excludedIngredients={excludedIngredients}&instructions={instructions}",</i></div><div><i> HttpMethod.GET, entity, new ParameterizedTypeReference<>() {}, uriVariables);</i></div><div><br /></div><div><br /></div><div>Became after IntelliJs converter applied:</div><div><br /></div><div><i> // Set up find parameters</i></div><div><i> val uriVariables: MutableMap<String, String?> = HashMap()</i></div><div><i> uriVariables["vegetarian"] = "true"</i></div><div><i> uriVariables["numberOfServings"] = "3"</i></div><div><i> uriVariables["includedIngredients"] = "onion"</i></div><div><i> uriVariables["excludedIngredients"] = "fish"</i></div><div><i> uriVariables["instructions"] = "First"</i></div><div><i> val headers = HttpHeaders()</i></div><div><i> headers[HttpHeaders.ACCEPT] = MediaType.APPLICATION_JSON_VALUE</i></div><div><i> val entity: HttpEntity<*> = HttpEntity<Any>(headers)</i></div><div><i><br /></i></div><div><i> // When</i></div><div><i> val foundRecipesEntity: ResponseEntity<List<RecipeResponse>> =</i></div><div><i> testRestTemplate.exchange<List<RecipeResponse>>("/recipes/findByFilter?vegetarian={vegetarian}&numberOfServings={numberOfServings}&includedIngredients={includedIngredients}&excludedIngredients={excludedIngredients}&instructions={instructions}",</i></div><div><i> HttpMethod.GET, entity, object : ParameterizedTypeReference<List<RecipeResponse?>?>() {}, uriVariables</i></div><div><i> )</i></div><div><i><br /></i></div><div>But .<i>exchange()</i> was red underlined, no matching method to invoke found. I had to change it into this:</div><div><br /></div><div><i> // Set up find parameters</i></div><div><i> val uriVariables: MutableMap<String, String> = HashMap()</i></div><div><i> uriVariables["vegetarian"] = "true"</i></div><div><i> uriVariables["numberOfServings"] = "3"</i></div><div><i> uriVariables["includedIngredients"] = "onion"</i></div><div><i> uriVariables["excludedIngredients"] = "fish"</i></div><div><i> uriVariables["instructions"] = "First"</i></div><div><i> val headers = HttpHeaders()</i></div><div><i> headers[HttpHeaders.ACCEPT] = MediaType.APPLICATION_JSON_VALUE</i></div><div><i><br /></i></div><div><i> // When</i></div><div><i> val foundRecipesEntity: ResponseEntity<List<RecipeResponse>>? =</i></div><div><i> testRestTemplate.exchange(</i></div><div><i> "/recipes/findByFilter?vegetarian={vegetarian}&numberOfServings={numberOfServings}&includedIngredients={includedIngredients}&excludedIngredients={excludedIngredients}&instructions={instructions}",</i></div><div><i> HttpMethod.GET, HttpEntity("parameters", headers),</i></div><div><i> typeReference<List<RecipeResponse>>(), uriVariables</i></div><div><i> )</i></div><div><br /></div><div>With the <i>typeReference </i>method added (you can also do it inline BTW):</div><div><br /></div><div><i> private inline fun <reified T> typeReference() = object : ParameterizedTypeReference<T>() {}</i></div><div><br /></div><div>And after some more cleaning up this worked too (can you spot the differences with the generated Kotlin from IntelliJ?):</div><div><br /></div><div><i> // Set up find parameters</i></div><div><i> val uriVariables: MutableMap<String, String> = HashMap()</i></div><div><i> uriVariables["vegetarian"] = "true"</i></div><div><i> uriVariables["numberOfServings"] = "3"</i></div><div><i> uriVariables["includedIngredients"] = "onion"</i></div><div><i> uriVariables["excludedIngredients"] = "fish"</i></div><div><i> uriVariables["instructions"] = "First"</i></div><div><i> val headers = HttpHeaders()</i></div><div><i> headers[HttpHeaders.ACCEPT] = MediaType.APPLICATION_JSON_VALUE</i></div><div><i> val entity: HttpEntity<*> = HttpEntity<Any>(headers)</i></div><div><i><br /></i></div><div><i> // When</i></div><div><i> val foundRecipesEntity: ResponseEntity<List<RecipeResponse>> =</i></div><div><i> testRestTemplate.exchange(</i></div><div><i> "/recipes/findByFilter?vegetarian={vegetarian}&numberOfServings={numberOfServings}&includedIngredients={includedIngredients}&excludedIngredients={excludedIngredients}&instructions={instructions}",</i></div><div><i> HttpMethod.GET, entity,</i></div><div><i> typeReference<List<RecipeResponse>>(), uriVariables</i></div><div><i> )</i></div><div><br /></div><div><br /></div><div>Note that <a href="https://mockk.io/">MockK</a> can be the next improvements to the Kotlin code, to allow more Kotlin-style of notation.</div><div><br /></div><h2>Method documentation generation</h2><div>I noticed my IntelliJ 2022.2.1 does not generate <i>@param, @return etc</i> documentation when typing <i>/**</i> above a (private) function definition.</div><div><br /></div><h2 style="text-align: left;">Miscellaneous</h2><div>I still have a few references to Java classes in the code, like this one:</div><div><br /></div><div><i> httpSecurity.addFilterBefore(jwtRequestFilter, UsernamePasswordAuthenticationFilter::class.java)</i></div><div><br /></div><div>Could not find a way to avoid having to reference a Java class in Kotlin this directly.</div><div><br /></div><div>And at the end of the process I removed all Lombok annotations and its dependency in the pom.xml.</div><h2 style="text-align: left;">Bonus tip</h2><div>Spring's Kotlin extensions overview can be found <a href="https://docs.spring.io/spring-framework/docs/current/kdoc-api/">here</a>.</div><div><br /></div><div><br /></div></div>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-35496138870266850972022-09-27T15:46:00.004+01:002022-09-27T15:46:57.546+01:00Kotlin for Java Developers course summary<h2 style="text-align: left;">Introduction</h2><div>This is my summary of relevant lessons learned during a Kotlin for Java developers course.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhANbGDIh2e8jwNfqQDtFpT3MKY3i48lGlk18YQrHoMYCLFWpIH99LffkWpHQnAZFZf2zCYQdjxr1OC59PtZQ20nnMHvPOL95uK-PC66NcCn6-6irlaIxKfVZpXLSIj03pjYoIh5KE127gjmhPT4ZKa0-a_ia2mhPCjQgFa2SPDT7tAtjF0Eh58J06E/s1200/kotlin-logo.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1200" data-original-width="1200" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhANbGDIh2e8jwNfqQDtFpT3MKY3i48lGlk18YQrHoMYCLFWpIH99LffkWpHQnAZFZf2zCYQdjxr1OC59PtZQ20nnMHvPOL95uK-PC66NcCn6-6irlaIxKfVZpXLSIj03pjYoIh5KE127gjmhPT4ZKa0-a_ia2mhPCjQgFa2SPDT7tAtjF0Eh58J06E/s320/kotlin-logo.png" width="320" /></a></div><div><br /></div><h2 style="text-align: left;">Summary</h2><h3 style="text-align: left;">General</h3><div><div>- When running standalone, needs Kotlin runtime</div><div>- Toplevel functions: not explicitly defined in a class, e.g <i>fun main()</i> method</div><div>- Useful other info: <a href="https://kotlinlang.org">https://kotlinlang.org</a></div><div>-<i> val number: Int and val number = 25</i> So has type inference. But if needed:<i> number: Short = 25</i></div><div>- Recommended good practice: delare variables with 'val' where possible</div><div>- Properties of a 'val' class instance you can modify</div><div>- Via class "constructor" you can also specify whether to set properties is allowed. E.g class <i>Employee(var name: String, val id: Int)</i>, then you can do:<i> employee.name = "abc"</i>, but not <i>employee.id = 5</i>. Note that construction of instance needs both parameters anyway.</div><div>- Collections: you can use <i>[i]</i> notation to get to an element. Or for Maps: use the key</div><div>- No difference between checked and unchecked exceptions, no need to specify them in method names. You can't even add a 'throws'.</div><div>- Kotlin has no static keyword syntactically, e.g see the '<i>fun main()</i>' declaration. But they do exist.</div><div>- No 'new' keyword</div><div>- The '<i>==</i>' operator checks for structural equality, so it works the same as <i>.equals()</i>! So .equals() you don't need to use. How to check for referential equality use: <i>'===</i>'</div><div> <span style="white-space: pre;"> </span>Annoying, in Javascript '===' means more equality, in Kotlin less (only referential equality).</div><div>- Bitoperators like '|' you spell out like: '<i>or</i>'</div><div>- Use '<i>is</i>' instead of 'instanceof'</div><div>- Smart casting: use the '<i>as</i>' keyword, e.g: <i>val newValue = something as Employee</i>. But if you did an 'is' check before, you don't need the cast anymore.</div><div>- Raw strings can be triple quoted, so you don't need to escape special characters, e.g: <i>val path = """c:\somedir\somedir2"""</i></div><div>- <i>trimMargin</i>() for indentation in code of raw strings that cover multiple lines</div><div>- Everything is a class, no native types like long or int.</div><div>- Kotlin does not automatically widen numbers. E.g: in Java: int a = 55;long b = a; is fine. But not in Kotlin. So all standard datatypes have <i>toLong</i>() and similar conversion methods</div><div>- Double is default just like in Java when declaring variable w/o specifying type, e.g: <i>val myDouble = 65.994</i>. Int also, e.g:<i> val myInt = 11</i></div><div>- <i>Any </i>class: similar to root class of Java class hierarchy</div><div>- <i>Unit </i>class: similar to void in Java, but different: it is a singleton unit instance</div><div>- <i>Nothing </i>class: subclass of any class. E.g when you have a function with an infinite loop, then return 'Nothing'.</div><div>- The RPEL can be used as "scratchbook" of executable code, which knows of your existing classes</div><h3 style="text-align: left;">Arrays</h3><div>- Initialize array using a lambda, even after declaration:<i> val evenNumbers = Array(16) {i -> i * 2}</i> So here i is the index of the array</div><div>- Mixed array: <i>val mixedArray = arrayOf("Hello", 22, BigDecimal(0.5), 'a')</i>. It is then an array of Array<Any>. Or just call .<i>toIntArray</i>() on the Kotlin Int array.</div><div>- To call a Java method with int[] as parameters, you can't pass in the Kotlin collection Array, so you have to use:<i> intArrayOf(1, 2, 3)</i> instead of arrayOf(1,2,3). Is also a bit of performance boost since directly you create a primitive type array for the JVM (of type int).</div><div>- You can't do <i>var someArray = Array<Int>(5)</i>. </div><div>- You can do for any type: <i>val myData = 'arrayOf(car1, car2, car3)</i>', where the 3 cars are of type Car.</div><div>- Convert int[] created via intArrayOf() to a Kotlin typed array: <i>val typedArray = intArrayOf(1,2,3).toTypedArray() </i></div><h3 style="text-align: left;">Null references</h3><div>- For types that can be nullable, you have to tell it can be null by appending a '?', e.g: '<i>val str: String? = null</i>'</div><div>- If you check for variable being null, Kotlin compiler from then on knows it can't be null, so all methods are available again (similar to smart-casting)</div><div>- '<i>str?.toUpperCase()</i>' performs behind the scenes a <i>'if (str != null) str.toUpperCase() else null</i>'. The '<i>?</i>' is called the "safe operator".</div><div>- So nested also possible, then it stops at the first one that is null. E.g: '<i>home?.address?.country?.getCountryCode()</i>'</div><div>- To set a default value in case it evaluates to null, use the elvis operator: '<i>?:</i>'. E.g: '<i>val str1: String? = null; val str2 = str1 ?: "this is the default value"</i>'</div><div>- Safe cast operator allows any cast, and if it can't do it, it sets the variable to null. E.g: '<i>val someArray: Any = arrayOf(1,2,3); val str = someArray as? String</i>' results in 'str' having value null. So '<i>?</i>' really means: this (variable) *might* from now on evaluate to null.</div><div>- If you are sure an expression can't evaluate to null, you can tell the compiler via the non-null assertion: <i>str!!.toUpperCase()</i>. After this point you don't have to add the '?' anymore for null safety checks (because you told the compiler it 100% can't be null). Use this e.g when you *do* want a nullpointer exception to be thrown.</div><div>- The '<i>let</i>' function: '<i>str?.let {println(it)}</i>', which says: if str isn't null, then let this function-call go ahead via a lambda expression.</div><div>- Note that the <i>==</i> operator is safe for nulls (note underwater it uses .equals()), e.g: '<i>val str1 : String? = null; val str2 = "abc"; str1 == str2</i>'</div><div>- To have an array of null values, you have to use '<i>arrayOfNulls</i>()' method, e.g: '<i>arrayOfNulls<Int?>(5)</i>' creates one of size 5. Or: '<i>arrayOfNulls<Int>(5)</i>'</div><div>- Use <i>!!</i> when you are sure the variable can never ever be null</div><h3 style="text-align: left;">Access modifiers</h3><div>- 4 types for toplevel items:</div><div><span style="white-space: pre;"> </span><b>Default </b>= public final (instead of package level in Java). Note classname doesn't have to match filename in kotlin. One file can have multiple public classes.</div><div><span style="white-space: pre;"> </span><b>Private </b>= Everything in the same file can access it. <span style="white-space: pre;"> </span>(Java doesn't allow a private class). When specified for a constuctor parameter, no getters/setters are generated! So really only the class itself can have access to that property, no-one else.</div><div><span style="white-space: pre;"> </span><b>Protected </b>= can't be used</div><div><span style="white-space: pre;"> </span><b>Internal </b>= is inside the module (see below) (no Java equivalent)</div><div>- Kotlin has the 'module' principle: group of files that are compiled together.</div><div>- For class members:</div><div><span style="white-space: pre;"> </span>Public, private, protected mean the same as in Java.</div><div><span style="white-space: pre;"> </span>Internal = visible in same module, usually outside the file</div><div>- In Kotlin, classes can't see private members belonging to inner classes</div><div>- Compile time:</div><div><span style="white-space: pre;"> </span>private in Kotlin is compiled to package private</div><div><span style="white-space: pre;"> </span>internal in Kotlin is compiled to public</div><div><span style="white-space: pre;"> </span>That means sometimes Java code can access Kotlin code that from w/in Kotlin (compiler) can't be accessed</div><div><span style="white-space: pre;"> </span>Kotlin generates really long strange names for those type of methods/classes defined as internal, so you are kindof warned when using them from Java.</div><div>- Create constructor verbose, Java-like: </div><div><span style="white-space: pre;"> </span><i>class Employee constructor(firstName: String) {</i></div><div><i> val firstname: String</i></div><div><i><br /></i></div><div><i><span style="white-space: pre;"> </span> init {</i></div><div><i><span style="white-space: pre;"> </span>this.firstName = firstName</i></div><div><i><span style="white-space: pre;"> </span> }</i></div><div><i> }</i></div><div><span style="white-space: pre;"> </span>Note the 'constructor' keyword as primary constructor (declared outside the curly braces). </div><div><span style="white-space: pre;"> </span>And the '<i>init</i>' block. The init block runs after all fields have been initialized. </div><div>- Create less verbose:</div><div><span style="white-space: pre;"> </span><i>class Employee constructor(firstName: String) {</i></div><div><i> val firstName: String = firstName</i></div><div><i> }</i></div><div>- Even less (note the 'val' keyword added; can also be 'var'):</div><div><span style="white-space: pre;"> </span><i>class Employee constructor(val firstName: String) {</i></div><div><i> }</i></div><div>- Even less:</div><div><span style="white-space: pre;"> </span><i>class Employee (val firstName: String) {</i></div><div><i> }</i></div><div>- But if you want to change visibility of the constructor, you will have to add the 'constructor' keyword: <i>class Employee protected constructor(...)</i></div><div>- For more constructors, just add 'constructor's in the code. But you also have to invoke the primary constructor if present. For that do:</div><div><span style="white-space: pre;"> </span><i>constructor(firstname: String, lastName: String): this(firstName) {</i></div><div><i><span style="white-space: pre;"> </span>xxx</i></div><div><i><span style="white-space: pre;"> </span>}</i></div><div><span style="white-space: pre;"> </span>Note no 'val' prefix anymore, since the 2nd constructor does not declare variables! Just declare the 2nd parameter as field in the class.</div><div>- 'val' and 'var' in the primary constructor generate the matching properties. If not specified, it is just a parameter in that primary constructor.</div><div>- You don't *have* to specify a primary constructor</div><div>- Kotlin classes only have properties, no fields (what's the difference? Note there's use of something named the "backing field", see below). Default for properties is also 'public'.</div><div>- Properties defined as 'var' will have setter generated too. </div><div>- '<i>private var</i>' properties: can't be accessed at all. This is because the (generated) getters + setters must have the same visibility as the property.</div><div>- If you want your own getters+setters, you can't declare that property in the primary constructor; it has to be declared in the class.</div><div>- Backing field (strange name to be just thrown in in the course):</div><div><span style="white-space: pre;"> </span><i>class Employee(val firstName: String, lastName: String = "") {</i></div><div><i> <span style="white-space: pre;"> </span>var lastName = lastName</i></div><div><i> get() {</i></div><div><i><span style="white-space: pre;"> </span>return field</i></div><div><i> }</i></div><div><i> }</i></div><div> Note the <i>get()</i> has to be immediately after the property. And also the special keyword 'field'.</div><div>- Similar for the <i>set()</i> method. But note the caller can do: <i>employee.lastName = "abc"</i>. No 'setLastName()' call needed.</div><div>- You have toplevel constants too just like toplevel functions. So you don't need to declare a constants file for example. <i>val MY_CONSTANT: String = "Text" . </i>Note also the 'const' keyword exists.</div><div>- Data class: '<i>data class Bike(val color: String, val model: String) {}</i>'. You get for free: toString(), equals()+hashCode(), copy() function.</div><div> Requirements: primary constructor has to have at least 1 parameter marked val or var (unlike lastName in previous example). Also: can't be abstract, sealed or inner classes.</div><div>- Parameters can be passed in in any order because you can specify the name of the parameter during the call. These are called "named arguments".</div><div>- Internal access modifier: private class + internal functions is meaningless, since the private class already restricts it to the file.</div><div>- <i>listOf</i>() returns an immutable list of items</div><h3 style="text-align: left;">Functions</h3><div>- Simplified function definition: <i>fun myFunction(param1: Int, param2: Int) = "Multiplied = ${param1 * param2}</i>". So the function returns what is after the equals sign. Defining it this way (so not with a {} block body) is called an expression body.</div><div>- Returning nothing means specifying Unit (or leaving it out since it is the default)</div><div>- Function parameters must have a type specified (even though compiler could derive the type!)</div><div>- Variable number of arguments: '<i>vararg</i>' keyword, e.g: '<i>fun addUp(vararg amounts: Int)</i>'</div><div>- Spread operator: unpacks the array to pass each element separately (used when having to pass to a vararg parameter): <i>addUp(*arrayOfInts)</i>. Yes a '*'.</div><div> In Java you could pass in an array of objects to a method with a varargs parameter, but not in Kotlin.</div><div> Another example: '<i>val newArray = arrayOf(*array1, *array2, anotherElement)</i>'. If you don't do this, you'll have 2 arrays + one element in 'newArray'!</div><div>- Extension functions lets you add functions to classes. Just add the class + a dot (the so-called "receiver type") to a function, e.g: '<i>fun String.replaceEWithA() {}</i>'. You access the string with 'this' keyword in that method.</div><div> Note it knows since you are putting the extension on String, no need for a String parameter in the new function. </div><div> Any public field in the class you add the extension function to can be accessed.</div><div> Feels a bit like a "hack" adding random functions to existing classes. People might also get lazy, not coming up with a new class, but just adding it to an existing one because it matched "good enough". Also for a newcomer on the project it is not possible to easily distinct that a method is an</div><div> extension function.</div><div>- Function parameters are of type 'val'.</div><div>- Inline functions: you can tell the compiler to inline by prefixing function definition with the '<i>inline</i>' keyword. Usually works best for functions with lambda parameters</div><h3 style="text-align: left;">Inheritance</h3><div>- Extend (subclass) a class: '<i>class RaceBike(): Bike() {}</i>'. Class Bike has to have the '<i>open</i>' keyword as prefix because default is final (not extendable).</div><div><span style="white-space: pre;"> </span>So: '<i>open class Bike() {}</i>'. And if you don't want to specify an empty primary constructor: add '<i>constructor(): super()</i>' to the subclass RaceBike.</div><div>- For an abstract class you can remove the 'open' keyword, because 'abstract' implies already that it will be extendable.</div><div>- Overriding a method in the subclass requires 'override' in the subclass method and 'open' in the superclass method.</div><div>- An 'abstract fun' also needs the 'override' keyword in the subclass. No need for the 'open' keyword here either, it is automatically. Same for the 'override' keyword!</div><div>- Subclass (of course) doesn't have to have the same number of parameters for the primary constructor</div><div>- Invoking "super.primary constructor" only makes sense/possible if you don't have any primary constructor defined for the parent</div><div>- Usually you'll provide a primary constructor and only add secondary ones when absolutely necessary</div><div>- In a subclass when creating a secondary constructor, call the constructor (IntelliJ might report it is missing superclass call or similar) like this: </div><div><span style="white-space: pre;"> </span>'<i>constructor(prop1, prop2,..., newParam: Int): this(prop1, prop2,...)</i>'</div><div>- Data classes can't be extended (nor be abstract class or inner class)</div><div>- Interfaces can be extended with the same syntax: '<i>interface MyChildInterface: MyParentInterface {}</i>'. Classes can implement multiple interfaces like Java.</div><div>- Interfaces can have properties. Which the implementing class has to define too with an override: '<i>override val number: Int = 30</i>'.</div><div>- If you want to set a property's value in an interface, giving it a concrete implementation, you have to define a '<i>get</i>()' directly below it. Not sure if you'd ever want that in an interface...</div><div>- Properties in interfaces don't have the backing field 'field' identifier (which classes do have in the custom get() and set()).</div><div>- Making a parameter optional (with default value) in a function in a superclass, doesn't make it an optional parameter in its subclasses.</div><h3 style="text-align: left;">object keyword</h3><div>- '<i>object</i>' keyword is used for: singletons, companion objects, object expressions.</div><div>- For singleton there's always only one instance created, very first time you use this class. So I guess not static classes in Java which are created I think at application startup.</div><div> You can access the functions of the singleton, quite normally: <i>MySingleton.someMethod()</i>. Note the singleton class is not constructed first.</div><div>- If you want to add e.g functions to a class that are statically accessible (so w/o having to create the class), wrap it with '<i>companion object {}</i>'.</div><div> The non-private methods in that block are from then on accessible, using special keyword: '<i>MyClass.Companion.someMethod()</i>'. But then the class is not a singleton but a regular class with a companion object block in it. You can also give the companion a name. Kotlin also is smart enough you can ommit Companion. This is really much more verbose than the Java 'static' keyword!</div><div>- Companion objects can also be used to call private constructors. You can use them to implement factory patterns.</div><div><span style="white-space: pre;"> </span>Of course you have to make the primary constructor (if exists) private too then to prevent instantiation outside the factory, e.g: </div><div><span style="white-space: pre;"> </span><i>class SomeClass private constructor(val input: String) {</i></div><div><i><span style="white-space: pre;"> </span>companion object: {</i></div><div><i><span style="white-space: pre;"> </span>fun createBasicSomeClass(val input: String) = SomeClass(input)</i></div><div><i><span style="white-space: pre;"> </span>}</i></div><div><i><span style="white-space: pre;"> </span>}</i></div><div>- Object expressions are where in Java you'd use anonymous classes, e.g implementing an interface that needs to be passed as parameter to a function.</div><div><span style="white-space: pre;"> </span>Also called anonymous instances.</div><div><span style="white-space: pre;"> </span>E.g: '<i>functionToCall(object: SomeInterface {</i></div><div><i><span style="white-space: pre;"> </span>override fun functionToImplement(text: String) = "implemented text $text"</i></div><div><i><span style="white-space: pre;"> </span>}</i></div><div><i><span style="white-space: pre;"> </span> '</i></div><div><span style="white-space: pre;"> </span>Note that the object created is not a singleton (confusing, since you use the word 'object' which is also used to create a Singleton)</div><div><span style="white-space: pre;"> </span>Also, the object expression code can access local variables in scope (they don't have to be 'val' (or "final" in Java terms), can be 'var' too). So also change that local variable's value!</div><h3 style="text-align: left;">Enums </h3><div>- Use '<i>enum class'</i> keyword. </div><div>- How annoying: if you add a '<i>fun</i>' to an enum, you have to add a '<i>;</i>' after the last enum value!! Odd that compiler can't figure it out w/o the ';'...</div><div>- Example invoking a function in enum for a given enum value: '<i>DepartmentEnum.ACCOUNTING.getInfo()</i>'</div><h3 style="text-align: left;">Importing</h3><div>- Recommended to (of course) use the same package structure as underlying directory structure. (though it is not a requirement from Kotlin)</div><div>- Toplevel functions (in files, not in a class) can be imported like a class</div><div>- What if you want to use a function from another module? Then (in IntelliJ) you have to add the dependency to that module first. Will be a regular Maven dependency in Maven I expect.</div><div>- Type aliases and extension functions you import the same (they are all top-level anyway)</div><div>- You can rename imports too using the '<i>as</i>' keyword. Useful when 2 third party libraries use the same name for some class.</div><h3 style="text-align: left;">For loop</h3><div>- while and do-while is same in Kotlin</div><div>- For loop syntax like in Java doesn't exist in Kotlin. </div><div>- Range: values are inclusive. E.g '<i>val range = 1..5</i>'. But also: <i>val charRange = 'a' .. 'z'</i>. Because they are comparable. <i>val stringRange = "ABD".."XYZ"</i></div><div>- '<i>in</i>' keyword for: '<i>println(3 in range)</i>' is true. But also '<i>"CCCCC" in stringRange</i>' is true! Because the first 'X' is already greater than the first 'C'.</div><div> '<i>"ZZZZZ" in stringRange</i>' is false because the first 'Z' is greater than 'X'</div><div>- '<i>val rangeBackwards = 5..1</i>' is usually not what you want because '<i>5 in rangeBackwards</i>' is false! Because 5 >= 5 but 5 <= 1. So you have to do: '<i>5.downTo(1)'</i>. Not really intuitive!</div><div>- You can also provide a step: <i>range.step(2)</i>. And '<i>.reversed()</i>'</div><div>- To print a range: '<i>for (i in range) println(i)</i>'. But not possible for a string-range because it has not iterator. Makes sense of course.</div><div>- But for a regular string it is possible: '<i>for (c in str) println(c)</i>'.</div><div>- Loop itself can also step: '<i>for (i in range step 4) println</i>'. Similar is there a '<i>downTo</i>' for counting down.</div><div>- To not include the last number: use keyword until: '<i>for (i in 1 until 10) println</i>'.</div><div>- Also variables can be used to define the range: '<i>val range = 1..someStr.length</i>'.</div><div>- Negate: '<i>val notInRange = 33 !in 1..5</i>' evaluates to true. But also: ''e' in "Hello"' is true.</div><div>- Index in for loop: '<i>for (index in myArray.indices) println("Index = $index, value = ${myArray[index]}")</i>'</div><div>- Much shorter: '<i>myArray.forEach { println(it) }</i>'. Or with index: <i>myArray.forEachIndexed {index, value -> println("$value is at array index $index") }</i></div><div>- Loops can given a name: '<i>myLoopName@ for (i in 1..5)</i>'. When having nested loops, in a subloop of myLoopName, you can say: '<i>break@myLoopName</i></div><div> Then you don't further continue running that loop with 'i' anymore either! Even when having more subloops before hitting the '<i>break@</i>' statement.</div><div><span style="white-space: pre;"> </span>It is almost a goto-statement! Risk of for example spaghetti-code.</div><div>- Works similar with '<i>continue@myLoopName</i>'.</div><h3 style="text-align: left;">If expression+statement</h3><div>- '<i>if</i>' can evaluate to a value! Not possible in Java. You have to put the return value as the last statement in a {} block. In both the if and else block!</div><div>- Comparable with the ternary operator in Java: '<i>val num = if (mycondition) 40 else 60</i>'. You can even write a full '<i>val num = if () {40} else {60}</i>'.</div><h3 style="text-align: left;">When expression</h3><div>- Is the Java 'switch' statement on steroids'</div><div>- '<i>when(condition) {</i></div><div><i><span style="white-space: pre;"> </span>10 -> println</i></div><div><i><span style="white-space: pre;"> </span>20 -> println</i></div><div><i><span style="white-space: pre;"> </span>else -> println("doesnt match")</i></div><div> '</div><div>- No need to add the 'break' statement. So no falling through at all</div><div>- More than 1 value possible in the match, e.g: '<i>10,11</i>' it will match on both those values. You can also use ranges. </div><div>- And expressions: '<i>x + 50 -> println()</i>'. And smart casting: '<i>when (something) { is String -> println() is BigDecimal -> println()}</i>'</div><div>- And also as in the if statement, you can return a value in each branch of the 'when'. Best practice is to *not* return a different type of value in the branches.</div><div>- An empty when, looking like an if statement: '<i>when { </i></div><div><i><span style="white-space: pre;"> </span>i < 50 -> println </i></div><div><i><span style="white-space: pre;"> </span>i >=50 println </i></div><div><i><span style="white-space: pre;"> </span>else println</i></div><div> }'. Which is much more concise than 3 if statement branches.</div><h3 style="text-align: left;">Try/Catch</h3><div>- Reminder: no distinction between checked and non-time exceptions.</div><div>- Also the try/catch you can use as expression</div><div>- E.g: '<i>return try { Integer.parseInt(str) } catch (e: NullPointerException) {0} finally { println("hello")}</i>'.</div><div> Note the finally block doesn't return an Int. But that is fine, since the catch block does and the normal block does. So the finally block is not used in the evaluation of a try/catch expression.</div><div>- Use case for <i>Nothing </i>return type for functions: as return type for a method that only throws an exception, e.g the <i>NotImplementedYet()</i> exception.</div><h3 style="text-align: left;">Lambda expressions basics</h3><div>- You can call them directly using the Kotlin library '<i>run</i>' function: '<i>run{println("lambda being run")}</i>'</div><div>- <i>'println(cars.minBy {car: Car -> car.builtYear}'). minBy</i> is a Kotlin collection function built-in. Note you can move the lambda outside () when the last parameter. And if the only parameter, even leave out the (). Note also the 'Car' could be left out, Kotlin can infer the type.</div><div>- In this case the compiler can infer even more, so we can use 'it': - '<i>println(cars.minBy {it.builtYear}</i>'). </div><div>- You can also use a member reference: '<i>println(cars.minBy(Car::builtYear))</i>'</div><div>- And toplevel function calls: '<i>run(::topLevelFun)</i>'. Where: '<i>fun topLevelFun(): println("hello")</i>'</div><div>- You can access local variables declared before the lambda in the lambda. In Java you can only access final variables in lambdas and anonymous classes.</div><h3 style="text-align: left;">Lambdas with receivers</h3><div>- '<i>with</i>' keyword: <i>'return with(StringBuilder()) { append("hello") toString() }</i>'. So can also be written as: '<i>return with(cars, {println(car)}</i>'.</div><div>- So the '<i>with</i>' turns the parameter you pass it into a receiver. That is how it is called. Inside the lambda you don't have to refer to the receiver object anymore. You can if you want with the 'this' keyword.</div><div>- '<i>apply</i>' keyword is another receiver keyword: '<i>StringBuilder().apply() { append("hello") }.toString()'</i>.</div><div>- A '<i>return</i>' in an inlined lambda also returns the function it is in, a so called non-local return.</div><div><span style="white-space: pre;"> </span><i>cars.forEach {</i></div><div><i><span style="white-space: pre;"> </span>if (it.builtYear == 2019) {</i></div><div><i> <span style="white-space: pre;"> </span>return</i></div><div><i><span style="white-space: pre;"> </span>}</i></div><div><i><span style="white-space: pre;"> </span>}</i></div><div><i> println("this is not executed when at least 1 car has builtYear 2019")</i></div><div>- You can have a local-return but then you need a label:</div><div><span style="white-space: pre;"> </span><i>cars.forEach returnBlock@ {</i></div><div><i><span style="white-space: pre;"> </span>if (it.builtYear == 2019) {</i></div><div><i> <span style="white-space: pre;"> </span>return@returnBlock</i></div><div><i><span style="white-space: pre;"> </span>}</i></div><div><i><span style="white-space: pre;"> </span>}</i></div><div><i><span style="white-space: pre;"> </span>println("this is now also executed even when at least 1 car with builtYear 2019")</i></div><div>- You can also use the label to refer to nested 'apply's and 'with's:</div><div><span style="white-space: pre;"> </span><i>"some text".apply sometext@ {</i></div><div><i><span style="white-space: pre;"> </span>"another text".aplly {</i></div><div><i><span style="white-space: pre;"> </span>println(toUpperCase()) // Only converts 'another text'</i></div><div><i><span style="white-space: pre;"> </span>println(this@sometext.toLowerCase()) // converts the outer 'some text'</i></div><div><i><span style="white-space: pre;"> </span>}</i></div><div><i><span style="white-space: pre;"> </span>}</i></div><div>- '<i>also</i>()' is similar to '<i>apply</i>()', but you don't need to use 'this'.</div><h3 style="text-align: left;">Collections: Lists</h3><div>- All read-only interfaces are covariant: be able to treat a class like its parent (superclass). E.g assign a List of 'Any's to a list of BigDecimals. See next Generics part for more details on this.</div><div>- For mutable collections you can't change the collection (of course), like adding elements. Seems usually these are also covariant (because they extend the (immutable) Collection interface which is covariant)</div><div>- '<i>listOf</i>' creates an immutable list. But '<i>arrayListOf</i>()' creates a mutable list! See the type of the underlying Java class: first one is Arrays$ArrayList which is inmutable, the 2nd is ArrayList, which is mutable. I guess using '<i>mutableList</i>' is more informative than using 'arrayListOf'.</div><div>- Handy: '<i>listOfNotNull(a,null,c)'</i> creates an immutable list of only a and c.</div><div>- Convert list to array: <i>'listOf(*arrayOf("a", "b", "c"))</i>'. Note the spread operator '*' there. But you can also do: '<i>array.toList()</i>'.</div><h3 style="text-align: left;">Kotlin's added collections functions</h3><div>- <i>last</i>(), <i>asReversed</i>()</div><div>- using the array notation to get an entry at given index: <i>list[5]</i> instead of list.get(5)</div><div>- <i>list.getOrNull()</i>: returns null when entry is null. Similar to Optional in Java.</div><div>- <i>max()</i>: entry with highest value</div><div>-<i> zip()</i>: creates Pair elements. This is Kotlin Pair so a bit different from Java Pair. For me a bit unintuitive name that 'zip'.</div><div>- '<i>val combinedList = list1 + list2</i>' // So you are really concatenating</div><div>- To combine & get no duplicates from 2 lists: '<i>val noDuplicatesList = list1.union(list2)</i>'. Just like union in SQL.</div><div>- To get no duplicates in 1 list: <i>list.distinct()</i></div><div>- Just like in Java, some functions return a new list, others work on the existing list. I expect you get at least an exception when trying the 2nd case on a immutable list.</div><h3 style="text-align: left;">Maps and destructuring declarations</h3><div>- <i>'mapOf<Int, Car>(1 to Car("green", 2012), 2 to Car("yellow", 2013))</i>'. The number is the key of the map. The <i><Int, Car></i> is redundant of course.</div><div>- And a mutable map: '<i>mutableMapOf()</i>'.</div><div>- In both cases the underlying implementation is LinkedHashMap, so the iteration order is predictable and easy to convert to a Set for example.</div><div>- But you can force a hashmap: '<i>hashMapOf()</i>'.</div><div>- Destructuring: '<i>val (firstValue, secondValue) = Pair(1, "one")</i>'. Kind of "unpacking". And even: '<i>for ((key, value) : in mutableMap) { println(key) println(value)}</i>'</div><div>- To be able to destructure a class, you need to use: <i>component functions</i></div><div>- To add them to your own class use:</div><div><span style="white-space: pre;"> </span><i>class Car(val color: String, val model: String, val year: Int) {</i></div><div><i><span style="white-space: pre;"> </span>operator fun component1(): color</i></div><div><i><span style="white-space: pre;"> </span>operator fun component2(): model</i></div><div><i><span style="white-space: pre;"> </span>operator fun component3(): year</i></div><div><i><span style="white-space: pre;"> </span>}</i></div><div> Unclear if these have to have such fixed names <i>componentN</i>().</div><div>- Data classes get the component functions already generated (created) for free.</div><h3 style="text-align: left;">Sets</h3><div>- To create an immutable set: '<i>setOf</i>()'. To add an element to a set: '<i>set.plus(str)</i>'. Duplicates added are ignored. To remove an element: '<i>set.minus(str)</i>'. (you get a new set when using immutable sets)</div><div>- '<i>drop(nr)</i>': drop the first 3 elements of the set</div><div>- Some functions work on the set itself, most return a new set with the function applied.</div><h3 style="text-align: left;">Other collections functions</h3><div>- <i>filter(lambda)</i>: returns a new instance of the collection</div><div>- '<i>val added10List = myIntsArray.map {it + 10}</i>': adds 10 to each element in myIntsArray and puts that in new list (a java.util.ArrayList)</div><div>- You can chain them of course: '<i>myCarsMap.filter { it.value.builtYear == '2012' }.map { it.value.color = "yellow"}</i>'</div><div>- To check for all elements matching a value, use the '<i>all(lambda)</i>' function, which returns a boolean. And '<i>any(lambda)</i>' for at least 1 matching.</div><div>-<i> count(lambda)</i></div><div>- '<i>groupBy(lambda)</i>'</div><div>- '<i>sortedBy(lambda)</i>'</div><div>- '<i>toSortedMap()</i>' to sort by key of the map</div><h3 style="text-align: left;">Sequences</h3><div>- <i>filter</i>() creates a whole new copy of the collection. Which can be huge. So to prevent too much memory usage, you can use sequences. Similar to Java streams, but re-invented by Kotlin because Android didn't support Java 8 yet back then. To make a map or list a sequence use: '<i>asSequence().filter()....</i>'</div><div>- 2nd case where you need to put a '<b>;</b>': when you have a lambda predicate which you want to put on 1 line, e.g: '<i>.filter { println("$it"); it[0] == 'yellow'}</i>'</div><div>- When you have a sequence, the filter(), map() etc return also a sequence, not the list/collection, because they are intermediate operations. So you need a <i>terminal operation </i>in the chain of calls. E.g: <i>'toList()</i>'. But for example 'find()' stops at the first match. Usually you want to 'map()' late(st) in the chain of calls. Just like Java streams.</div><h3 style="text-align: left;">Generics</h3><div>- You always have to specify the generic type in Kotlin in e.g<i> List<String></i>. So 'List' is not allowed. Also not for '<i>myList is List'</i>. There you can use: '<i>myList is List<*></i>'. That is called the <i>star projection</i>.</div><div>- <i>fun <T> myFunction(collection: List<T>) {}</i></div><div>- Or even as extension function: <i>fun <T> List<T>.myFunction() { println(......) }</i></div><div>- <i>fun <T: Number> myFunction(collection: List<T>)</i> to limit to certain subtypes (you specified the Number as upperbound)</div><div>- Multiple upperbounds: <i>fun<T> doIt(item1: T, item2: T) where T: CharSequence, T: Apppendable {}</i></div><div>- Note that T accepts the nullable type too! By default, if no upperbound specified, it is '<i>Any?</i>'. So nullable! I call this inconsistent with the "try to not support null" goal :)</div><div>- You can make it not accept nullable type by specifiying of course: '<i>T: Any</i>'.</div><div>- Just like in Java, at runtime the generic type is erased (not available anymore)</div><div>- In Java you can't do '<i>list instanceof List<String></i>'. But in Kotlin you can do: '<i>list is List<String></i>'. But it can't for a list of type Any.</div><h3 style="text-align: left;">Reified parameters (generics related)</h3><div>- <i>Reification</i>: prevents the type from being erased at runtime</div><div>- To do that: make the function 'inline' and 'reified T': </div><div><span style="white-space: pre;"> </span><i>inline fun <reified T> getElements(list: List<Any>): List<T> {</i></div><div><i><span style="white-space: pre;"> </span>for (element in list) {</i></div><div><i><span style="white-space: pre;"> </span>if (element is T) <----------- now compiles!</i></div><div><i><span style="white-space: pre;"> </span>}</i></div><div><i><span style="white-space: pre;"> </span>}</i></div><div> Then call it: <i>val myListOfAny: List<Any> = listOf("string", 1, 20.0f); getElements<BigDecimal(myListOfAny)</i></div><h3 style="text-align: left;">Covariance (generics)</h3><div>- <i>MutableList<Short></i> is not automatically the same as <i>ImmutableList<Short></i> at compile time in case of for example function parameters. Immutable collections are covariant, so then the "subtyping" is fine. But if you define the same parameter collection as mutable, the "casting" can't take place (compile error).</div><div>- So <i>MutableList </i>is invariant and wants the exact type (you see this also by the 'out' not present at that class's definition; more on that below)</div><div>- And so List has the '<i>out</i>' keyword, so is covariant. So subtyping is preserved. And since it is an immutable interface/class, nothing can change the list anyway. Still some functions have 'T' as parameter while 'out' is there! Like '<i>contains</i>()'. <i>@UnsafeVariance</i> is then used for such a (in) parameter, telling compiler that the list won't be changed.</div><div>- Note that: List is really a <b>class</b>. But <i>List<String></i> is a <b>type</b>! E.g: Short is a subclass of Number. List<Number> is supertype of List<Short></div><div><span style="white-space: pre;"> </span>A nullable type is a supertype of non-nullable types (a wider type).</div><div>- So <i>List<Short></i> is not by default the same as List<Number>, even though the 2nd is a super-type(!) of the first.</div><div>- Mutable collections interface = <b>not covariant</b>. Collections interface = <b>covariant</b>.</div><div>- So if you want the sub-typing(!) to be preserved, you have to prefix the type with the 'out' keyword. E.g: '<i>class Garden<out T: Flower></i>' </div><div>- That keyword implies you can use that class only in "out position". By default parameters are of '<i>in</i>' position. And usually the out position is the return type. By specifying the T as 'out' in the Garden above, you can *only* use it as return type!</div><div><span style="white-space: pre;"> </span>So '<i>fun getFlower(nr: Int): T {}</i>' is fine. But '<i>fun waterFlower(flower: T)</i>' won't compile!</div><div> <span style="white-space: pre;"> </span>Because otherwise you can pass in a different type than the original Garden was created with (e.g: rose vs daffodil); the compiler can't tell.</div><div><span style="white-space: pre;"> </span>Note that Garden is a type, not a class, due to that generics part.</div><div><span style="white-space: pre;"> </span>So in/out is protecting the class from invalid updating.</div><div>- So if you have a covariant class ('<i>out</i>' keyword), subtyping is preserved</div><div>- Constructors don't have in/out parameters. So you can always pass in a covariant class as constructor parameter, since it is at construction type.</div><div><span style="white-space: pre;"> </span>But if you specify a var parameter: can't be covariant class as input, because again you could pass in the wrong type, because it generates also </div><div><span style="white-space: pre;"> </span>a setter. But a '<i>private var</i>' is ok again because nobody outside the class can access it.</div><div>- Covariant type: subtyping preserved so you can pass an instance of the type or the subtype as for example a parameter. But you can't change the instance (thus when having the 'out' keyword).</div><div>- In Java it would be like this: List<? extends Car>, where you now accept anything that extends Car. This you see in Java in method declarations only. (a variable of that type you can't anything to)</div><h3 style="text-align: left;">Contravariance</h3><div>- Is the opposite of Covariance. With Covariance you are preserving the subtyping (incorrectly said(?): accept all of its sub"classes". Is it classes or types or both?). With covariance you start at a subclass and you want to accept instances of that subclass, or any of its superclasses.</div><div>- So you use the keyword '<i>in</i>':<i> interface FlowerStuff<in T> { fun doSomething(flower: T) }</i>. So if T is a direct or indirect superclass of a class, then the class will match T. And you can't have in that case the T as return type, so '<i>fun getFLower(): T</i>' does not compile. Again you won't be guaranteed to get the type you want if that would have been allowed. Or said differently: not all flowers are a rose. But all roses (subclass) are a flower (superclass).</div><div>- Again (depends on where you looking at from the inheritance tree; some say it is "flipping the inheritance tree for a class": </div><div><span style="white-space: pre;"> </span><b>Covariance</b>: you want to accept a class and all its subclasses ("looking down the inheritance tree")</div><div><span style="white-space: pre;"> </span><b>Contravariance</b>: you want to accept a class and all its superclasses ("looking up the inheritance tree")</div><div><span style="white-space: pre;"> </span>You are widening a generic type to include a class and its subclasses (covariance), or a class and its subclasses (contravariance).</div><div><span style="white-space: pre;"> </span>"<b>declaration site variance</b>", so used during declaring an interface. In Java you only have "use-site variance".</div><div>- In Java it would be: List<? super Car>: accepts Car and any of its superclasses. This you see in Java in method declarations only. (a variable of that type you can't read from)</div><h3 style="text-align: left;">Use-site variance</h3><div>- Generic types are invariant. So even if '<i>Ford: Car</i>', you can't call '<i>Car copyCars(source: MutableList<Car>, dest: MutableList<Car>)</i>' with '<i>copyCars(mutableListof(Ford(), Ford()), mutableListOf(Ford())</i>'. You have to change the Car type in the copyCars to 'T'.</div><div>- And to have '<i>copyCars(fords, cars)</i>' compile, you have to add '<i>out</i>' (covariance!) to the '<i>source: MutableList<out T></i>'. Because that parameter is not changed in the code, so we can do that. This is called use-site covariance, because we didn't add it to the Car class or its subclass like Ford. You can also add 'in' to the dest parameter, but is not needed in this case.</div><div>- Also called<b> type-projection</b>.</div><div>- In Java declaration-site variance doesn't exist, so you'd have to add the in/out to each method of the class. But so in Kotlin you can do it at class level, i.e declaration-site variance.</div><h3 style="text-align: left;">I/O</h3><div>- Just extension functions added to the java io classes like java.io.File.</div><div>- <i>var lines = File("myfile.txt").reader.readLines()</i></div><div>- Or to have the reader resource close automatically at the end:<i> val lines = File("myfile.txt").reader().use {it.readText}</i></div><div>- <i>reader().forEachLine()</i> to read per line</div><div>- instead of try with resources in Java, use the '<i>use()</i>' function, that closes the resource always, even if exception.</div><div>- <i>File(".").walkTopDown()</i> for file tree traversal</div><h3 style="text-align: left;">Java interoperability</h3><h4 style="text-align: left;">Calling Java from kotlin:</h4><div><span style="white-space: pre;"> </span>- Primitive arrays in the Java method as parameter like int[]: you have to use <i>toIntArray()</i> or <i>intArrayOf()</i>.</div><div><span style="white-space: pre;"> </span>- Nullability: <i>@Nullable @NotNull</i> are Jetbrains IntelliJ(!) hints that can be used. So they are NOT Kotlin annotations, but you give in IntelliJ the Kotlin compiler these hints. Quite ugly in your code base of course. It might also recognize Lombok's or javax's.</div><div><span style="white-space: pre;"> </span>But, you only get an IllegalArgumentException at runtime, so compiler won't complain if you assign null to a field.</div><div><span style="white-space: pre;"> </span>E.g: Java: 'void setColor(@NotNull color) {}' and then in Kotlin: '<i>var car: Car = null'</i> compiles fine, but IAE at runtime because the Kotlin compiler generates the not-null check.</div><div><span style="white-space: pre;"> </span>Default when none of these 2 annotations: it is nullable type for Kotlin</div><div><span style="white-space: pre;"> </span>- You can directly access the field of the Java class like car.color when the Java class has matching getter + setter methods. Or make the field public.</div><div><span style="white-space: pre;"> </span>- Exceptions from java: no need to add a throws to the Kotlin function</div><div><span style="white-space: pre;"> </span>- varargs in Java: you can't pass a Kotlin array, you'll have to spread (unpack, *) the array</div><div><span style="white-space: pre;"> </span>- void from java method: in Kotlin you'll see <i>Unit</i></div><div><span style="white-space: pre;"> </span>- Use Java Object methods that are not in Kotlin's Any, like the Object.notify() method: <i>(car.anObject as java.lang.Object).notify()</i>.</div><div><span style="white-space: pre;"> </span>- Static fields and methods in Java: are converted to companion objects, so just like this: <i>Car.myStaticfield</i> and <i>Car.myStaticMethod()</i>.</div><div><span style="white-space: pre;"> </span>- Single Abstract Methods in Java, e.g the Runnable interface which only has one method run() which you have to implement: you can pass them a Kotlin lambda (just like the Java lambda)</div><h4 style="text-align: left;">Calling Kotlin from Java:</h4><div><span style="white-space: pre;"> </span>- call toplevel Kotlin functions: Kotlin compiler creates a class based on the .kt filename. So: 'Car.kt' then you invoke '<i>Car.kt.myMethod()</i>'.</div><div><span style="white-space: pre;"> </span>But you can change that generated classname using the <i>@file:JvmName("myclassname")</i></div><div><span style="white-space: pre;"> </span>- Extension functions you can call from java by prefixing the classname too</div><div><span style="white-space: pre;"> </span>- getX() and setX() getters setters are available in Java for 'var' fields</div><div><span style="white-space: pre;"> </span>- To be able to directly access fields in Kotlin classes, you have to add the <i>@JvmField</i> on the field in the Kotlin class. Has a few use constrictions</div><div><span style="white-space: pre;"> </span>- Companion object access: usually: Car.Companion.myMethod(). If you want to not have to use the Companion part, annotate the companion method with</div><div><span style="white-space: pre;"> </span><i>@JvmStatic fun myMethod() {}</i></div><div><span style="white-space: pre;"> </span>- '<i>object MySingleton() { fun myMethod() }</i>' will be callable from java as: MySingleton.INSTANCE.myMethod(). Also here you can annotate the method with <i>@JvmStatic</i> to avoid that INSTANCE part.</div><div><span style="white-space: pre;"> </span>- For a '<i>const val</i>' in Kotlin, you don't need that <i>@JvmStatic</i>, it is directly accessible: MySingleton.myConstant</div><div><span style="white-space: pre;"> </span>- When passing null to a function in Kotlin that is expecting a non-null type, at runtime the Kotlin will check for null (generated by the Kotlin compiler)</div><div><span style="white-space: pre;"> </span>- To have Kotlin functions tell Java code an exception can be thrown, annotate method with: <i>@Throws(IOException::class)</i></div><div><span style="white-space: pre;"> </span>- When default parameter values for parameters in Kotin functions: to have Kotlin generate all combinations for those optional parameters use: <i>@JvmOverloads</i></div><div><br /></div></div><h1 style="text-align: left;">Not covered</h1><div><ul style="text-align: left;"><li>coroutines</li><li>DSL</li><li>Spring boot integration, annotations how?</li><li>Lombok annotations work with Kotlin?</li><li>How to do unittests</li><li>How to do integrationtests<br /></li></ul></div><div><br /></div><h1 style="text-align: left;">Considerations</h1><div><ul><li>Why are "methods" called functions in Kotlin? Functions sounds less Object Oriented...</li></ul></div>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-74026805841480654822022-09-14T13:09:00.001+01:002022-09-14T13:13:43.675+01:00Connection refused Spring Boot application to MySql in Docker solution<p style="text-align: left;"></p><h1 style="text-align: left;">Introduction</h1><p style="text-align: left;">The Spring Boot Java application was initially using the <a href="https://www.h2database.com/html/main.html">H2 embedded database</a>, first the in-memory version (losing all data each run), then the file based version. The database structure is created using <a href="https://www.liquibase.org/">Liquibase</a>.<br /><br />Then the requirement was to have the Spring Boot Java application still not dockerized (just started with <i>mvn spring-boot:run</i>), but have it connect to a <a href="https://dev.mysql.com/doc/relnotes/mysql/8.0/en/">MySql 8 database</a> than runs inside a Docker container. Most examples you can find have also the Spring Boot application in a <a href="https://www.docker.com/">Docker</a> container.<br />The official Spring documentation only shortly mentions the possibility of running MySql in Docker, but no more details on how to do that: <a href="https://spring.io/guides/gs/accessing-data-mysql/">https://spring.io/guides/gs/accessing-data-mysql/</a></p><p style="text-align: left;">This is how the setup should be working:</p><p style="text-align: left;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgf0OOhXiJcoXWOfL1HunSl6Fosz1vBoO2FEUYlgP6AE_rnX75BbiezqXTvTiCq9RYgDjlBuvNSle5_Q28nXaQ_ftWUMZN1zTqepcVrOor67g_Oy6cJsfZydnaNyITgq--V-aIxEB1Q1lCVdg0neOFbK59-GVtLq0FZkxiUsm8owYQnm4xWfNzXX3T5/s1625/blogpost-image-merged.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="810" data-original-width="1625" height="160" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgf0OOhXiJcoXWOfL1HunSl6Fosz1vBoO2FEUYlgP6AE_rnX75BbiezqXTvTiCq9RYgDjlBuvNSle5_Q28nXaQ_ftWUMZN1zTqepcVrOor67g_Oy6cJsfZydnaNyITgq--V-aIxEB1Q1lCVdg0neOFbK59-GVtLq0FZkxiUsm8owYQnm4xWfNzXX3T5/s320/blogpost-image-merged.png" width="320" /></a></div><br />I set up that docker container using the Windows 10 WSL 1 shell within IntelliJ (note WSL 2 is now recommended): <br /><br /><i>docker pull mysql/mysql-server:8.0<br />docker run --name recipesmysql8 -d mysql/mysql-server:8.0<br /></i><br />After changing the One Time Password (OTP) for root, creating the database user, creating a 'recipes' database, this is how <i>docker ps</i> looked:<br /><br /><span style="font-size: xx-small;"><i>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES<br />3a507fc66690 mysql/mysql-server:8.0 "/entrypoint.sh mysq…" 6 days ago Up 6 days (healthy) 3306/tcp, 33060-33061/tcp recipesmysql8<br /></i></span><br />And MySql is indeed running: <i>docker logs recipesmysql8</i> shows:<br /><br /><span style="font-size: xx-small;"><i>2022-08-31T19:02:05.722742Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 1<br />2022-08-31T19:02:05.755538Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.<br />2022-08-31T19:02:06.104813Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.<br />2022-08-31T19:02:06.416522Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.<br />2022-08-31T19:02:06.416574Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.<br />2022-08-31T19:02:06.443572Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock<br />2022-08-31T19:02:06.443740Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30' socket: '/var/lib/mysql/mysql.sock' port: 3306 MyS<br />QL Community Server - GPL.</i></span><br /><br />So all running fine, ports seemed fine. The Spring Boot application configuration for JDBC is this:<br /><br /><i>spring.datasource.url=jdbc:mysql://localhost:3306/recipes<br />spring.datasource.username=recipes<br />spring.datasource.password=mypasswd<br />spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver<br /></i><br />But then the <i>mvn spring-boot:run</i> gave this relatively vague error:<p></p><p style="margin-left: 40px; text-align: left;"><i>com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure</i><br /><i>The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.</i><br /><i> ...</i><br /><i> Caused by: java.net.ConnectException: Connection refused: no further information</i><br /></p><p style="text-align: left;">So unable to connect, but why? It does not seem an incorrect username/password combination, then I would expect some unauthorized/not authenticated type of message.<br />Then I found this handy command determining what the real IP should be of the machine that the MySql docker runs in:<br /><br /><i>docker inspect -f '{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq)<br /></i><br />I found my MySql container at: <i>/recipesmysql8 - 172.17.0.2</i><br />Same result, shorter answer: <i>docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' recipesmysql8</i><br />Or even just without the filtering: <i>docker inspect recipesmysql8</i><br /><br />So the JDBC configuration I changed to: <br /><br /><i>spring.datasource.url=jdbc:mysql://172.17.0.2:3306/recipes<br />spring.datasource.username=recipes<br />spring.datasource.password=mypasswd<br />spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver<br /></i><br />But then the <i>mvn spring-boot:run</i> gave this timeout error:</p><p style="margin-left: 40px; text-align: left;"><i>Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure</i><br /><i>The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.</i><br /><i> ...</i><br /><i> Caused by: java.net.ConnectException: Connection timed out: no further information</i><br /></p><p style="text-align: left;">So a similar error message, but this time a connection timeout. Is maybe the problem connecting from within IntelliJ where I execute the <i>mvn spring-boot:run</i> command, to Docker that runs in WSL?<br /><br />I tried by disabling the antivirus software and local firewall and WIFI, but those didn't work either, at most a different error message like:<i> </i></p><p style="margin-left: 40px; text-align: left;"><i>Caused by: java.net.NoRouteToHostException: No route to host: no further information</i></p><p style="text-align: left;">And even with the mysql shell client does it give an error (bit less clear):<br /><br /><i>mysql -h 172.17.0.2 -P 3306 --protocol=tcp -u root -p<br />Enter password: <br />ERROR 2003 (HY000): Can't connect to MySQL server on '172.17.0.2' (11)<br /></i><br /></p><h1 style="text-align: left;">Solution</h1><p style="text-align: left;">Then I stopped my container, started a new one with explicitly ports-mapping specified: <br /><br /><i>docker run --name recipesmysql8v2 -p 3306:3306 -d mysql/mysql-server:8.0</i><br /></p><p style="text-align: left;"><i>docker ps </i>output: <br /><span style="font-size: xx-small;"><i>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS<br /> NAMES<br />819333cf718e mysql/mysql-server:8.0 "/entrypoint.sh mysq…" 2 minutes ago Up 2 minutes (healthy) 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060-33061/tc<br />p recipesmysql8v2<br /></i></span><br />Then a different error message for <i>mysql -h 127.0.0.1 -P 3306 --protocol=tcp -u recipes -p</i> appeared:<br /><br /><i>ERROR 1130 (HY000): Host '172.17.0.1' is not allowed to connect to this MySQL server</i><br /><br />That looked promising.<br /><br />Note: due to having a new container, I had to reset the OTP for user 'root' again of course. You can find that OTP by issuing <i>docker logs recipesmysqlv2 | grep GENERATED</i>.<br />Then issue these commands:<br /><br /><i>docker exec -it your_container_name_or_id bash<br />mysql -u root -p<br />Enter password: <enter the one found with the above grep shell command> <br />ALTER USER 'root'@'localhost' IDENTIFIED BY 'your secret password';<br /></i><br />Create the Spring Boot application database again: <i>create database recipes;</i><br />Add the Spring Boot application user again: <i>create user 'recipes'@'%' identified by 'L7z$11Oeylh4';</i><br /><br />Give that user only the necessary permissions:<br /><i>grant create, select, insert, delete, update on recipes.* to 'recipes'@'%';</i><br /><br />After that, these entries should be in the mysql.user table: <br /><i>mysql> SELECT host, user FROM mysql.user;<br />+-----------+------------------+<br />| host | user |<br />+-----------+------------------+<br />| % | recipes |<br />| localhost | healthchecker |<br />| localhost | mysql.infoschema |<br />| localhost | mysql.session |<br />| localhost | mysql.sys |<br />| localhost | root |<br />+-----------+------------------+<br />6 rows in set (0.00 sec)</i><br /><br />Note the recipes user entry with host '%', which allows access from any host, which you probably want to make more secure in a production environment. See <a href="https://downloads.mysql.com/docs/mysql-secure-deployment-guide-8.0-en.pdf">https://downloads.mysql.com/docs/mysql-secure-deployment-guide-8.0-en.pdf</a> for tips.<br /><br />Now you should be able to connect from the mysql client prompt in several ways:<br /><br /><i>mysql -h 127.0.0.1 -P 3306 --protocol=tcp -u recipes -p<br />mysql -h localhost -P 3306 --protocol=tcp -u recipes -p<br />mysql -h 0.0.0.0 -P 3306 --protocol=tcp -u recipes -p<br /></i><br />Note that the 172.17.0.2 still does not connect! For that to work you'll have to probably add that host to the above mysql.user table (not tried to see if that works).<br /><br />Instead of mysql client you can also use the standard *nix command telnet to see if at least the port is reachable:<br /><br /><i>telnet 127.0.0.1 3306<br />Trying 127.0.0.1...<br />Connected to 127.0.0.1.<br />Escape character is '^]'.</i><br /><br />(I don't remember what it showed when the initial issue with the docker container named 'recipesmysql8' was there)<br /><br />And after starting the Spring Boot application: connection worked and tables and indexes were created!<br /><br />Note: only from <b><i>within </i></b>the docker image can you connect like this: <i>docker exec -it recipesmysql8v2 bash</i> and execute <i>mysql -u root -p</i> after having issued <i>ALTER USER 'root'@'localhost' IDENTIFIED BY 'nR128^n8f3kx';</i><br />Because running <i>mysql -h 127.0.0.1 -P 3306 --protocol=tcp -u root -p</i> from within the commandline WSL still gives: <i>ERROR 1045 (28000): Access denied for user 'root'@'172.17.0.1' (using password: YES)</i><br /><br />Probable cause: when you only have MySql in a Docker container, it is not in the same "network" as Docker containers, so the port 3306 is not reachable from outside Docker runtime. So you will have to tell Docker how you want to expose the port(s). Another way to solve this is to have the Spring Boot application also as a Docker application; and potentially configure it via docker-compose, see here for an example on how to do this: <a href="https://www.javainuse.com/devOps/docker/docker-mysql">https://www.javainuse.com/devOps/docker/docker-mysql</a>.<br /><br /><br /><br /></p>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-77381090444964622742022-04-06T15:52:00.002+01:002022-04-06T15:53:40.796+01:00Configuring MySQL test-containers in your Spring Boot Java Integration Tests<h1 style="text-align: left;">Introduction</h1><p style="text-align: left;">In your Integration Tests (IT) you often try to use the H2 in-memory database, to improve the speed of your integration tests. But on the other hand you want to mimic the production database as much as possible in your integration-tests.</p><p style="text-align: left;">Setting H2 in <a href="https://www.h2database.com/html/features.html#compatibility">MySQL database compatibility mode</a> <i>tries </i>to emulate MySQL as much as possible, but only a small subset of the differences are implemented. What for example not works correctly in H2 for JSON fields is that it escapes strings with "". For that reason you usually want to switch to for example starting a Docker database <a href="https://www.testcontainers.org/">testcontainer</a> in your IT tests, which uses a <i>real </i>MySQL database. With the Java-specific version in <a href="https://github.com/testcontainers/testcontainers-java">https://github.com/testcontainers/testcontainers-java</a>. <br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg89kTxRSARHVZNg6cpY17KdC7_A3nL-zE9sNUvqNLqTCAnp8vHOnVIwJeM5SJuoCB5QgGeFIcCDZ1x9khF-_6I-C66fUDxJ1Y4_AuUwVgCuEdsnaBW5k1ff4GIyeqeqTmaW_kejOxPCVReL_ACO4PVEvXlIyzrMfP6R2ObLgwjzat9SGQZqjXQ3igp/s1024/testcontainers-image.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="512" data-original-width="1024" height="160" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg89kTxRSARHVZNg6cpY17KdC7_A3nL-zE9sNUvqNLqTCAnp8vHOnVIwJeM5SJuoCB5QgGeFIcCDZ1x9khF-_6I-C66fUDxJ1Y4_AuUwVgCuEdsnaBW5k1ff4GIyeqeqTmaW_kejOxPCVReL_ACO4PVEvXlIyzrMfP6R2ObLgwjzat9SGQZqjXQ3igp/s320/testcontainers-image.png" width="320" /></a></div><br /><h1 style="text-align: left;">Configuration</h1><p style="text-align: left;">There are several good-to-know tips when configuring the testcontainers in your ITs.</p><ol style="text-align: left;"><li>The simplest configuration is using a datasource URL in the Spring Boot properties file. This has the disadvantage that whatever database name you specify, the testcontainers library still creates a DB named 'test'. So below it will be named 'integration_test_db' you'd think, but it is still named 'test' when the IT runs:<br /><br /><i><b>spring.datasource.url</b>=jdbc:tc:mysql:5.7.32:///integration_test_db?sessionVariables=sql_mode='STRICT_TRANS_TABLES'&TC_MY_CNF=mysql&TC_INITSCRIPT=mysql/init_mysql_integration_tests.sql<br /></i><br />To be able to do everything on the started database, including giving it the name you want, use this URL (or <b>see below the Java version</b>). Notice the user 'root' in the URL:<br /><i><b>spring.datasource.url</b>=jdbc:tc:mysql:5.7.32:///integration_test_db?user=root&password=&sessionVariables=sql_mode='STRICT_TRANS_TABLES'&TC_MY_CNF=mysql&TC_INITSCRIPT=mysql/init_mysql_integration_tests.sql<br /><br /></i>Via: <a href="https://github.com/testcontainers/testcontainers-java/issues/932">https://github.com/testcontainers/testcontainers-java/issues/932</a><br /><br /></li><li>To initialize your database, specify the script via this extra datasource URL variable:<br /><br /><i><b>TC_INITSCRIPT</b>=mysql/init_mysql_integration_tests.sql</i><br /><br />Note the default directory it looks in is <i>..../resources</i> for the scripts. So the full path is <i>..../resources/mysql/<br /><br /></i></li><li><i></i>An example to prevent GROUP BY error from strict mode, add this in your TC_INITSCRIPT:<br /><br /><i>SET GLOBAL sql_mode = 'STRICT_TRANS_TABLES';<br />SET SESSION sql_mode = 'STRICT_TRANS_TABLES';</i><br /><br /></li><li>To configure it in the IT Java class itself:<br /><br /><i>@RunWith(SpringRunner.class)<br />@SpringBootTest(classes = { SomeClassA.class, SomeClassB.class, ApplicationConfiguration.class}, webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)<br />@TestPropertySource(locations = {<br /> "classpath:/application-test-mysql.properties" })<br />@ContextConfiguration(initializers = {ThisITClass.Initializer.class})<br /><br />static class Initializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {<br /> public void initialize(ConfigurableApplicationContext configurableApplicationContext) {<br /> TestPropertyValues.of(<br /> "spring.datasource.url=" + mySQLContainer.getJdbcUrl(),<br /> "spring.datasource.username=" + mySQLContainer.getUsername(),<br /> "spring.datasource.password=" + mySQLContainer.getPassword()<br /> ).applyTo(configurableApplicationContext.getEnvironment());<br /> }<br /> }<br /><br />@ClassRule<br />public static MySQLContainer mySQLContainer = new MySQLContainer<>("mysql:5.7.31")<br /> .withUsername("root") // So now you can do a GRANT too for example<br /> .withPassword("") // Only possible for user 'root'<br /> .withEnv("MYSQL_ROOT_HOST", "%")<br /> .withDatabaseName("integration_test_db") // So this name will now be used, not 'test'<br /> .withInitScript("mysql/init_mysql_integration_tests.sql")</i><br /> ;<br /><br /></li><li>The MySQL docker image used in the tests is retrieved from DockerHub <a href="https://hub.docker.com/_/mysql">https://hub.docker.com/_/mysql<br /></a><br /></li></ol>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-62438684835080750042021-12-26T13:32:00.003+00:002021-12-26T13:33:18.102+00:00Sentry in Spring Boot application reports uncaught exception while @RestControllerAdvice is handling exception<h2 style="text-align: left;">Introduction</h2><p>Even when you have a <a href="https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/web/bind/annotation/RestControllerAdvice.html">@RestControllerAdvice</a> configured and you know it gets invoked, it can be that <a href="https://sentry.io/">Sentry.io</a>'s monitoring library still reports it as an (uncaught) exception. Reason for this is that when you configure Sentry as specified without setting its order in the exception-handler-resolver chain, it's by default set as lowest, so invoked as first in the chain. </p><h2 style="text-align: left;">Solution</h2><p>Thus to fix this, set it to a higher order number. Safest seems to set it to Integer.MAX_VALUE, that way it will always be invoked as the latest.</p><p>The most recent Sentry integration examples can be found <a href="https://docs.sentry.io/platforms/java/guides/spring/">here</a> for Spring and <a href="https://docs.sentry.io/platforms/java/guides/spring-boot/">here</a> for Spring Boot. Maybe you want to consider logging via <a href="https://logback.qos.ch/">Logback</a> or Log4J too, see the related Sentry integration examples on the same page.</p><p>Another solution is overriding the <i>getOrder()</i> method for older Sentry versions. Below is an example, using the suggesting configuration from this old obsolete Sentry <a href="https://docs.sentry.io/clients/java/integrations/#spring">page</a>.</p><div style="text-align: left;"><span style="font-size: x-small;"><i>@Configuration<br />@Slf4j<br />public class SentryConfig {<br /> @Bean<br /> public HandlerExceptionResolver sentryExceptionResolver() {</i></span></div><div style="text-align: left;"><span style="font-size: x-small;"><i><br /> return new SentryExceptionResolver() {<br /> @Override<br /> public ModelAndView resolveException(HttpServletRequest request,<br /> HttpServletResponse response,<br /> Object handler,<br /> Exception ex) {<br /> log.info("Sentry resolving this exception: ", ex);<br /> // You could skip exceptions that are considered client-side errors and return null in that case so it is considered taken care of.<br /> return super.resolveException(request, response, handler, ex);<br /> }</i></span></div><div style="text-align: left;"><span style="font-size: x-small;"><i><br /> @Override<br /> public int getOrder() {<br /> // Ensure other resolver(s) can run first, otherwise this handler is invoked as first and thus reporting an issue for Sentry<br /> return Integer.MAX_VALUE;<br /> }<br /> };<br /> }<br /> @Bean<br /> public ServletContextInitializer sentryServletContextInitializer() {<br /> return new io.sentry.spring.SentryServletContextInitializer();<br /> }<br />}</i></span></div><p style="text-align: left;"><br /></p><p style="text-align: left;">Similar solutions mentioned here: https://stackoverflow.com/questions/48401974/how-to-have-my-own-error-handlers-before-sentry-in-a-spring-application</p><p><br /></p>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-148998313453566492020-11-19T19:33:00.006+00:002020-12-12T14:26:24.274+00:00Spring @Scheduled using DynamoDB AWS X-Ray throws SegmentNotFoundException: failed to begin subsegment<h2 style="text-align: left;">Introduction</h2><div><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;"><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-Ym1JgldXU1s/X7bIJgKd0kI/AAAAAAAABAA/30SoO8t71ioNmsYqf5dBpDtwtjbu5CSwgCLcBGAsYHQ/s700/xray-example.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="492" data-original-width="700" src="https://1.bp.blogspot.com/-Ym1JgldXU1s/X7bIJgKd0kI/AAAAAAAABAA/30SoO8t71ioNmsYqf5dBpDtwtjbu5CSwgCLcBGAsYHQ/s320/xray-example.PNG" width="320" /></a></div></span></div><div><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">AWS X-Ray is designed to automatically intercept incoming web requests, see at this <a href="https://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html">introduction</a>. And also <a href="https://docs.aws.amazon.com/xray/latest/devguide/aws-x-ray-auto-instrumentation-agent-for-java.html">here</a>. </span></div><div><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;" /><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">But when you start your own thread (either via a Runnable or plain new Thread() or @Scheduled), X-Ray cannot initialise itself: there are no web requests to intercept for it. </span><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">Then it throws an exception like this: </span></div><div><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;"><br /></span></div><div><i><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">Suppressing AWS X-Ray context missing exception (SegmentNotFoundException): Failed to begin subsegment named 'AmazonDynamoDBv2': segment cannot be found.</span><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /></i><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;"><br /></span></div><div><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">In the above example the distributed <a href="https://aws.amazon.com/blogs/database/building-distributed-locks-with-the-dynamodb-lock-client/">DynamoDB Lock Client</a> was used, which uses DDB in its implementation for acquiring and releasing a lock.</span></div><div><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">Regular web requests were not throwing this X-Ray exception.</span></div><div><br /></div><h2 style="text-align: left;">Investigation</h2><div><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">Amazon explains that crucial bit of knowledge that web requests are "automagically" setting up the X-Ray recorder a bit <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-troubleshooting.html#troubleshooting-java">here</a>. </span><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">But that was not fully explaining it with an example. E.g only adding </span></div><div><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;"><br /></span></div><div><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;"><i>AWSXRay.beginSubsegment("AmazonDynamoDBv2") </i></span></div><div><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;"><br /></span></div><div><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">(and ending it) but that didn't fix it. That then gave this exception:</span><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /><i><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">Suppressing AWS X-Ray context missing exception (SubsegmentNotFoundException): Failed to end subsegment: subsegment cannot be found.</span><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /></i><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">My suspicion here is that the lock-client already closed the exactly same named subsegment "AmazonDynamoDBv2".</span></div><div><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">Also, I'm not creating any worker thread myself, Spring is doing it for me.</span></div><div><br /></div><div><span face="Helvetica Neue, Helvetica, Arial, sans-serif" style="color: #1d2228;"><span style="background-color: white;">Note that you at least can avoid exceptions be thrown <a href="https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java-configuration.html#xray-sdk-java-configuration-envvars">by setting environment variable</a> </span></span><code class="code" style="background-color: #f2f3f3; color: #16191f; font-family: Monaco, Menlo, Consolas, "Courier Prime", Courier, "Courier New", monospace; font-size: 16px;">AWS_XRAY_CONTEXT_MISSING</code><span face=""Amazon Ember", "Helvetica Neue", Roboto, Arial, sans-serif" style="background-color: white; color: #16191f; font-size: 16px;"> </span><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228;"> to LOG_ERROR. That will only log the above exception.</span></div><div><br /></div><h2 style="text-align: left;">Solution</h2><div><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">Creating a 'parent' segment <i>and</i> setting the trace entity <i>and</i> the subsegment did the job:</span><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /><i><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">Entity parentSegment = AWSXRay.beginSegment("beginSegmentForSomeScheduledTask");</span><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">AWSXRay.getGlobalRecorder().setTraceEntity(parentSegment);</span><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">AWSXRay.beginSubsegment("AmazonDynamoDBv2");</span><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /></i><br style="background-color: white; color: #1d2228; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" /><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">I did test only creating the subsegment and only creating and setting the parentSegment. But those raised the exception again. <br /></span></div><div><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">I did not further investigate whether the "AmazonDynamoDBv2" name of the subsegment is essential.</span></div><div><span face=""Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #1d2228; font-size: 16px;">And of course the matching closeSegment() and closeSubsegment() calls of course; not the reverse order: close the last one begun as first.</span><br /></div><div><br /></div><div><a href="https://stackoverflow.com/questions/53841672/aws-xray-sdk-issue-failed-to-begin-subsegment-named-amazon-s3-segment-cannot?rq=1">This thread</a> pointed me in the right direction. This was a next option I would have tried next: setting up my <a href="https://github.com/aws/aws-xray-sdk-java/issues/61">own filter</a> to run it earlier in the (filter) chain; though the @Scheduled task of course does not have a filter. Another workaround would have been to put the X-Ray logging at a very high level:</div><div><br /></div><div><i>logging.level.com.amazonaws.xray = SEVERE</i></div><div><br /></div><div>Also helped in explaining is this <a href="https://github.com/aws/aws-xray-sdk-java/issues/227">X-Ray reported issue</a>.</div><div><br /></div><div>Update: additionally, the error was also due to DynamoDB Lock Client! I did not specify <i>withCreateHeartbeatBackgroundThread()</i> when creating the lock client, but the exception of X-Ray showed that it was trying to send a heartbeat. After explicitly setting <i>withCreateHeartbeatBackgroundThread(false)</i> the exception (and error) regarding segment cannot found was fully fixed.<br /></div><div><br /></div><div><br /></div><div><br /></div><div><br /></div>Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-17940842247964009442020-04-05T14:33:00.002+01:002020-04-05T14:33:43.535+01:00PACT provider test with Serverless Java 11 Lambda and JUnit 5 (Jupiter)<h2>
Introduction</h2>
This blogpost shows how to create a <a href="https://github.com/DiUS/pact-jvm/tree/master/provider/pact-jvm-provider-junit5">Pact provider</a> test using:<br />
<ol>
<li><a href="https://github.com/DiUS/pact-jvm/releases/tag/4_0_0">Pact 4.0.0</a></li>
<li><a href="https://serverless.com/framework/docs/providers/aws/guide/intro/">AWS Serverless Lambda</a></li>
<li><a href="https://openjdk.java.net/projects/jdk/11/">Java 11</a></li>
<li><a href="https://junit.org/junit5/docs/current/user-guide/#overview-what-is-junit-5">JUnit 5 (Jupiter)</a></li>
</ol>
<h2>
Example Provider Test</h2>
Below is the example test class. Note how a mock service is used to simulate the endpoint defined (also) in serverless.yml. <br />
<br />
<pre>package com.ttlnews.pact.tests.provider;
import au.com.dius.pact.provider.junit.Provider;
import au.com.dius.pact.provider.junit.State;
import au.com.dius.pact.provider.junit.loader.PactBroker;
import au.com.dius.pact.provider.junit.loader.PactBrokerAuth;
import au.com.dius.pact.provider.junit5.HttpTestTarget;
import au.com.dius.pact.provider.junit5.PactVerificationContext;
import au.com.dius.pact.provider.junit5.PactVerificationInvocationContextProvider;
import com.amazonaws.services.lambda.runtime.Context;
import lombok.extern.slf4j.Slf4j;
import net.jcip.annotations.NotThreadSafe;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.TestTemplate;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockserver.integration.ClientAndServer;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.mockserver.integration.ClientAndServer.startClientAndServer;
import static org.mockserver.matchers.Times.exactly;
import static org.mockserver.model.Header.header;
import static org.mockserver.model.HttpRequest.request;
import static org.mockserver.model.HttpResponse.response;
import static org.mockserver.model.JsonBody.json;
/**
* Java 11 serverless provider Pact contract provider implementation using Pact with Junit5 (Jupiter) for provider SERVICE_A.
*
* Dependencies needed:
*
<dependency>
<groupId>au.com.dius</groupId>
<artifactId>pact-jvm-consumer-junit5</artifactId>
<version>4.0.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>au.com.dius</groupId>
<artifactId>pact-jvm-provider-junit5</artifactId>
<version>4.0.0</version>
<scope>test</scope>
</dependency>
*
*
*/
@Provider(SERVICE_A)
// Below environment variables must be set when running this test
@PactBroker(host = "${PACT_HOST}",
authentication = @PactBrokerAuth(username = "${PACT_HOST_USERNAME}", password = "${PACT_HOST_PASSWD}"))
@NotThreadSafe // Pact contract tests can't seem to handle methods running in parallel, so prevent maven failsafe/surefire plugin to run Pact tests in parallel
@Slf4j
public class ServiceAProviderTest {
private static final String SERVICE_A = "service-A";
private Context lambdaContext;
private SomeRepository someRepository;
private ClientAndServer mockClientAndServer;
@BeforeEach
void before(PactVerificationContext context) {
lambdaContext = mock(Context.class);
someRepository = new InMemoryMockedRepo();
mockClientAndServer = startClientAndServer(8888);
context.setTarget(new HttpTestTarget("127.0.0.1", 8888));
}
@AfterEach
void stopServer() {
mockClientAndServer.stop();
}
// This triggers the defined contract test(s) at the host PACT_HOST
@TestTemplate
@ExtendWith(PactVerificationInvocationContextProvider.class)
void pactVerificationTestTemplate(PactVerificationContext context) {
context.verifyInteraction();
}
@State("Create a new resource A")
public void shouldCreateResourceA() {
log.info("Set up state");
// Given
// Maybe some more mocking of 'lambdaContext' needed, depending on your case
// Lambda being tested
CreateResourceALambda createResourceALambda = new CreateResourceA(someRepository);
CreateResourceALambdaRequest createResourceALambdaRequest = CreateResourceALambdaRequest.builder()
.someValue("25")
.build();
// When
final LambdaResult lambdaResult = createResourceALambda.executeRequest(createResourceALambdaRequest, lambdaContext);
final String body = lambdaResult.getBody();
assertNotNull(body);
// Prepare the mockserver to return what the lambda returns when it is invoked (set up above in the 'body' variable)
// Of course the path to this CreateResourceALambda is defined in the serverless.yml, but that we can't access now. So need to repeat that
// endpoint here.
mockClientAndServer.when(
request()
.withMethod("POST")
.withPath("/resources/")
.withHeaders(
header("x-request-trace-id"), // Any value is fine
header("Authorization"), // Any value is fine
header("Content-Type", "application/json")
)
.withBody(json("{someValue: '25'}")) // Indeed need to use this strange JSON format
,
exactly(1))
.respond(
// Response as defined by the matching Pact consumer test; in this case found at host PACT_HOST
response()
.withStatusCode(201)
.withHeaders(
header("Content-type", "application/json; charset=utf-8"),
header("Authorization") // Any value is fine
)
.withBody(body)
);
}
}</pre>
<pre> </pre>
Migrating from your Pact tests from JUnit4 to Junit5 can be found <a href="https://medium.com/@kristine_jetzke/migrating-pact-contract-tests-from-junit4-to-juni5-d02194637903">here</a>.
<br />
<pre> </pre>
Note to self: use https://www.opinionatedgeek.com/codecs/htmlencoder to encode code, open the HTML view, and then wrap the code in <pre> open en close tag.Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-37757722265392474902019-10-02T11:34:00.000+01:002019-10-02T11:34:05.734+01:00Snyk .snyk ignore file with multiple entries format"<a href="https://snyk.io/about/">Snyk</a> enables you to find, and more importantly fix known vulnerabilities in your open source." <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-TwBvqOfCxQM/XSS1hXn290I/AAAAAAAAA8Y/5B93Q2fq_RcdNQJABJQdqCqSPI958p4hgCLcBGAs/s1600/snyk-bw.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="978" height="200" src="https://1.bp.blogspot.com/-TwBvqOfCxQM/XSS1hXn290I/AAAAAAAAA8Y/5B93Q2fq_RcdNQJABJQdqCqSPI958p4hgCLcBGAs/s200/snyk-bw.png" width="121" /></a></div>
<br />
Some vulnerabilities you'd want to ignore, because for example they are (for you) either false positives, or there is no workaround for them (yet). You can specify them via the so-called CLI <i>snyk ignore</i> command, or in the UI manually, or via a <i>.snyk</i> ignore file. An introduction focused on NPM can be found <a href="https://snyk.io/docs/ignoring-issues/">here</a>.<br />
The format with an example for one entry is specified here: <a href="https://support.snyk.io/hc/en-us/articles/360000923498-How-can-I-ignore-a-vulnerability-">https://support.snyk.io/hc/en-us/articles/360000923498-How-can-I-ignore-a-vulnerability-</a><br />
<br />
But what is the format for multiple entries? Most of the time with yaml (-like) formats you might expect a list, so each entry should be prefixed with a dash "-". But that is not the case here. The correct format for specifying multiple entries in the .snyk ignore file is:
<br />
<blockquote class="tr_bq">
<i>version: v1.5.0<br />ignore:<br /> 'SNYK-JAVA-xxxx-123456':<br /> - '* > com.abc:def':<br /> reason: 'Fix in progress'<br /> expires: '2020-01-05T08:00.00.000Z'<br /> 'SNYK-JAVA-xxxx-678900':<br /> - '* > org.abc:ghi':<br /> reason: 'Windows issue, systems run on Linux'<br /> expires: '2020-02-06T09:00.00.000Z'</i></blockquote>
<span style="white-space: pre;"> </span>
<br />
Note each block after the 'ignore' line is indented with two spaces.Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-80324925499662368802019-07-27T16:17:00.001+01:002019-07-27T16:17:38.877+01:00PACT Consumer Driven Contract Testing: how to allow any body in the response<br />
Consumer Driven Contract testing is a way to ensure that services (such as an API provider and a client) can communicate with each other. Without contract testing, the only way to know that services can communicate is by using expensive and brittle integration tests.<br />
<a href="https://docs.pact.io/">PACT</a> is a contract testing tool. <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-9EiNSqbbdIo/XTxjLYH2lOI/AAAAAAAAA8k/NJZk-AUCNqsSwC78VYbrJalOv-taZW-4wCLcBGAs/s1600/pact-logo.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="125" data-original-width="600" height="66" src="https://1.bp.blogspot.com/-9EiNSqbbdIo/XTxjLYH2lOI/AAAAAAAAA8k/NJZk-AUCNqsSwC78VYbrJalOv-taZW-4wCLcBGAs/s320/pact-logo.png" width="320" /></a></div>
<br />
For <a href="https://docs.pact.io/getting_started/matching">response matching</a> <span data-offset-key="4fc914fe0e814629bd16a0dc33e33c36:0">you want to be </span><span data-offset-key="4fc914fe0e814629bd16a0dc33e33c36:1"><em data-slate-leaf="true">as loose as possible</em></span><span data-offset-key="4fc914fe0e814629bd16a0dc33e33c36:2"> with the matching for the response (</span><span data-offset-key="4fc914fe0e814629bd16a0dc33e33c36:3"><code class="code-81e98f88" data-slate-leaf="true" spellcheck="false">will_respond_with(...)</code></span><span data-offset-key="4fc914fe0e814629bd16a0dc33e33c36:4" data-slate-fragment="JTdCJTIyb2JqZWN0JTIyJTNBJTIyZG9jdW1lbnQlMjIlMkMlMjJkYXRhJTIyJTNBJTdCJTdEJTJDJTIybm9kZXMlMjIlM0ElNUIlN0IlMjJvYmplY3QlMjIlM0ElMjJibG9jayUyMiUyQyUyMnR5cGUlMjIlM0ElMjJwYXJhZ3JhcGglMjIlMkMlMjJpc1ZvaWQlMjIlM0FmYWxzZSUyQyUyMmRhdGElMjIlM0ElN0IlN0QlMkMlMjJub2RlcyUyMiUzQSU1QiU3QiUyMm9iamVjdCUyMiUzQSUyMnRleHQlMjIlMkMlMjJsZWF2ZXMlMjIlM0ElNUIlN0IlMjJvYmplY3QlMjIlM0ElMjJsZWFmJTIyJTJDJTIydGV4dCUyMiUzQSUyMllvdSUyMHdhbnQlMjB0byUyMGJlJTIwJTIyJTJDJTIybWFya3MlMjIlM0ElNUIlNUQlN0QlMkMlN0IlMjJvYmplY3QlMjIlM0ElMjJsZWFmJTIyJTJDJTIydGV4dCUyMiUzQSUyMmFzJTIwbG9vc2UlMjBhcyUyMHBvc3NpYmxlJTIyJTJDJTIybWFya3MlMjIlM0ElNUIlN0IlMjJvYmplY3QlMjIlM0ElMjJtYXJrJTIyJTJDJTIydHlwZSUyMiUzQSUyMml0YWxpYyUyMiUyQyUyMmRhdGElMjIlM0ElN0IlN0QlN0QlNUQlN0QlMkMlN0IlMjJvYmplY3QlMjIlM0ElMjJsZWFmJTIyJTJDJTIydGV4dCUyMiUzQSUyMiUyMHdpdGglMjB0aGUlMjBtYXRjaGluZyUyMGZvciUyMHRoZSUyMHJlc3BvbnNlJTIwKCUyMiUyQyUyMm1hcmtzJTIyJTNBJTVCJTVEJTdEJTJDJTdCJTIyb2JqZWN0JTIyJTNBJTIybGVhZiUyMiUyQyUyMnRleHQlMjIlM0ElMjJ3aWxsX3Jlc3BvbmRfd2l0aCguLi4pJTIyJTJDJTIybWFya3MlMjIlM0ElNUIlN0IlMjJvYmplY3QlMjIlM0ElMjJtYXJrJTIyJTJDJTIydHlwZSUyMiUzQSUyMmNvZGUlMjIlMkMlMjJkYXRhJTIyJTNBJTdCJTdEJTdEJTVEJTdEJTJDJTdCJTIyb2JqZWN0JTIyJTNBJTIybGVhZiUyMiUyQyUyMnRleHQlMjIlM0ElMjIpJTIwdGhvdWdoLiUyMFRoaXMlMjBzdG9wcyUyMHRoZSUyMHRlc3RzJTIwYmVpbmclMjBicml0dGxlJTIwb24lMjB0aGUlMjBwcm92aWRlciUyMHNpZGUuJTIwJTIyJTJDJTIybWFya3MlMjIlM0ElNUIlNUQlN0QlNUQlN0QlNUQlN0QlNUQlN0Q=">) though. This stops the tests being brittle on the provider side. </span><br />
Most of the time you don't care about the exact values of the (JSON or XML) response, but you do care about the types of the values, e.g a string or a number.<br />
In that case you'll be using 'type matching'.<br />
<br />
Sadly for JVM matchers there's no such handy method that with one invocation makes sure only types are validated, as is for example with Ruby/Groovy/Javascript/Node: <i><span data-slate-fragment="JTdCJTIyb2JqZWN0JTIyJTNBJTIyZG9jdW1lbnQlMjIlMkMlMjJkYXRhJTIyJTNBJTdCJTdEJTJDJTIybm9kZXMlMjIlM0ElNUIlN0IlMjJvYmplY3QlMjIlM0ElMjJibG9jayUyMiUyQyUyMnR5cGUlMjIlM0ElMjJwYXJhZ3JhcGglMjIlMkMlMjJpc1ZvaWQlMjIlM0FmYWxzZSUyQyUyMmRhdGElMjIlM0ElN0IlN0QlMkMlMjJub2RlcyUyMiUzQSU1QiU3QiUyMm9iamVjdCUyMiUzQSUyMnRleHQlMjIlMkMlMjJsZWF2ZXMlMjIlM0ElNUIlN0IlMjJvYmplY3QlMjIlM0ElMjJsZWFmJTIyJTJDJTIydGV4dCUyMiUzQSUyMlBhY3QlM0ElM0FTb21ldGhpbmdMaWtlJTIyJTJDJTIybWFya3MlMjIlM0ElNUIlN0IlMjJvYmplY3QlMjIlM0ElMjJtYXJrJTIyJTJDJTIydHlwZSUyMiUzQSUyMmNvZGUlMjIlMkMlMjJkYXRhJTIyJTNBJTdCJTdEJTdEJTVEJTdEJTVEJTdEJTVEJTdEJTVEJTdE" style="white-space: pre;">Pact::SomethingLike.</span></i><br />
Not much documentation on the Java/JVM matchers can be found at the official Pact site itself. Mostly the examples are for Ruby/Groovy/Javascript/Node.<br />
<br />
For the Java and Virtual Machine integration you'd use <a href="https://github.com/DiUS/pact-jvm">pact-jvm</a> with matchers like <i>.stringType()</i> for the body using the <a href="https://github.com/DiUS/pact-jvm/tree/master/consumer/pact-jvm-consumer">PACT DSL</a> or <a href="https://github.com/DiUS/pact-jvm/tree/master/consumer/pact-jvm-consumer-java8">this lambda extension</a>.<br />
<br />
But hard to find was how to specify allowing either an empty body in the response or any data in the body in the response, while the consumer does not require a body at all (or in other words: is fine with an empty body or any fields in it).<br />
<b>Solution</b>: for that you need to completely <b>omit </b>the <i>.body()</i> in the consumer contract definition.<br />
If you specify a <i>.body(new PactDslJsonBody()),</i> the contract generated matcher will specify <i>"body" : {}</i>, and therefor <i>requiring </i>an empty body. And if then the provider (test) generates one or more fields in the response, you'll see this as the message in the failing test:<br />
<br />
<span></span><br />
<pre class="yiv0854443874ydp2c4a09f1c-mrkdwn__pre" style="background-attachment: scroll; background-color: #f8f8f8; background-image: none; background-repeat: repeat; background-size: auto; border-radius: 4px; border-style: solid; border-width: 1px; color: #1d1c1d; font-family: Monaco, Menlo, Consolas, monospace !important; font-size: 12px; font-style: normal; font-weight: 400; letter-spacing: normal; line-height: 1.50001; margin: 4px 0px; padding: 8px; text-align: left; text-decoration-color: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; word-spacing: 0px;">Expected an empty Map but received Map(..... fields added by provider ...)</pre>
<br />
<br />So full example which accepts an empty body or a body with elements in it:<br />
<br />
<pre>.consumer(<span class="pl-s"><span class="pl-pds">"</span>Some Consumer<span class="pl-pds">"</span></span>)
.hasPactWith(<span class="pl-s"><span class="pl-pds">"</span>Some Provider<span class="pl-pds">"</span></span>)
.given(<span class="pl-s"><span class="pl-pds">"</span>a certain state on the provider<span class="pl-pds">"</span></span>)
.uponReceiving(<span class="pl-s"><span class="pl-pds">"</span>a request for something<span class="pl-pds">"</span></span>)
.path(<span class="pl-s"><span class="pl-pds">"</span>/hello<span class="pl-pds">"</span></span>)
.method(<span class="pl-s"><span class="pl-pds">"</span>POST<span class="pl-pds">"</span></span>)
.toPact()</pre>
<br />
<br />Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-7432653615755371382019-07-09T16:13:00.002+01:002019-07-09T16:13:46.658+01:00Maven Failsafe plugin build error: SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-PSMLIEMsOHY/XSSuNlXQsAI/AAAAAAAAA8M/N9B3NGmA10YgAJvCKFIpJBu5a1x6_5zkACLcBGAs/s1600/mavenlogo.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="107" data-original-width="462" height="73" src="https://1.bp.blogspot.com/-PSMLIEMsOHY/XSSuNlXQsAI/AAAAAAAAA8M/N9B3NGmA10YgAJvCKFIpJBu5a1x6_5zkACLcBGAs/s320/mavenlogo.gif" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
When you get this exception during running <i>mvn clean verify, </i>which executes both the <a href="https://maven.apache.org/surefire/maven-surefire-plugin/">Surefire</a> and <a href="https://maven.apache.org/surefire/maven-failsafe-plugin/">Failsafe</a> plugin (for which the last one uses the Surefire plugin!):<br />
<br />
<i>SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?</i><br />
<br />
it can mean several things. For example as the message says maybe somewhere in your integration tests you call System.exit, which you shouldn't do. That one is easy to find. Also <a href="https://stackoverflow.com/questions/23260057/the-forked-vm-terminated-without-saying-properly-goodbye-vm-crash-or-system-exi">this Stackoverflow post</a> covers that plus some other options, like shortage of memory.<br />
<br />
It has also higher chance on happening when using the <a href="https://github.com/carlossg/docker-maven">docker-maven image</a> and 3.x-openjdk-8 or 3.y-openjdk-10.<br />
<br />
But if that doesn't fix it, what then? Then due to<a href="https://github.com/carlossg/docker-maven/issues/90#issuecomment-434221709"> this issue</a> you probably have to add this property in your pom.xml in the configuration of <b><i>both </i></b>the Surefire and the Failsafe plugin:<br />
<br />
<blockquote class="tr_bq">
<span style="background-color: rgba(27, 31, 35, 0.05); color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.9px;"><usesystemclassloader>false</usesystemclassloader></span></blockquote>
<br />
Thus not only for the Failsafe plugin, for both! :)<br />
<br />
PS: adding that configuration property also is a workaround for <a href="https://issues.apache.org/jira/browse/SUREFIRE-1588">https://issues.apache.org/jira/browse/SUREFIRE-1588</a>, also reported here: <a href="https://stackoverflow.com/questions/50661648/spring-boot-fails-to-run-maven-surefire-plugin-classnotfoundexception-org-apache/50661649#50661649">https://stackoverflow.com/questions/50661648/spring-boot-fails-to-run-maven-surefire-plugin-classnotfoundexception-org-apache/50661649#50661649</a><br />
<br />Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-25238545708941993142019-03-31T15:25:00.000+01:002019-03-31T15:28:08.047+01:00ReadProvisionedThroughputExceeded using Kinesis<h2>
ReadProvisionedThroughputExceeded</h2>
When using Kinesis it is possible you'll get an ReadProvisionedThroughputExceeded when not carefully designing the shards consumption of messages.<br />
<br />
That exception indicates "<a href="https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-cloudwatch.html">The number of GetRecords calls throttled for the stream over the specified time period</a>".<br />
<br />
A cause for this can be you have to many consumer applications at the same time issue GetRecord calls to the same shard.<br />
<br />
One recommended solution is to increase the number of shards. But that has a price-tag attached to it. Another solution is process (batches) of records in parallel by spinning off threads within the processRecords() method in KCL. But that will require careful orchestration of threads to keep checkpointing correctly. An interesting analysis of why using KCL and especially version 2 can be found <a href="https://medium.com/@ozturkaburak/aws-kinesis-multiple-instance-consumer-65ed620e3798">here</a>.<br />
<br />
For a more detailed analysis, see the longer blogpost <a href="https://ttlnews.blogspot.com/2019/03/lessons-learned-using-aws-kinesis-to.html">here</a>.<br />
<div>
<br /></div>
<div>
<div class="post-body" style="background-color: white; color: #333333; font-family: Georgia, serif; font-size: 13px;">
<div style="line-height: 1.6em; margin-bottom: 0.75em;">
</div>
<div style="clear: both;">
</div>
</div>
<div class="post-footer" style="background-color: white; color: #333333; font-family: Verdana, sans-serif; font-size: 8.19px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; letter-spacing: 0.1em; line-height: 1.4em; margin: 0.75em 0px; text-transform: uppercase;">
</div>
</div>
Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0tag:blogger.com,1999:blog-8474871406958844140.post-28948370513657364562019-03-31T15:08:00.002+01:002019-03-31T15:25:50.151+01:00ProvisionedThroughputExceededException using KCL (Kinesis Client Library)<h2>
ProvisionedThroughputExceededException</h2>
<div>
When using the Amazon Kinesis Client Library (KCL) one would not expect the table created by KCL <i>itself</i> to keep track of its streams, shards and checkpoints to ever be under-configured.</div>
<div>
But it can be! Then you get the <a href="https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/model/ProvisionedThroughputExceededException.html">ProvisionedThroughputExceededException</a>. </div>
<div>
<br /></div>
<div>
Normally it will recover by retrying <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html">itself</a>. But if it can't, other solutions exist:</div>
<div>
<ol>
<li>Change the read/write provisioned for that table to 'on-demand'. </li>
<li>Make sure your hashkeys are evenly distributed, as mentioned here <a href="https://stackoverflow.com/questions/29101371/getting-provisionedthroughputexceededexception-error-when-iterating-over-dynamod">not-evenly distributed hash-keys</a>.</li>
<li>Modify how many consumers (applications) are reading from the same shard(s) at about the same time, either by changing that design or <a href="https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-additional-considerations.html">increase the number of shards</a>.</li>
</ol>
<div>
For a more detailed analysis, see the longer blogpost <a href="https://ttlnews.blogspot.com/2019/03/lessons-learned-using-aws-kinesis-to.html">here</a>. </div>
</div>
Techiehttp://www.blogger.com/profile/09242813498513889831noreply@blogger.com0