Tuesday, September 27, 2022

Kotlin for Java Developers course summary

Introduction

This is my summary of relevant lessons learned during a Kotlin for Java developers course.


Summary

General

- When running standalone, needs Kotlin runtime
- Toplevel functions: not explicitly defined in a class, e.g fun main() method
- Useful other info: https://kotlinlang.org
- val number: Int and val number = 25    So has type inference.  But if needed: number: Short = 25
- Recommended good practice: delare variables with 'val' where possible
- Properties of a 'val' class instance you can modify
- Via class "constructor" you can also specify whether to set properties is allowed. E.g class Employee(var name: String, val id: Int), then you can do: employee.name = "abc", but not employee.id = 5.  Note that construction of instance needs both parameters anyway.
- Collections: you can use [i] notation to get to an element.  Or for Maps: use the key
- No difference between checked and unchecked exceptions, no need to specify them in method names. You can't even add a 'throws'.
- Kotlin has no static keyword syntactically, e.g see the 'fun main()' declaration. But they do exist.
- No 'new' keyword
- The '==' operator checks for structural equality, so it works the same as .equals()! So .equals() you don't need to use. How to check for referential equality use: '==='
      Annoying, in Javascript '===' means more equality, in Kotlin less (only referential equality).
- Bitoperators like '|' you spell out like: 'or'
- Use 'is' instead of 'instanceof'
- Smart casting: use the 'as' keyword, e.g: val newValue = something as Employee. But if you did an 'is' check before, you don't need the cast anymore.
- Raw strings can be triple quoted, so you don't need to escape special characters, e.g: val path = """c:\somedir\somedir2"""
- trimMargin() for indentation in code of raw strings that cover multiple lines
- Everything is a class, no native types like long or int.
- Kotlin does not automatically widen numbers. E.g: in Java: int a = 55;long b = a; is fine. But not in Kotlin. So all standard datatypes have toLong() and similar conversion methods
- Double is default just like in Java when declaring variable w/o specifying type, e.g: val myDouble = 65.994. Int also, e.g: val myInt = 11
- Any class: similar to root class of Java class hierarchy
- Unit class: similar to void in Java, but different: it is a singleton unit instance
- Nothing class: subclass of any class. E.g when you have a function with an infinite loop, then return 'Nothing'.
- The RPEL can be used as "scratchbook" of executable code, which knows of your existing classes

Arrays

- Initialize array using a lambda, even after declaration: val evenNumbers = Array(16) {i -> i * 2}     So here i is the index of the array
- Mixed array: val mixedArray = arrayOf("Hello", 22, BigDecimal(0.5), 'a').  It is then an array of Array<Any>.  Or just call .toIntArray() on the Kotlin Int array.
- To call a Java method with int[] as parameters, you can't pass in the Kotlin collection Array, so you have to use: intArrayOf(1, 2, 3) instead of arrayOf(1,2,3). Is also a bit of performance boost since directly you create a primitive type array for the JVM (of type int).
- You can't do var someArray = Array<Int>(5)
- You can do for any type: val myData = 'arrayOf(car1, car2, car3)', where the 3 cars are of type Car.
- Convert int[] created via intArrayOf() to a Kotlin typed array: val typedArray = intArrayOf(1,2,3).toTypedArray() 

Null references

- For types that can be nullable, you have to tell it can be null by appending a '?', e.g: 'val str: String? = null'
- If you check for variable being null, Kotlin compiler from then on knows it can't be null, so all methods are available again (similar to smart-casting)
- 'str?.toUpperCase()' performs behind the scenes a 'if (str != null) str.toUpperCase() else null'. The '?' is called the "safe operator".
- So nested also possible, then it stops at the first one that is null. E.g: 'home?.address?.country?.getCountryCode()'
- To set a default value in case it evaluates to null, use the elvis operator: '?:'. E.g: 'val str1: String? = null; val str2 = str1 ?: "this is the default value"'
- Safe cast operator allows any cast, and if it can't do it, it sets the variable to null. E.g: 'val someArray: Any = arrayOf(1,2,3); val str = someArray as? String' results in 'str' having value null.  So '?' really means: this (variable) *might* from now on evaluate to null.
- If you are sure an expression can't evaluate to null, you can tell the compiler via the non-null assertion: str!!.toUpperCase().  After this point you don't have to add the '?' anymore for null safety checks (because you told the compiler it 100% can't be null). Use this e.g when you *do* want a nullpointer exception to be thrown.
- The 'let' function: 'str?.let {println(it)}', which says: if str isn't null, then let this function-call go ahead via a lambda expression.
- Note that the == operator is safe for nulls (note underwater it uses .equals()), e.g: 'val str1 : String? = null; val str2 = "abc"; str1 == str2'
- To have an array of null values, you have to use 'arrayOfNulls()' method, e.g: 'arrayOfNulls<Int?>(5)' creates one of size 5. Or: 'arrayOfNulls<Int>(5)'
- Use !! when you are sure the variable can never ever be null

Access modifiers

- 4 types for toplevel items:
Default = public final (instead of package level in Java). Note classname doesn't have to match filename in kotlin. One file can have multiple public classes.
Private = Everything in the same file can access it. (Java doesn't allow a private class). When specified for a constuctor parameter, no getters/setters are generated! So really only the class itself can have access to that property, no-one else.
Protected = can't be used
Internal = is inside the module (see below) (no Java equivalent)
- Kotlin has the 'module' principle: group of files that are compiled together.
- For class members:
Public, private, protected mean the same as in Java.
Internal = visible in same module, usually outside the file
- In Kotlin, classes can't see private members belonging to inner classes
- Compile time:
private in Kotlin is compiled to package private
internal in Kotlin is compiled to public
That means sometimes Java code can access Kotlin code that from w/in Kotlin (compiler) can't be accessed
Kotlin generates really long strange names for those type of methods/classes defined as internal, so you are kindof warned when using them from Java.
- Create constructor verbose, Java-like: 
class Employee constructor(firstName: String) {
         val firstname: String

     init {
this.firstName = firstName
    }
        }
Note the 'constructor' keyword as primary constructor (declared outside the curly braces). 
And the 'init' block. The init block runs after all fields have been initialized. 
- Create less verbose:
class Employee constructor(firstName: String) {
            val firstName: String = firstName
        }
- Even less (note the 'val' keyword added; can also be 'var'):
class Employee constructor(val firstName: String) {
        }
- Even less:
class Employee (val firstName: String) {
        }
- But if you want to change visibility of the constructor, you will have to add the 'constructor' keyword: class Employee protected constructor(...)
- For more constructors, just add 'constructor's in the code. But you also have to invoke the primary constructor if present. For that do:
constructor(firstname: String, lastName: String): this(firstName) {
xxx
}
Note no 'val' prefix anymore, since the 2nd constructor does not declare variables! Just declare the 2nd parameter as field in the class.
- 'val' and 'var' in the primary constructor generate the matching properties. If not specified, it is just a parameter in that primary constructor.
- You don't *have* to specify a primary constructor
- Kotlin classes only have properties, no fields (what's the difference? Note there's use of something named the "backing field", see below). Default for properties is also 'public'.
- Properties defined as 'var' will have setter generated too. 
- 'private var' properties: can't be accessed at all.  This is because the (generated) getters + setters must have the same visibility as the property.
- If you want your own getters+setters, you can't declare that property in the primary constructor; it has to be declared in the class.
- Backing field (strange name to be just thrown in in the course):
class Employee(val firstName: String, lastName: String = "") {
        var lastName = lastName
                get() {
return field
                }
       }
   Note the get() has to be immediately after the property. And also the special keyword 'field'.
- Similar for the set() method. But note the caller can do: employee.lastName = "abc". No 'setLastName()' call needed.
- You have toplevel constants too just like toplevel functions. So you don't need to declare a constants file for example. val MY_CONSTANT: String = "Text" . Note also the 'const' keyword exists.
- Data class: 'data class Bike(val color: String, val model: String) {}'. You get for free: toString(), equals()+hashCode(), copy() function.
    Requirements: primary constructor has to have at least 1 parameter marked val or var (unlike lastName in previous example). Also: can't be abstract, sealed or inner classes.
- Parameters can be passed in in any order because you can specify the name of the parameter during the call. These are called "named arguments".
- Internal access modifier: private class + internal functions is meaningless, since the private class already restricts it to the file.
- listOf() returns an immutable list of items

Functions

- Simplified function definition: fun myFunction(param1: Int, param2: Int) = "Multiplied = ${param1 * param2}". So the function returns what is after the equals sign. Defining it this way (so not with a {} block body) is called an expression body.
- Returning nothing means specifying Unit (or leaving it out since it is the default)
- Function parameters must have a type specified (even though compiler could derive the type!)
- Variable number of arguments: 'vararg' keyword, e.g: 'fun addUp(vararg amounts: Int)'
- Spread operator: unpacks the array to pass each element separately (used when having to pass to a vararg parameter): addUp(*arrayOfInts). Yes a '*'.
    In Java you could pass in an array of objects to a method with a varargs parameter, but not in Kotlin.
    Another example: 'val newArray = arrayOf(*array1, *array2, anotherElement)'.  If you don't do this, you'll have 2 arrays + one element in 'newArray'!
- Extension functions lets you add functions to classes. Just add the class + a dot (the so-called "receiver type") to a function, e.g: 'fun String.replaceEWithA() {}'. You access the string with 'this' keyword in that method.
    Note it knows since you are putting the extension on String, no need for a String parameter in the new function. 
    Any public field in the class you add the extension function to can be accessed.
    Feels a bit like a "hack" adding random functions to existing classes. People might also get lazy, not coming up with a new class, but just adding it to an existing one because it matched "good enough". Also for a newcomer on the project it is not possible to easily distinct that a method is an
       extension function.
- Function parameters are of type 'val'.
- Inline functions: you can tell the compiler to inline by prefixing function definition with the 'inline' keyword. Usually works best for functions with lambda parameters

Inheritance

- Extend (subclass) a class: 'class RaceBike(): Bike() {}'. Class Bike has to have the 'open' keyword as prefix because default is final (not extendable).
So: 'open class Bike() {}'.  And if you don't want to specify an empty primary constructor: add 'constructor(): super()' to the subclass RaceBike.
- For an abstract class you can remove the 'open' keyword, because 'abstract' implies already that it will be extendable.
- Overriding a method in the subclass requires 'override' in the subclass method and 'open' in the superclass method.
- An 'abstract fun' also needs the 'override' keyword in the subclass. No need for the 'open' keyword here either, it is automatically. Same for the 'override' keyword!
- Subclass (of course) doesn't have to have the same number of parameters for the primary constructor
- Invoking "super.primary constructor" only makes sense/possible if you don't have any primary constructor defined for the parent
- Usually you'll provide a primary constructor and only add secondary ones when absolutely necessary
- In a subclass when creating a secondary constructor, call the constructor (IntelliJ might report it is missing superclass call or similar) like this: 
'constructor(prop1, prop2,..., newParam: Int): this(prop1, prop2,...)'
- Data classes can't be extended (nor be abstract class or inner class)
- Interfaces can be extended with the same syntax: 'interface MyChildInterface: MyParentInterface {}'. Classes can implement multiple interfaces like Java.
- Interfaces can have properties. Which the implementing class has to define too with an override: 'override val number: Int = 30'.
- If you want to set a property's value in an interface, giving it a concrete implementation, you have to define a 'get()' directly below it. Not sure if you'd ever want that in an interface...
- Properties in interfaces don't have the backing field 'field' identifier (which classes do have in the custom get() and set()).
- Making a parameter optional (with default value) in a function in a superclass, doesn't make it an optional parameter in its subclasses.

object keyword

- 'object' keyword is used for: singletons, companion objects, object expressions.
- For singleton there's always only one instance created, very first time you use this class. So I guess not static classes in Java which are created I think at application startup.
     You can access the functions of the singleton, quite normally: MySingleton.someMethod().   Note the singleton class is not constructed first.
- If you want to add e.g functions to a class that are statically accessible (so w/o having to create the class), wrap it with 'companion object {}'.
     The non-private methods in that block are from then on accessible, using special keyword: 'MyClass.Companion.someMethod()'. But then the class is not a singleton but a regular class with a companion object block in it.  You can also give the companion a name. Kotlin also is smart enough you can ommit Companion. This is really much more verbose than the Java 'static' keyword!
- Companion objects can also be used to call private constructors. You can use them to implement factory patterns.
Of course you have to make the primary constructor (if exists) private too then to prevent instantiation outside the factory, e.g: 
class SomeClass private constructor(val input: String) {
companion object: {
fun createBasicSomeClass(val input: String) = SomeClass(input)
}
}
- Object expressions are where in Java you'd use anonymous classes, e.g implementing an interface that needs to be passed as parameter to a function.
Also called anonymous instances.
E.g: 'functionToCall(object: SomeInterface {
override fun functionToImplement(text: String) = "implemented text $text"
}
      '
Note that the object created is not a singleton (confusing, since you use the word 'object' which is also used to create a Singleton)
Also, the object expression code can access local variables in scope (they don't have to be 'val' (or "final" in Java terms), can be 'var' too). So also change that local variable's value!

Enums 

- Use 'enum class' keyword. 
- How annoying: if you add a 'fun' to an enum, you have to add a ';' after the last enum value!! Odd that compiler can't figure it out w/o the ';'...
- Example invoking a function in enum for a given enum value: 'DepartmentEnum.ACCOUNTING.getInfo()'

Importing

- Recommended to (of course) use the same package structure as underlying directory structure. (though it is not a requirement from Kotlin)
- Toplevel functions (in files, not in a class) can be imported like a class
- What if you want to use a function from another module? Then (in IntelliJ) you have to add the dependency to that module first. Will be a regular Maven dependency in Maven I expect.
- Type aliases and extension functions you import the same (they are all top-level anyway)
- You can rename imports too using the 'as' keyword. Useful when 2 third party libraries use the same name for some class.

For loop

- while and do-while is same in Kotlin
- For loop syntax like in Java doesn't exist in Kotlin. 
- Range: values are inclusive. E.g 'val range = 1..5'.  But also: val charRange = 'a' .. 'z'. Because they are comparable. val stringRange = "ABD".."XYZ"
- 'in' keyword for: 'println(3 in range)' is true. But also '"CCCCC" in stringRange' is true! Because the first 'X' is already greater than the first 'C'.
      '"ZZZZZ" in stringRange' is false because the first 'Z' is greater than 'X'
- 'val rangeBackwards = 5..1' is usually not what you want because '5 in rangeBackwards' is false! Because 5 >= 5 but 5 <= 1. So you have to do: '5.downTo(1)'. Not really intuitive!
- You can also provide a step: range.step(2). And '.reversed()'
- To print a range: 'for (i in range) println(i)'.  But not possible for a string-range because it has not iterator. Makes sense of course.
- But for a regular string it is possible: 'for (c in str) println(c)'.
- Loop itself can also step: 'for (i in range step 4) println'. Similar is there a 'downTo' for counting down.
- To not include the last number: use keyword until: 'for (i in 1 until 10) println'.
- Also variables can be used to define the range: 'val range = 1..someStr.length'.
- Negate: 'val notInRange = 33 !in 1..5' evaluates to true. But also: ''e' in "Hello"' is true.
- Index in for loop: 'for (index in myArray.indices) println("Index = $index, value = ${myArray[index]}")'
- Much shorter: 'myArray.forEach { println(it) }'. Or with index: myArray.forEachIndexed {index, value -> println("$value is at array index $index") }
- Loops can given a name: 'myLoopName@ for (i in 1..5)'. When having nested loops, in a subloop of myLoopName, you can say: 'break@myLoopName
    Then you don't further continue running that loop with 'i' anymore either!  Even when having more subloops before hitting the 'break@' statement.
It is almost a goto-statement! Risk of for example spaghetti-code.
- Works similar with 'continue@myLoopName'.

If expression+statement

- 'if' can evaluate to a value! Not possible in Java. You have to put the return value as the last statement in a {} block. In both the if and else block!
- Comparable with the ternary operator in Java: 'val num = if (mycondition) 40 else 60'. You can even write a full 'val num = if () {40} else {60}'.

When expression

- Is the Java 'switch' statement on steroids'
- 'when(condition) {
10 -> println
20 -> println
else -> println("doesnt match")
  '
- No need to add the 'break' statement. So no falling through at all
- More than 1 value possible in the match, e.g: '10,11' it will match on both those values. You can also use ranges. 
- And expressions: 'x + 50 -> println()'. And smart casting: 'when (something) { is String -> println() is BigDecimal -> println()}'
- And also as in the if statement, you can return a value in each branch of the 'when'. Best practice is to *not* return a different type of value in the branches.
- An empty when, looking like an if statement: 'when { 
i < 50 -> println 
i >=50 println 
else println
     }'. Which is much more concise than 3 if statement branches.

Try/Catch

- Reminder: no distinction between checked and non-time exceptions.
- Also the try/catch you can use as expression
- E.g: 'return try { Integer.parseInt(str) } catch (e: NullPointerException) {0} finally { println("hello")}'.
      Note the finally block doesn't return an Int. But that is fine, since the catch block does and the normal block does. So the finally block is not used in the evaluation of a try/catch expression.
- Use case for Nothing return type for functions: as return type for a method that only throws an exception, e.g the NotImplementedYet() exception.

Lambda expressions basics

- You can call them directly using the Kotlin library 'run' function: 'run{println("lambda being run")}'
- 'println(cars.minBy {car: Car -> car.builtYear}'). minBy is a Kotlin collection function built-in. Note you can move the lambda outside () when the last parameter. And if the only parameter, even leave out the (). Note also the 'Car' could be left out, Kotlin can infer the type.
- In this case the compiler can infer even more, so we can use 'it': - 'println(cars.minBy {it.builtYear}'). 
- You can also use a member reference: 'println(cars.minBy(Car::builtYear))'
- And toplevel function calls: 'run(::topLevelFun)'. Where: 'fun topLevelFun(): println("hello")'
- You can access local variables declared before the lambda in the lambda. In Java you can only access final variables in lambdas and anonymous classes.

Lambdas with receivers

- 'with' keyword: 'return with(StringBuilder()) { append("hello") toString() }'.  So can also be written as: 'return with(cars, {println(car)}'.
- So the 'with' turns the parameter you pass it into a receiver. That is how it is called. Inside the lambda you don't have to refer to the receiver object anymore. You can if you want with the 'this' keyword.
- 'apply' keyword is another receiver keyword: 'StringBuilder().apply() { append("hello") }.toString()'.
- A 'return' in an inlined lambda also returns the function it is in, a so called non-local return.
cars.forEach {
if (it.builtYear == 2019) {
                return
}
}
        println("this is not executed when at least 1 car has builtYear 2019")
- You can have a local-return but then you need a label:
cars.forEach returnBlock@ {
if (it.builtYear == 2019) {
                return@returnBlock
}
}
println("this is now also executed even when at least 1 car with builtYear 2019")
- You can also use the label to refer to nested 'apply's and 'with's:
"some text".apply sometext@ {
"another text".aplly {
println(toUpperCase()) // Only converts 'another text'
println(this@sometext.toLowerCase()) // converts the outer 'some text'
}
}
- 'also()' is similar to 'apply()', but you don't need to use 'this'.

Collections: Lists

- All read-only interfaces are covariant: be able to treat a class like its parent (superclass). E.g assign a List of 'Any's to a list of BigDecimals. See next Generics part for more details on this.
- For mutable collections you can't change the collection (of course), like adding elements. Seems usually these are also covariant (because they extend the (immutable) Collection interface which is covariant)
- 'listOf' creates an immutable list. But 'arrayListOf()' creates a mutable list!  See the type of the underlying Java class: first one is Arrays$ArrayList which is inmutable, the 2nd is ArrayList, which is mutable. I guess using 'mutableList' is more informative than using 'arrayListOf'.
- Handy: 'listOfNotNull(a,null,c)' creates an immutable list of only a and c.
- Convert list to array: 'listOf(*arrayOf("a", "b", "c"))'. Note the spread operator '*' there. But you can also do: 'array.toList()'.

Kotlin's added collections functions

- last(), asReversed()
- using the array notation to get an entry at given index: list[5] instead of list.get(5)
- list.getOrNull(): returns null when entry is null. Similar to Optional in Java.
- max(): entry with highest value
- zip(): creates Pair elements.  This is Kotlin Pair so a bit different from Java Pair. For me a bit unintuitive name that 'zip'.
- 'val combinedList = list1 + list2' // So you are really concatenating
- To combine & get no duplicates from 2 lists: 'val noDuplicatesList = list1.union(list2)'. Just like union in SQL.
- To get no duplicates in 1 list: list.distinct()
- Just like in Java, some functions return a new list, others work on the existing list. I expect you get at least an exception when trying the 2nd case on a immutable list.

Maps and destructuring declarations

- 'mapOf<Int, Car>(1 to Car("green", 2012), 2 to Car("yellow", 2013))'. The number is the key of the map. The <Int, Car> is redundant of course.
- And a mutable map: 'mutableMapOf()'.
- In both cases the underlying implementation is LinkedHashMap, so the iteration order is predictable and easy to convert to a Set for example.
- But you can force a hashmap: 'hashMapOf()'.
- Destructuring: 'val (firstValue, secondValue) = Pair(1, "one")'. Kind of "unpacking". And even: 'for ((key, value) : in mutableMap) { println(key) println(value)}'
- To be able to destructure a class, you need to use: component functions
- To add them to your own class use:
class Car(val color: String, val model: String, val year: Int) {
operator fun component1(): color
operator fun component2(): model
operator fun component3(): year
}
    Unclear if these have to have such fixed names componentN().
- Data classes get the component functions already generated (created) for free.

Sets

- To create an immutable set: 'setOf()'. To add an element to a set: 'set.plus(str)'. Duplicates added are ignored. To remove an element: 'set.minus(str)'. (you get a new set when using immutable sets)
- 'drop(nr)': drop the first 3 elements of the set
- Some functions work on the set itself, most return a new set with the function applied.

Other collections functions

- filter(lambda): returns a new instance of the collection
- 'val added10List = myIntsArray.map {it + 10}': adds 10 to each element in myIntsArray and puts that in new list (a java.util.ArrayList)
- You can chain them of course: 'myCarsMap.filter { it.value.builtYear == '2012' }.map { it.value.color = "yellow"}'
- To check for all elements matching a value, use the 'all(lambda)' function, which returns a boolean. And 'any(lambda)' for at least 1 matching.
- count(lambda)
- 'groupBy(lambda)'
- 'sortedBy(lambda)'
- 'toSortedMap()' to sort by key of the map

Sequences

- filter() creates a whole new copy of the collection. Which can be huge. So to prevent too much memory usage, you can use sequences. Similar to Java streams, but re-invented by Kotlin because Android didn't support Java 8 yet back then. To make a map or list a sequence use: 'asSequence().filter()....'
- 2nd case where you need to put a ';': when you have a lambda predicate which you want to put on 1 line, e.g: '.filter { println("$it"); it[0] == 'yellow'}'
- When you have a sequence, the filter(), map() etc return also a sequence, not the list/collection, because they are intermediate operations. So you need a terminal operation in the chain of calls. E.g: 'toList()'. But for example 'find()' stops at the first match. Usually you want to 'map()' late(st) in the chain of calls. Just like Java streams.

Generics

- You always have to specify the generic type in Kotlin in e.g List<String>. So 'List' is not allowed. Also not for 'myList is List'. There you can use: 'myList is List<*>'. That is called the star projection.
- fun <T> myFunction(collection: List<T>) {}
- Or even as extension function: fun <T> List<T>.myFunction() { println(......) }
- fun <T: Number> myFunction(collection: List<T>) to limit to certain subtypes (you specified the Number as upperbound)
- Multiple upperbounds: fun<T> doIt(item1: T, item2: T) where T: CharSequence, T: Apppendable {}
- Note that T accepts the nullable type too!  By default, if no upperbound specified, it is 'Any?'. So nullable! I call this inconsistent with the "try to not support null" goal :)
- You can make it not accept nullable type by specifiying of course: 'T: Any'.
- Just like in Java, at runtime the generic type is erased (not available anymore)
- In Java you can't do 'list instanceof List<String>'. But in Kotlin you can do: 'list is List<String>'. But it can't for a list of type Any.

Reified parameters (generics related)

- Reification: prevents the type from being erased at runtime
- To do that: make the function 'inline' and 'reified T': 
inline fun <reified T> getElements(list: List<Any>): List<T> {
for (element in list) {
if (element is T)  <----------- now compiles!
}
}
   Then call it: val myListOfAny: List<Any> = listOf("string", 1, 20.0f); getElements<BigDecimal(myListOfAny)

Covariance (generics)

- MutableList<Short> is not automatically the same as ImmutableList<Short> at compile time in case of for example function parameters. Immutable collections are covariant, so then the "subtyping" is fine. But if you define the same parameter collection as mutable, the "casting" can't take place (compile error).
- So MutableList is invariant and wants the exact type (you see this also by the 'out' not present at that class's definition; more on that below)
- And so List has the 'out' keyword, so is covariant. So subtyping is preserved. And since it is an immutable interface/class, nothing can change the list anyway. Still some functions have 'T' as parameter while 'out' is there! Like 'contains()'. @UnsafeVariance is then used for such a (in) parameter, telling compiler that the list won't be changed.
- Note that: List is really a class. But List<String> is a type! E.g: Short is a subclass of Number. List<Number> is supertype of List<Short>
A nullable type is a supertype of non-nullable types (a wider type).
- So List<Short> is not by default the same as List<Number>, even though the 2nd is a super-type(!) of the first.
- Mutable collections interface = not covariant. Collections interface = covariant.
- So if you want the sub-typing(!) to be preserved, you have to prefix the type with the 'out' keyword. E.g: 'class Garden<out T: Flower>
- That keyword implies you can use that class only in "out position".  By default parameters are of 'in' position. And usually the out position is the return type. By specifying the T as 'out' in the Garden above, you can *only* use it as return type!
So 'fun getFlower(nr: Int): T {}' is fine.  But 'fun waterFlower(flower: T)' won't compile!
  Because otherwise you can pass in a different type than the original Garden was created with (e.g: rose vs daffodil); the compiler can't tell.
Note that Garden is a type, not a class, due to that generics part.
So in/out is protecting the class from invalid updating.
- So if you have a covariant class ('out' keyword), subtyping is preserved
- Constructors don't have in/out parameters. So you can always pass in a covariant class as constructor parameter, since it is at construction type.
But if you specify a var parameter: can't be covariant class as input, because again you could pass in the wrong type, because it generates also 
a setter. But a 'private var' is ok again because nobody outside the class can access it.
- Covariant type: subtyping preserved so you can pass an instance of the type or the subtype as for example a parameter. But you can't change the instance (thus when having the 'out' keyword).
- In Java it would be like this: List<? extends Car>,  where you now accept anything that extends Car. This you see in Java in method declarations only. (a variable of that type you can't anything to)

Contravariance

- Is the opposite of Covariance. With Covariance you are preserving the subtyping (incorrectly said(?): accept all of its sub"classes". Is it classes or types or both?). With covariance you start at a subclass and you want to accept instances of that subclass, or any of its superclasses.
- So you use the keyword 'in': interface FlowerStuff<in T> { fun doSomething(flower: T) }. So if T is a direct or indirect superclass of a class, then the class will match T. And you can't have in that case the T as return type, so 'fun getFLower(): T' does not compile. Again you won't be guaranteed to get the type you want if that would have been allowed. Or said differently: not all flowers are a rose. But all roses (subclass) are a flower (superclass).
- Again (depends on where you looking at from the inheritance tree; some say it is "flipping the inheritance tree for a class": 
Covariance: you want to accept a class and all its subclasses   ("looking down the inheritance tree")
Contravariance: you want to accept a class and all its superclasses ("looking up the inheritance tree")
You are widening a generic type to include a class and its subclasses (covariance), or a class and its subclasses (contravariance).
"declaration site variance", so used during declaring an interface. In Java you only have "use-site variance".
- In Java it would be: List<? super Car>: accepts Car and any of its superclasses. This you see in Java in method declarations only. (a variable of that type you can't read from)

Use-site variance

- Generic types are invariant. So even if 'Ford: Car', you can't call 'Car copyCars(source: MutableList<Car>, dest: MutableList<Car>)' with 'copyCars(mutableListof(Ford(), Ford()), mutableListOf(Ford())'. You have to change the Car type in the copyCars to 'T'.
- And to have 'copyCars(fords, cars)' compile, you have to add 'out' (covariance!) to the 'source: MutableList<out T>'. Because that parameter is not changed in the code, so we can do that. This is called use-site covariance, because we didn't add it to the Car class or its subclass like Ford. You can also add 'in' to the dest parameter, but is not needed in this case.
- Also called type-projection.
- In Java declaration-site variance doesn't exist, so you'd have to add the in/out to each method of the class. But so in Kotlin you can do it at class level, i.e declaration-site variance.

I/O

- Just extension functions added to the java io classes like java.io.File.
- var lines = File("myfile.txt").reader.readLines()
- Or to have the reader resource close automatically at the end: val lines = File("myfile.txt").reader().use {it.readText}
- reader().forEachLine() to read per line
- instead of try with resources in Java, use the 'use()' function, that closes the resource always, even if exception.
- File(".").walkTopDown() for file tree traversal

Java interoperability

Calling Java from kotlin:

- Primitive arrays in the Java method as parameter like int[]: you have to use toIntArray() or intArrayOf().
- Nullability: @Nullable @NotNull are Jetbrains IntelliJ(!) hints that can be used. So they are NOT Kotlin annotations, but you give in IntelliJ the Kotlin compiler these hints. Quite ugly in your code base of course. It might also recognize Lombok's or javax's.
But, you only get an IllegalArgumentException at runtime, so compiler won't complain if you assign null to a field.
E.g: Java: 'void setColor(@NotNull color) {}' and then in Kotlin: 'var car: Car = null' compiles fine, but IAE at runtime because the Kotlin compiler generates the not-null check.
Default when none of these 2 annotations: it is nullable type for Kotlin
- You can directly access the field of the Java class like car.color when the Java class has matching getter + setter methods. Or make the field public.
- Exceptions from java: no need to add a throws to the Kotlin function
- varargs in Java: you can't pass a Kotlin array, you'll have to spread (unpack, *) the array
- void from java method: in Kotlin you'll see Unit
- Use Java Object methods that are not in Kotlin's Any, like the Object.notify() method: (car.anObject as java.lang.Object).notify().
- Static fields and methods in Java: are converted to companion objects, so just like this: Car.myStaticfield and Car.myStaticMethod().
- Single Abstract Methods in Java, e.g the Runnable interface which only has one method run() which you have to implement: you can pass them a Kotlin lambda (just like the Java lambda)

Calling Kotlin from Java:

- call toplevel Kotlin functions: Kotlin compiler creates a class based on the .kt filename. So: 'Car.kt' then you invoke 'Car.kt.myMethod()'.
But you can change that generated classname using the @file:JvmName("myclassname")
- Extension functions you can call from java by prefixing the classname too
- getX() and setX() getters setters are available in Java for 'var' fields
- To be able to directly access fields in Kotlin classes, you have to add the @JvmField on the field in the Kotlin class. Has a few use constrictions
- Companion object access: usually: Car.Companion.myMethod(). If you want to not have to use the Companion part, annotate the companion method with
@JvmStatic fun myMethod() {}
- 'object MySingleton() { fun myMethod() }' will be callable from java as: MySingleton.INSTANCE.myMethod(). Also here you can annotate the method with @JvmStatic to avoid that INSTANCE part.
- For a 'const val' in Kotlin, you don't need that @JvmStatic, it is directly accessible: MySingleton.myConstant
- When passing null to a function in Kotlin that is expecting a non-null type, at runtime the Kotlin will check for null (generated by the Kotlin compiler)
- To have Kotlin functions tell Java code an exception can be thrown, annotate method with: @Throws(IOException::class)
- When default parameter values for parameters in Kotin functions: to have Kotlin generate all combinations for those optional parameters use: @JvmOverloads

Not covered

  • coroutines
  • DSL
  • Spring boot integration, annotations how?
  • Lombok annotations work with Kotlin?
  • How to do unittests
  • How to do integrationtests

Considerations

  • Why are "methods" called functions in Kotlin? Functions sounds less Object Oriented...

Wednesday, September 14, 2022

Connection refused Spring Boot application to MySql in Docker solution

Introduction

The Spring Boot Java application was initially using the H2 embedded database, first the in-memory version (losing all data each run), then the file based version. The database structure is created using Liquibase.

Then the requirement was to have the Spring Boot Java application still not dockerized (just started with mvn spring-boot:run), but have it connect to a MySql 8 database than runs inside a Docker container. Most examples you can find have also the Spring Boot application in a Docker container.
The official Spring documentation only shortly mentions the possibility of running MySql in Docker, but no more details on how to do that: https://spring.io/guides/gs/accessing-data-mysql/

This is how the setup should be working:


I set up that docker container using the Windows 10 WSL 1 shell within IntelliJ (note WSL 2 is now recommended):

docker pull mysql/mysql-server:8.0
docker run --name recipesmysql8 -d mysql/mysql-server:8.0

After changing the One Time Password (OTP) for root, creating the database user, creating a 'recipes' database, this is how docker ps looked:

CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                PORTS                       NAMES
3a507fc66690        mysql/mysql-server:8.0   "/entrypoint.sh mysq…"   6 days ago          Up 6 days (healthy)   3306/tcp, 33060-33061/tcp   recipesmysql8

And MySql is indeed running: docker logs recipesmysql8 shows:

2022-08-31T19:02:05.722742Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 1
2022-08-31T19:02:05.755538Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-31T19:02:06.104813Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-31T19:02:06.416522Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-31T19:02:06.416574Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-31T19:02:06.443572Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-31T19:02:06.443740Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MyS
QL Community Server - GPL.


So all running fine, ports seemed fine. The Spring Boot application configuration for JDBC is this:

spring.datasource.url=jdbc:mysql://localhost:3306/recipes
spring.datasource.username=recipes
spring.datasource.password=mypasswd
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver

But then the mvn spring-boot:run gave this relatively vague error:

com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    ...
   Caused by: java.net.ConnectException: Connection refused: no further information

So unable to connect, but why? It does not seem an incorrect username/password combination, then I would expect some unauthorized/not authenticated type of message.
Then I found this handy command determining what the real IP should be of the machine that the MySql docker runs in:

docker inspect -f '{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq)

I found my MySql container at: /recipesmysql8 - 172.17.0.2
Same result, shorter answer: docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' recipesmysql8
Or even just without the filtering: docker inspect recipesmysql8

So the JDBC configuration I changed to:

spring.datasource.url=jdbc:mysql://172.17.0.2:3306/recipes
spring.datasource.username=recipes
spring.datasource.password=mypasswd
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver

But then the mvn spring-boot:run gave this timeout error:

Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    ...
    Caused by: java.net.ConnectException: Connection timed out: no further information

So a similar error message, but this time a connection timeout. Is maybe the problem connecting from within IntelliJ where I execute the mvn spring-boot:run command, to Docker that runs in WSL?

I tried by disabling the antivirus software and local firewall and WIFI, but those didn't work either, at most a different error message like:

Caused by: java.net.NoRouteToHostException: No route to host: no further information

And even with the mysql shell client does it give an error (bit less clear):

mysql -h 172.17.0.2 -P 3306 --protocol=tcp -u root -p
Enter password:
ERROR 2003 (HY000): Can't connect to MySQL server on '172.17.0.2' (11)

Solution

Then I stopped my container, started a new one with explicitly ports-mapping specified:

docker run --name recipesmysql8v2 -p 3306:3306 -d mysql/mysql-server:8.0

docker ps output:
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                   PORTS
    NAMES
819333cf718e        mysql/mysql-server:8.0   "/entrypoint.sh mysq…"   2 minutes ago       Up 2 minutes (healthy)   0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060-33061/tc
p   recipesmysql8v2

Then a different error message for mysql -h 127.0.0.1 -P 3306 --protocol=tcp -u recipes -p appeared:

ERROR 1130 (HY000): Host '172.17.0.1' is not allowed to connect to this MySQL server

That looked promising.

Note: due to having a new container, I had to reset the OTP for user 'root' again of course. You can find that OTP by issuing docker logs recipesmysqlv2 | grep GENERATED.
Then issue these commands:

docker exec -it your_container_name_or_id bash
mysql -u root -p
Enter password: <enter the one found with the above grep shell command>
ALTER USER 'root'@'localhost' IDENTIFIED BY 'your secret password';

Create the Spring Boot application database again: create database recipes;
Add the Spring Boot application user again: create user 'recipes'@'%' identified by 'L7z$11Oeylh4';

Give that user only the necessary permissions:
grant create, select, insert, delete, update on recipes.* to 'recipes'@'%';

After that, these entries should be in the mysql.user table:
mysql> SELECT host, user FROM mysql.user;
+-----------+------------------+
| host      | user             |
+-----------+------------------+
| %         | recipes          |
| localhost | healthchecker    |
| localhost | mysql.infoschema |
| localhost | mysql.session    |
| localhost | mysql.sys        |
| localhost | root             |
+-----------+------------------+
6 rows in set (0.00 sec)


Note the recipes user entry with host '%', which allows access from any host, which you probably want to make more secure in a production environment. See https://downloads.mysql.com/docs/mysql-secure-deployment-guide-8.0-en.pdf for tips.

Now you should be able to connect from the mysql client prompt in several ways:

mysql -h 127.0.0.1 -P 3306 --protocol=tcp -u recipes -p
mysql -h localhost -P 3306 --protocol=tcp -u recipes -p
mysql -h 0.0.0.0 -P 3306 --protocol=tcp -u recipes -p

Note that the 172.17.0.2 still does not connect! For that to work you'll have to probably add that host to the above mysql.user table (not tried to see if that works).

Instead of mysql client you can also use the standard *nix command telnet to see if at least the port is reachable:

telnet 127.0.0.1 3306
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.


(I don't remember what it showed when the initial issue with the docker container named 'recipesmysql8' was there)

And after starting the Spring Boot application: connection worked and tables and indexes were created!

Note: only from within the docker image can you connect like this: docker exec -it recipesmysql8v2 bash and execute mysql -u root -p after having issued ALTER USER 'root'@'localhost' IDENTIFIED BY 'nR128^n8f3kx';
Because running mysql -h 127.0.0.1 -P 3306 --protocol=tcp -u root -p from within the commandline WSL still gives: ERROR 1045 (28000): Access denied for user 'root'@'172.17.0.1' (using password: YES)

Probable cause: when you only have MySql in a Docker container, it is not in the same "network" as Docker containers, so the port 3306 is not reachable from outside Docker runtime. So you will have to tell Docker how you want to expose the port(s). Another way to solve this is to have the Spring Boot application also as a Docker application; and potentially configure it via docker-compose, see here for an example on how to do this: https://www.javainuse.com/devOps/docker/docker-mysql.



Wednesday, April 6, 2022

Configuring MySQL test-containers in your Spring Boot Java Integration Tests

Introduction

In your Integration Tests (IT) you often try to use the H2 in-memory database, to improve the speed of your integration tests. But on the other hand you want to mimic the production database as much as possible in your integration-tests.

Setting H2 in MySQL database compatibility mode tries to emulate MySQL as much as possible, but only a small subset of the differences are implemented. What for example not works correctly in H2 for JSON fields is that it escapes strings with "". For that reason you usually want to switch to for example starting a Docker database testcontainer in your IT tests, which uses a real MySQL database. With the Java-specific version in https://github.com/testcontainers/testcontainers-java.


Configuration

There are several good-to-know tips when configuring the testcontainers in your ITs.

  1. The simplest configuration is using a datasource URL in the Spring Boot properties file. This has the disadvantage that whatever database name you specify, the testcontainers library still creates a DB named 'test'. So below it will be named 'integration_test_db' you'd think, but it is still named 'test' when the IT runs:

    spring.datasource.url=jdbc:tc:mysql:5.7.32:///integration_test_db?sessionVariables=sql_mode='STRICT_TRANS_TABLES'&TC_MY_CNF=mysql&TC_INITSCRIPT=mysql/init_mysql_integration_tests.sql

    To be able to do everything on the started database, including giving it the name you want, use this URL (or see below the Java version). Notice the user 'root' in the URL:
    spring.datasource.url=jdbc:tc:mysql:5.7.32:///integration_test_db?user=root&password=&sessionVariables=sql_mode='STRICT_TRANS_TABLES'&TC_MY_CNF=mysql&TC_INITSCRIPT=mysql/init_mysql_integration_tests.sql

    Via: https://github.com/testcontainers/testcontainers-java/issues/932

  2. To initialize your database, specify the script via this extra datasource URL variable:

    TC_INITSCRIPT=mysql/init_mysql_integration_tests.sql

    Note the default directory it looks in is ..../resources for the scripts. So the full path is ..../resources/mysql/

  3. An example to prevent GROUP BY error from strict mode, add this in your TC_INITSCRIPT:

    SET GLOBAL sql_mode = 'STRICT_TRANS_TABLES';
    SET SESSION sql_mode = 'STRICT_TRANS_TABLES';


  4. To configure it in the IT Java class itself:

    @RunWith(SpringRunner.class)
    @SpringBootTest(classes = { SomeClassA.class, SomeClassB.class, ApplicationConfiguration.class}, webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
    @TestPropertySource(locations = {
            "classpath:/application-test-mysql.properties" })
    @ContextConfiguration(initializers = {ThisITClass.Initializer.class})

    static class Initializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
            public void initialize(ConfigurableApplicationContext configurableApplicationContext) {
                TestPropertyValues.of(
                        "spring.datasource.url=" + mySQLContainer.getJdbcUrl(),
                        "spring.datasource.username=" + mySQLContainer.getUsername(),
                        "spring.datasource.password=" + mySQLContainer.getPassword()
                ).applyTo(configurableApplicationContext.getEnvironment());
            }
        }

    @ClassRule
    public static MySQLContainer mySQLContainer = new MySQLContainer<>("mysql:5.7.31")
                .withUsername("root") // So now you can do a GRANT too for example
                .withPassword("") // Only possible for user 'root'
                .withEnv("MYSQL_ROOT_HOST", "%")
                .withDatabaseName("integration_test_db") // So this name will now be used, not 'test'
                .withInitScript("mysql/init_mysql_integration_tests.sql")

                ;

  5. The MySQL docker image used in the tests is retrieved from DockerHub https://hub.docker.com/_/mysql

Sunday, December 26, 2021

Sentry in Spring Boot application reports uncaught exception while @RestControllerAdvice is handling exception

Introduction

Even when you have a @RestControllerAdvice configured and you know it gets invoked, it can be that Sentry.io's monitoring library still reports it as an (uncaught) exception. Reason for this is that when you configure Sentry as specified without setting its order in the exception-handler-resolver chain, it's by default set as lowest, so invoked as first in the chain. 

Solution

Thus to fix this, set it to a higher order number. Safest seems to set it to Integer.MAX_VALUE, that way it will always be invoked as the latest.

The most recent Sentry integration examples can be found here for Spring and here for Spring Boot. Maybe you want to consider logging via Logback or Log4J too, see the related Sentry integration examples on the same page.

Another solution is overriding the getOrder() method for older Sentry versions. Below is an example, using the suggesting configuration from this old obsolete Sentry page.

@Configuration
@Slf4j
public class SentryConfig {
    @Bean
    public HandlerExceptionResolver sentryExceptionResolver() {

        return new SentryExceptionResolver() {
            @Override
            public ModelAndView resolveException(HttpServletRequest request,
                    HttpServletResponse response,
                    Object handler,
                    Exception ex) {
                log.info("Sentry resolving this exception: ", ex);
                // You could skip exceptions that are considered client-side errors and return null in that case so it is considered taken care of.
                return super.resolveException(request, response, handler, ex);
            }

            @Override
            public int getOrder() {
                // Ensure other resolver(s) can run first, otherwise this handler is invoked as first and thus reporting an issue for Sentry
                return Integer.MAX_VALUE;
            }
        };
    }
    @Bean
    public ServletContextInitializer sentryServletContextInitializer() {
        return new io.sentry.spring.SentryServletContextInitializer();
    }
}


Similar solutions mentioned here: https://stackoverflow.com/questions/48401974/how-to-have-my-own-error-handlers-before-sentry-in-a-spring-application


Thursday, November 19, 2020

Spring @Scheduled using DynamoDB AWS X-Ray throws SegmentNotFoundException: failed to begin subsegment

Introduction

AWS X-Ray is designed to automatically intercept incoming web requests, see at this introduction.  And also here

But when you start your own thread (either via a Runnable or plain new Thread() or @Scheduled), X-Ray cannot initialise itself: there are no web requests to intercept for it. Then it throws an exception like this: 

Suppressing AWS X-Ray context missing exception (SegmentNotFoundException): Failed to begin subsegment named 'AmazonDynamoDBv2': segment cannot be found.

In the above example the distributed DynamoDB Lock Client was used, which uses DDB in its implementation for acquiring and releasing a lock.

Regular web requests were not throwing this X-Ray exception.

Investigation

Amazon explains that crucial bit of knowledge that web requests are "automagically" setting up the X-Ray recorder a bit here

But that was not fully explaining it with an example. E.g only adding 

AWSXRay.beginSubsegment("AmazonDynamoDBv2") 

(and ending it) but that didn't fix it. That then gave this exception:

Suppressing AWS X-Ray context missing exception (SubsegmentNotFoundException): Failed to end subsegment: subsegment cannot be found.

My suspicion here is that the lock-client already closed the exactly same named subsegment "AmazonDynamoDBv2".
Also, I'm not creating any worker thread myself, Spring is doing it for me.

Note that you at least can avoid exceptions be thrown by setting environment variable AWS_XRAY_CONTEXT_MISSING   to LOG_ERROR. That will only log the above exception.

Solution

Creating a 'parent' segment and setting the trace entity and the subsegment did the job:

Entity parentSegment = AWSXRay.beginSegment("beginSegmentForSomeScheduledTask");
AWSXRay.getGlobalRecorder().setTraceEntity(parentSegment);
AWSXRay.beginSubsegment("AmazonDynamoDBv2");

I did test only creating the subsegment and only creating and setting the parentSegment. But those raised the exception again.
I did not further investigate whether the "AmazonDynamoDBv2" name of the subsegment is essential.
And of course the matching closeSegment() and closeSubsegment() calls of course; not the reverse order: close the last one begun as first.

This thread pointed me in the right direction. This was a next option I would have tried next: setting up my own filter to run it earlier in the (filter) chain; though the @Scheduled task of course does not have a filter. Another workaround would have been to put the X-Ray logging at a very high level:

logging.level.com.amazonaws.xray = SEVERE

Also helped in explaining is this X-Ray reported issue.

Update: additionally, the error was also due to DynamoDB Lock Client! I did not specify withCreateHeartbeatBackgroundThread() when creating the lock client, but the exception of X-Ray showed that it was trying to send a heartbeat. After explicitly setting withCreateHeartbeatBackgroundThread(false) the exception (and error) regarding segment cannot found was fully fixed.




Sunday, April 5, 2020

PACT provider test with Serverless Java 11 Lambda and JUnit 5 (Jupiter)

Introduction

This blogpost shows how to create a Pact provider test using:
  1. Pact 4.0.0
  2. AWS Serverless Lambda
  3. Java 11
  4. JUnit 5 (Jupiter)

Example Provider Test

Below is the example test class. Note how a mock service is used to simulate the endpoint defined (also) in serverless.yml.

package com.ttlnews.pact.tests.provider;

import au.com.dius.pact.provider.junit.Provider;
import au.com.dius.pact.provider.junit.State;
import au.com.dius.pact.provider.junit.loader.PactBroker;
import au.com.dius.pact.provider.junit.loader.PactBrokerAuth;
import au.com.dius.pact.provider.junit5.HttpTestTarget;
import au.com.dius.pact.provider.junit5.PactVerificationContext;
import au.com.dius.pact.provider.junit5.PactVerificationInvocationContextProvider;
import com.amazonaws.services.lambda.runtime.Context;
import lombok.extern.slf4j.Slf4j;
import net.jcip.annotations.NotThreadSafe;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.TestTemplate;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockserver.integration.ClientAndServer;

import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.mockserver.integration.ClientAndServer.startClientAndServer;
import static org.mockserver.matchers.Times.exactly;
import static org.mockserver.model.Header.header;
import static org.mockserver.model.HttpRequest.request;
import static org.mockserver.model.HttpResponse.response;
import static org.mockserver.model.JsonBody.json;

/**
 * Java 11 serverless provider Pact contract provider implementation using Pact with Junit5 (Jupiter) for provider SERVICE_A.
 * 
 * Dependencies needed:
 *
     <dependency>
     <groupId>au.com.dius</groupId>
     <artifactId>pact-jvm-consumer-junit5</artifactId>
     <version>4.0.0</version>
     <scope>test</scope>
     </dependency>
    
     <dependency>
     <groupId>au.com.dius</groupId>
     <artifactId>pact-jvm-provider-junit5</artifactId>
     <version>4.0.0</version>
     <scope>test</scope>
     </dependency>
 *
 * 
 */
@Provider(SERVICE_A)
// Below environment variables must be set when running this test
@PactBroker(host = "${PACT_HOST}",
        authentication = @PactBrokerAuth(username = "${PACT_HOST_USERNAME}", password = "${PACT_HOST_PASSWD}"))
@NotThreadSafe // Pact contract tests can't seem to handle methods running in parallel, so prevent maven failsafe/surefire plugin to run Pact tests in parallel
@Slf4j
public class ServiceAProviderTest {

    private static final String SERVICE_A = "service-A";
    
    private Context lambdaContext;
    private SomeRepository someRepository;

    private ClientAndServer mockClientAndServer;

    @BeforeEach
    void before(PactVerificationContext context) {

        lambdaContext = mock(Context.class);
        someRepository = new InMemoryMockedRepo();

        mockClientAndServer = startClientAndServer(8888);

        context.setTarget(new HttpTestTarget("127.0.0.1", 8888));

    }

    @AfterEach
    void stopServer() {
        mockClientAndServer.stop();
    }

    // This triggers the defined contract test(s) at the host PACT_HOST
    @TestTemplate
    @ExtendWith(PactVerificationInvocationContextProvider.class)
    void pactVerificationTestTemplate(PactVerificationContext context) {
        context.verifyInteraction();
    }

    @State("Create a new resource A")
    public void shouldCreateResourceA() {

        log.info("Set up state");

        // Given
        // Maybe some more mocking of 'lambdaContext' needed, depending on your case
        
        // Lambda being tested
        CreateResourceALambda createResourceALambda = new CreateResourceA(someRepository);

        CreateResourceALambdaRequest createResourceALambdaRequest = CreateResourceALambdaRequest.builder()
                .someValue("25")
                .build();

        // When
        final LambdaResult lambdaResult = createResourceALambda.executeRequest(createResourceALambdaRequest, lambdaContext);
        final String body = lambdaResult.getBody();
        assertNotNull(body);

        // Prepare the mockserver to return what the lambda returns when it is invoked (set up above in the 'body' variable) 
        // Of course the path to this CreateResourceALambda is defined in the serverless.yml, but that we can't access now. So need to repeat that
        // endpoint here.
        mockClientAndServer.when(
                request()
                        .withMethod("POST")
                        .withPath("/resources/")
                        .withHeaders(

                                header("x-request-trace-id"),  // Any value is fine
                                header("Authorization"),  // Any value is fine
                                header("Content-Type", "application/json")

                        )
                        .withBody(json("{someValue: '25'}")) // Indeed need to use this strange JSON format
                ,
                exactly(1))
                .respond(
                        // Response as defined by the matching Pact consumer test; in this case found at host PACT_HOST
                        response()
                                .withStatusCode(201)
                                .withHeaders(
                                        header("Content-type", "application/json; charset=utf-8"),
                                        header("Authorization") // Any value is fine
                                )
                                .withBody(body)
                );
    }
}
 
Migrating from your Pact tests from JUnit4 to Junit5 can be found here.
 
Note to self: use https://www.opinionatedgeek.com/codecs/htmlencoder to encode code, open the HTML view, and then wrap the code in <pre> open en close tag.

Wednesday, October 2, 2019

Snyk .snyk ignore file with multiple entries format

"Snyk enables you to find, and more importantly fix known vulnerabilities in your open source."


Some vulnerabilities you'd want to ignore, because for example they are (for you) either false positives, or there is no workaround for them (yet). You can specify them via the so-called CLI snyk ignore command, or in the UI manually, or via a .snyk ignore file. An introduction focused on NPM can be found here.
The format with an example for one entry is specified here: https://support.snyk.io/hc/en-us/articles/360000923498-How-can-I-ignore-a-vulnerability-

But what is the format for multiple entries? Most of the time with yaml (-like) formats you might expect a list, so each entry should be prefixed with a dash "-". But that is not the case here. The correct format for specifying multiple entries in the .snyk ignore file is:
version: v1.5.0
ignore:
  'SNYK-JAVA-xxxx-123456':
    - '* > com.abc:def':
      reason: 'Fix in progress'
      expires: '2020-01-05T08:00.00.000Z'
  'SNYK-JAVA-xxxx-678900':
    - '* > org.abc:ghi':
      reason: 'Windows issue, systems run on Linux'
      expires: '2020-02-06T09:00.00.000Z'
 
Note each block after the 'ignore' line is indented with two spaces.

Saturday, July 27, 2019

PACT Consumer Driven Contract Testing: how to allow any body in the response


Consumer Driven Contract testing is a way to ensure that services (such as an API provider and a client) can communicate with each other. Without contract testing, the only way to know that services can communicate is by using expensive and brittle integration tests.
PACT is a contract testing tool.  


For response matching you want to be as loose as possible with the matching for the response (will_respond_with(...)) though. This stops the tests being brittle on the provider side.
 Most of the time you don't care about the exact values of the (JSON or XML) response, but you do care about the types of the values, e.g a string or a number.
In that case you'll be using 'type matching'.

Sadly for JVM matchers there's no such handy method that with one invocation makes sure only types are validated, as is for example with Ruby/Groovy/Javascript/Node: Pact::SomethingLike.
Not much documentation on the Java/JVM matchers can be found at the official Pact site itself. Mostly the examples are for Ruby/Groovy/Javascript/Node.

For  the Java and Virtual Machine integration you'd use pact-jvm with matchers like .stringType() for the body using the PACT DSL or this lambda extension.

But hard to find was how to specify allowing either an empty body in the response or any data in the body in the response, while the consumer does not require a body at all (or in other words: is fine with an empty body or any fields in it).
Solution: for that you need to completely omit the .body() in the consumer contract definition.
If you specify a .body(new PactDslJsonBody()), the contract generated matcher will specify "body" : {}, and therefor requiring an empty body. And if then the provider (test) generates one or more fields in the response, you'll see this as the message in the failing test:


Expected an empty Map but received Map(..... fields added by provider ...)


So full example which accepts an empty body or a body with elements in it:

.consumer("Some Consumer")
.hasPactWith("Some Provider")
.given("a certain state on the provider")
    .uponReceiving("a request for something")
        .path("/hello")
        .method("POST")
.toPact()


Tuesday, July 9, 2019

Maven Failsafe plugin build error: SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?


When you get this exception during running mvn clean verify, which executes both the Surefire and Failsafe plugin (for which the last one uses the Surefire plugin!):

SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?

it can mean several things. For example as the message says maybe somewhere in your integration tests you call System.exit, which you shouldn't do. That one is easy to find. Also this Stackoverflow post covers that plus some other options, like shortage of memory.

It has also higher chance on happening when using the docker-maven image and 3.x-openjdk-8 or 3.y-openjdk-10.

But if that doesn't fix it, what then?  Then due to this issue you probably have to add this property in your pom.xml in the configuration of both the Surefire and the Failsafe plugin:

<usesystemclassloader>false</usesystemclassloader>

Thus not only for the Failsafe plugin, for both! :)

PS: adding that configuration property also is a workaround for https://issues.apache.org/jira/browse/SUREFIRE-1588, also reported here: https://stackoverflow.com/questions/50661648/spring-boot-fails-to-run-maven-surefire-plugin-classnotfoundexception-org-apache/50661649#50661649

Sunday, March 31, 2019

ReadProvisionedThroughputExceeded using Kinesis

ReadProvisionedThroughputExceeded

When using Kinesis it is possible you'll get an ReadProvisionedThroughputExceeded when not carefully designing the shards consumption of messages.

That exception indicates "The number of GetRecords calls throttled for the stream over the specified time period".

A cause for this can be you have to many consumer applications at the same time issue GetRecord calls to the same shard.

One recommended solution is to increase the number of shards. But that has a price-tag attached to it. Another solution is process (batches) of records in parallel by spinning off threads within the processRecords() method in KCL. But that will require careful orchestration of threads to keep checkpointing correctly. An interesting analysis of why using KCL and especially version 2 can be found here.

For a more detailed analysis, see the longer blogpost here.

ProvisionedThroughputExceededException using KCL (Kinesis Client Library)

ProvisionedThroughputExceededException

When using the Amazon Kinesis Client Library (KCL) one would not expect the table created by KCL itself to keep track of its streams, shards and checkpoints to ever be under-configured.
But it can be! Then you get the ProvisionedThroughputExceededException

Normally it will recover by retrying itself. But if it can't, other solutions exist:
  1. Change the read/write provisioned for that table to 'on-demand'. 
  2. Make sure your hashkeys are evenly distributed, as mentioned here not-evenly distributed hash-keys.
  3. Modify how many consumers (applications) are reading from the same shard(s) at about the same time, either by changing that design or increase the number of shards.
For a more detailed analysis, see the longer blogpost here

Wednesday, March 20, 2019

Lessons learned using AWS Kinesis to process business events and commands (messages)

Introduction

When using AWS Kinesis as the means of communicating business events, several challenges arise. For the basic key concepts of Kinesis see here

Of course the "normal" use case to use Kinesis is described as here in the section "When should I use Amazon Kinesis Data Streams, and when should I use Amazon SQS?" Another comparison between Kinesis and SQS can be found here.

Especially built for Kafka-like stream-processing of millions of events, where usually the producers (e.g think IoT devices) are much faster than the consumer applications.
Usually business events are not generated millions per minute or second. Events from IoT devices are usually not considered business events.
Any next time I would not recommend using Kinesis as a transport mechanism for business events; the main reason for me is its intended use-case is not matching that type of use. Plus the many technical challenges I had to solve, for most of which any "regular" messaging tool like RabbitMQ would suffice.
Additionally, usually one wants business events to arrive almost instantly - or at least as fast as possible. Kinesis is fast but e.g RabbitMQ is often sufficient as transport for business events.
And yes RabbitMQ is now also available as a managed solution - hosted at AWS but managed not by AWS itself. For example with CloudAMQP.

In the below discussion a message can be an event or a command (though most often you'd want the commands to be synchronous, since they should only be delivered to one service anyway)

High level Kinesis architecture overview

Below is a high level overview of Kinesis' architecture: 


There can be multiple streams, and within each stream are one or more shards. Messages in a shard are guaranteed to be delivered at the consumer in the order they were published onto the shard.
The partition-key used while publishing determines into what shard the message is published. Messages with the same partition key get always published into the same shard.

A design decision one has to make is: do I make one stream were all services publish their messages, or do I want a stream per XYZ? Where XYZ could be for example a Bounded Context.
No good reason was found to split into multiple streams, so I decided to go for one Kinesis stream all services publish on.

Lessons learned

When using Kinesis to handle business events between micro-services several challenges were to overcome. Below the lessons learned are described.

More than five different applications processing the same shard at the same time

Kinesis limits documentation  states that each shard within a stream supports up to five read transactions per second. And if you need more than that, it is recommended to increase the number of shards.
If you don't comply to that limit, you'll see ReadProvisionedThroughputExceeded or ProvisionedThroughputExceededException exceptions appear in your logs. This can easily be reached if you have large documents (e.g 1MB) in your DynamoDB too; even with the AWS Console you then can't even view such a large document in your browser!
Increasing the number of shards will spread the published messages across more shards and thus reduce the number of read transactions per second per shard.
BUT: in a microservices architecture you can easily have 10 or more services. And all these services need to process all messages on all shards.
So with for example 10 services, each of those services (container or lambda) will need to poll each shard regularly in some form. And the more services, the more chance that more than 5 services poll any given shard per second... Causing the above ReadProvisionedThroughputExceeded exception.
Thus: increasing the number of shards won't help, because still each service will need to poll each shard.

One workaround for this could be:
  • have each service (container, lambda) publish to one central stream. Note that multiple streams in the end will also not help you, since you'll reach that 5 reads per second limit soon as the number of services increases to 5 or higher
  • have a stream per service that needs to consume messages
  • have a smart lambda that reads from the central stream and re-publishes each message on each of the per-service-streams using the same partitionkey the publisher used
  • in case of error in that smart lambda: the lambda should store the failed events somewhere, e.g DynamoDB for later investigation. But when such an error occurs, that means what is received by the consumer is not in the same order anymore as the order the publisher published the messages (the consumer misses a message! And might get it in later when the investigation decides to republish it on the central stream)
  • note that this introduces at least one second delay because of the smart lambda being allowed to poll once a second. And then if the consuming service is also a lambda, another second delay is introduced.
The above solution makes sure each service-stream has only one type of application consuming and any read limits can be fixed by increasing the number of shards (because only one service will be reading from it anyway).
Note Enhanced Fanout was not released yet then, which seems to also solve the described issue.

Options tried by changing the configurations: 
  1. KCL: increasing withIdleTimeBetweenReadsInMillis  and withIdleTimeBetweenCallsInMillis: only helps in a limited way plus reduces throughput.
  2. KCL: same for withMaxrecords(). Even if there are no records, all services will still from time to time have to poll the shard...
  3. Increasing AWS provisioned maximum could of course help but what's the final maximum there? It will probably need to be increased for every X new services.

Increasing throughput

When the consumer needs to be really fast in processing messages from a Kinesis stream shard, one can spin off a new thread per record in the batch. But several things have to be taken into account when doing that:
  • checkpointing is still at record (or batch) level. So you still have to in some way checkpoint only if you know all message before that moment are processed fine too. So you'll need some thread-orchestration to determine to what record to checkpoint
  • if you care about the order, then a thread per record causes processing-ordering issues; so in that case you'll have to first group all records together that apply to the same entity, then start a thread to process these as a whole 

No direct Kinesis consumer

A totally different approach is to have your container services not be bothered with Kinesis at all, and put Lambdas in front of them, which invoke the services via regular REST calls. This alleviates the consumers completely from processing messages.
An example framework that provides this is Zalando's Nakadi. Though implemented for Kafka, still a viable option. This architecture is definitely something to consider.
A more advanced solution could be to have the lambdas put the messages in an SQS queue. This provides even more decoupling and if your service is not available, the lambdas can still deliver the messages into the SQS; care has to be taken of how to support multiple consumers reading from the same SQS queue. Of course then you almost must start to wonder why you are using Kinesis when you are putting a regular queue behind it...

Kinesis Client Libray (KCL) challenges

For the Java consumers the recommended option by AWS is to use the Kinesis Client Library, KCL for short. A short high level introduction can be found here.
This library takes away a lot of standard challenges in a distributed solution. See below's KPL link to implementing efficient producers for an explanation.
But still quite a few challenges exist. E.g how does it handle multiple instances running a Worker? Should you checkpoint per batch or per record?
Are batches provided to the worker one at at time, sequentially or in parallel? How to handle failed messages?
Should workers have a unique ID?

Here's a summary of lessons learned:
  • a shard will be consumed by a record processor thread (not necessarily always the same thread!) of one single worker only at any given moment. And thus batches will never be handed over to processRecords() in parallel; first batch 1, then batch 2 etc.   consumeShard() is protected via synchronized.
  • starting multiple workers on same host won't improve throughput; one on each host seems most effective, otherwise Workers start maybe also stealing leases too many times/too much
  • give each worker a unique ID per host so you can see which host has what lease
  • seems best to pass in your own Executor to the KCL when using Spring, so Spring knows about that threadpool. 
Note that checkpointing too much (or querying your own DynamoDB database while processing each message) can easily cause a ProvisionedThroughputExceededException.
Therefore you also don't want to do that too often. In the end this is the high level algorithm (pseudo code) that one can use to cover all the above concerns:

In processRecords(records) {
   lastSuccessfulRecord = null;
   for (each record in records) {
      try {
         parse record into domain object;
         process(domain object);
         lastSuccessfulRecord = record;
      } catch (ex) {
         checkpointUpToIncluding(lastSuccessfulRecord);
      }
   checkpointBatch(); // All records in the batch successfully processed
}
Another question for next implementation will be: should the service really be doing smart checkpointing (i.e only checkpoint when record is processed successfully; if not successful, keep processing but at next restart of service start from the last checkpoint)? Because the "normal" messaging implementation of putting a message on a Dead Letter Queue might be a better solution. Though in that solution you'll have to think about edge scenarios, like when a message A to modify entity E is put on the DLQ, but a following message B related to entity E is processed successfully; should message A be replayed (published) on the main stream again? Or should as soon as one message for E is put onto the DLQ, all messages after that for E also be put on the DLQ?

Kinesis Producer Library (KPL) challenges

For the Java producers the recommended option by AWS is to use the Kinesis Producer Library, KPL for short. For key concepts see here.
See here for a thorough explanation of its advantages with example code.

The only challenge there was: how large should the batch size be?  It seems to depend mostly on how well your consumers can handle batches.
In the end I set it to values between 10 and 100.
Note that the standard AWS Kinesis SDK does not implement handling batches. That means if you use the KPL with batchsize 10, then when these batches of records are received by e.g Javascript lambdas using the standards SDK, they by default can't process them correctly because batches are not supported; unless you write code to unwrap the batch.
As a temporary workaround there the batchsize could be set to size 1 of course, which then makes sure each consumer gets to process one message at a time and not in a batch.

Hope these lessons learned help somebody while implementing Kinesis.



Wednesday, April 4, 2018

gitlab-runner cache key bug: cache not being created anymore in gitlab steps

Introduction

Besides auto-updating also to the most recent version of the Docker image of maven:3-jdk-8, the Gitlab runners were also always updated to the most recent version.

Again not best-practice and again I learned the hard way: builds suddenly starting to fail.


The issue and workarounds/solutions

 Because suddenly indeed the docker image build of the application started to fail with this error:

Step 9/12 : ADD service1/target/*.jar app.jar
ADD failed: no source files were specified
ERROR: Job failed: exit code 1

It turned out the service1/target directory was empty (or missing, I forgot).
Investigating some more showed that the cache produced in previous steps in Gitlab was not there anymore. The version that this started appearing was:

Running with gitlab-runner 10.1.0 (c1ecf97f)
on gitlab-runner-hosted (70e74c0e)
From older successful builds I saw that at the end of the previous step, when the cache is created, you see something like this:   

Creating cache developbranch:1...
WARNING: target/: no matching files
service1/target/: found 87 matching files
service2/target/: found 33 matching files
untracked: found 200 files
Created cache
Job succeeded

But those loglines were now suddenly missing! And I noticed that the gitlab-runner version changed between the last successful build and this failing one.
So I had an area to focus on: the caching didn't work anymore.

The cache definition in the gitlab-ci.yml was:

cache:  key: "$CI_COMMIT_REF_NAME"
  paths:    - target/
    - service1/target/
    - service2/target/
  untracked: true

I suspected maybe the environment variable of the key: field being empty or something.
But when I added logging in other script steps, the $CI_COMMIT_REF_NAME variable was filled with the value 'developbranch'.  So it is not empty.
Then I had an epiphany and prefixed the above environment variable with a string, making the cache key: definition look like this:

cache:  key: prefix-"$CI_COMMIT_REF_NAME"
  paths:    - target/
    - service1/target/
    - service2/target/
  untracked: true

In the above you can see I prefixed the key with the hardcoded string "prefix-".  And indeed that did it, the creating of the cache worked again and looked like this:

Creating cache prefix-developbranch:1...
WARNING: target/: no matching files
service1/target/: found 87 matching files
service2/target/: found 33 matching files
untracked: found 200 files
Created cache
Job succeeded


So also here I learned (as I really already knew): don't auto-update but do that in a controlled way, so you know when to expected potentially failing builds.









Friday, March 30, 2018

Gitlab maven:3-jdk-8 Cannot get the revision information from the scm repository, cannot run program "git" in directory error=2, No such file or directory

Introduction

In a Gitlab project setup the most recent Docker image for maven was always retrieved from the internet before each run by having specified image: maven:3-jdk-8 in the gitlab-ci.yml. The image details can be found here.

Of course this is not a best-practice; your build can suddenly start failing at a certain point because an update to the image might have something changed internally causing things to fail.
What you want is controlled updates. That way you can anticipate on builds failing and plan the upgrades in your schedule.

The issue and workarounds/solutions

And indeed suddenly on March 29 2018 our builds started failing with this error:

[ERROR] Failed to execute goal org.codehaus.mojo:buildnumber-maven-plugin:1.4:create (useLastCommittedRevision) on project abc: Cannot get the revision information from the scm repository :
[ERROR] Exception while executing SCM command.: Error while executing command. Error while executing process. Cannot run program "git" (in directory "/builds/xyz"): error=2, No such file or directory

That message is quite unclear: is git missing? Or is the directory wrong? Or could the maven buildnumber plugin not find the SCM repository?
After lots of investigation it turned out the maven:3-jdk-8 image indeed had changed about 18 hours before.
And after running the maven command in a local version of that Docker image indeed the same error occured!  Awesome, the error was reproducable.
And after installing git again in the image with:

- apt-get update
- apt-get install git -y


the error disappeared!  But a new one appeared:

[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
This also hadn't happened before. After some searching it turned out it might be the surefire and failsafe plugins being outdated.
So I updated them to 2.21.0 and indeed the build succeeded.

Here's the issue reported in the Docker Maven github. UPDATE: it is caused by an openjdk issue (on which the maven:3-jdk-8 is based upon.

This issue made us realize we really need an internal Docker repository. And so we implemented that :)

One disadvantage about Docker images is that you can't specify a commit hash to use. Yes you can specify a digest instead of a tag, but that is a unique UUID hashcode only. You can't see from that hashcode anymore the (related) tagname.







Saturday, February 3, 2018

How to prevent Chromium from rebooting a Raspberry Pi 3 model B Rev 1.2




I tried to use a Raspberry Pi 3 model B Rev 1.2 as a dashboard for monitoring a couple of systems using Chromium as browser.

Tip: use  this to have it never turn off the display:
sudo xset s off
sudo xset -dpms
sudo xset s noblank

I had only two tabs open all the time and was using the Revolver browser extension to rotate the tabs. One tab had the default Datadog page open, another a custom dashboard within Kibana that refreshed every 15 minutes.
Using all default settings, within a few hours the Pi would reboot out of its own! So something got it to do that.

It seemed the browser (tabs) or the Javascript in them were just leaking so much memory that the Pi ran out of memory.  I tried multiple times with the same default setup, but the behavior was the same each time.

So I tried a couple of other things:
  1. Have the Revolver plugin fully reload the page. Still a reboot of the Pi, though it took a bit longer

  2. Added --process-per-site to the startup shortcut of Chromium. This causes Chrome to create less processes and that should reduce the memory usage a bit. But still a reboot of the Pi; though again it took a bit longer.
    Note that this also comes with its own weaknesses.

  3. Added --disable-gpu-program-cache to the startup shortcut of Chromium. Again still rebooted the Pi after a while.

  4. Tried other browsers like Midori and Firefox Iceweasel.  Midori does not have a Revolver-like plugin, so it didn't fit the requirements. Firefox's only add-on that should work gave some kind of "invalid format" error (don't remember exactly) when trying to install it. The other add-ons for Firefox were not compatible with Iceweasel.

So in the end I did not find a solution :(  I just built a cron-job that would restart the browser every 5 hours.
If you found a way to fix this problem, let the world know in the comments!