Tuesday, September 27, 2022

Kotlin for Java Developers course summary

Introduction

This is my summary of relevant lessons learned during a Kotlin for Java developers course.


Summary

General

- When running standalone, needs Kotlin runtime
- Toplevel functions: not explicitly defined in a class, e.g fun main() method
- Useful other info: https://kotlinlang.org
- val number: Int and val number = 25    So has type inference.  But if needed: number: Short = 25
- Recommended good practice: delare variables with 'val' where possible
- Properties of a 'val' class instance you can modify
- Via class "constructor" you can also specify whether to set properties is allowed. E.g class Employee(var name: String, val id: Int), then you can do: employee.name = "abc", but not employee.id = 5.  Note that construction of instance needs both parameters anyway.
- Collections: you can use [i] notation to get to an element.  Or for Maps: use the key
- No difference between checked and unchecked exceptions, no need to specify them in method names. You can't even add a 'throws'.
- Kotlin has no static keyword syntactically, e.g see the 'fun main()' declaration. But they do exist.
- No 'new' keyword
- The '==' operator checks for structural equality, so it works the same as .equals()! So .equals() you don't need to use. How to check for referential equality use: '==='
      Annoying, in Javascript '===' means more equality, in Kotlin less (only referential equality).
- Bitoperators like '|' you spell out like: 'or'
- Use 'is' instead of 'instanceof'
- Smart casting: use the 'as' keyword, e.g: val newValue = something as Employee. But if you did an 'is' check before, you don't need the cast anymore.
- Raw strings can be triple quoted, so you don't need to escape special characters, e.g: val path = """c:\somedir\somedir2"""
- trimMargin() for indentation in code of raw strings that cover multiple lines
- Everything is a class, no native types like long or int.
- Kotlin does not automatically widen numbers. E.g: in Java: int a = 55;long b = a; is fine. But not in Kotlin. So all standard datatypes have toLong() and similar conversion methods
- Double is default just like in Java when declaring variable w/o specifying type, e.g: val myDouble = 65.994. Int also, e.g: val myInt = 11
- Any class: similar to root class of Java class hierarchy
- Unit class: similar to void in Java, but different: it is a singleton unit instance
- Nothing class: subclass of any class. E.g when you have a function with an infinite loop, then return 'Nothing'.
- The RPEL can be used as "scratchbook" of executable code, which knows of your existing classes

Arrays

- Initialize array using a lambda, even after declaration: val evenNumbers = Array(16) {i -> i * 2}     So here i is the index of the array
- Mixed array: val mixedArray = arrayOf("Hello", 22, BigDecimal(0.5), 'a').  It is then an array of Array<Any>.  Or just call .toIntArray() on the Kotlin Int array.
- To call a Java method with int[] as parameters, you can't pass in the Kotlin collection Array, so you have to use: intArrayOf(1, 2, 3) instead of arrayOf(1,2,3). Is also a bit of performance boost since directly you create a primitive type array for the JVM (of type int).
- You can't do var someArray = Array<Int>(5). 
- You can do for any type: val myData = 'arrayOf(car1, car2, car3)', where the 3 cars are of type Car.
- Convert int[] created via intArrayOf() to a Kotlin typed array: val typedArray = intArrayOf(1,2,3).toTypedArray() 

Null references

- For types that can be nullable, you have to tell it can be null by appending a '?', e.g: 'val str: String? = null'
- If you check for variable being null, Kotlin compiler from then on knows it can't be null, so all methods are available again (similar to smart-casting)
- 'str?.toUpperCase()' performs behind the scenes a 'if (str != null) str.toUpperCase() else null'. The '?' is called the "safe operator".
- So nested also possible, then it stops at the first one that is null. E.g: 'home?.address?.country?.getCountryCode()'
- To set a default value in case it evaluates to null, use the elvis operator: '?:'. E.g: 'val str1: String? = null; val str2 = str1 ?: "this is the default value"'
- Safe cast operator allows any cast, and if it can't do it, it sets the variable to null. E.g: 'val someArray: Any = arrayOf(1,2,3); val str = someArray as? String' results in 'str' having value null.  So '?' really means: this (variable) *might* from now on evaluate to null.
- If you are sure an expression can't evaluate to null, you can tell the compiler via the non-null assertion: str!!.toUpperCase().  After this point you don't have to add the '?' anymore for null safety checks (because you told the compiler it 100% can't be null). Use this e.g when you *do* want a nullpointer exception to be thrown.
- The 'let' function: 'str?.let {println(it)}', which says: if str isn't null, then let this function-call go ahead via a lambda expression.
- Note that the == operator is safe for nulls (note underwater it uses .equals()), e.g: 'val str1 : String? = null; val str2 = "abc"; str1 == str2'
- To have an array of null values, you have to use 'arrayOfNulls()' method, e.g: 'arrayOfNulls<Int?>(5)' creates one of size 5. Or: 'arrayOfNulls<Int>(5)'
- Use !! when you are sure the variable can never ever be null

Access modifiers

- 4 types for toplevel items:
Default = public final (instead of package level in Java). Note classname doesn't have to match filename in kotlin. One file can have multiple public classes.
Private = Everything in the same file can access it. (Java doesn't allow a private class). When specified for a constuctor parameter, no getters/setters are generated! So really only the class itself can have access to that property, no-one else.
Protected = can't be used
Internal = is inside the module (see below) (no Java equivalent)
- Kotlin has the 'module' principle: group of files that are compiled together.
- For class members:
Public, private, protected mean the same as in Java.
Internal = visible in same module, usually outside the file
- In Kotlin, classes can't see private members belonging to inner classes
- Compile time:
private in Kotlin is compiled to package private
internal in Kotlin is compiled to public
That means sometimes Java code can access Kotlin code that from w/in Kotlin (compiler) can't be accessed
Kotlin generates really long strange names for those type of methods/classes defined as internal, so you are kindof warned when using them from Java.
- Create constructor verbose, Java-like: 
class Employee constructor(firstName: String) {
         val firstname: String

     init {
this.firstName = firstName
    }
        }
Note the 'constructor' keyword as primary constructor (declared outside the curly braces). 
And the 'init' block. The init block runs after all fields have been initialized. 
- Create less verbose:
class Employee constructor(firstName: String) {
            val firstName: String = firstName
        }
- Even less (note the 'val' keyword added; can also be 'var'):
class Employee constructor(val firstName: String) {
        }
- Even less:
class Employee (val firstName: String) {
        }
- But if you want to change visibility of the constructor, you will have to add the 'constructor' keyword: class Employee protected constructor(...)
- For more constructors, just add 'constructor's in the code. But you also have to invoke the primary constructor if present. For that do:
constructor(firstname: String, lastName: String): this(firstName) {
xxx
}
Note no 'val' prefix anymore, since the 2nd constructor does not declare variables! Just declare the 2nd parameter as field in the class.
- 'val' and 'var' in the primary constructor generate the matching properties. If not specified, it is just a parameter in that primary constructor.
- You don't *have* to specify a primary constructor
- Kotlin classes only have properties, no fields (what's the difference? Note there's use of something named the "backing field", see below). Default for properties is also 'public'.
- Properties defined as 'var' will have setter generated too. 
- 'private var' properties: can't be accessed at all.  This is because the (generated) getters + setters must have the same visibility as the property.
- If you want your own getters+setters, you can't declare that property in the primary constructor; it has to be declared in the class.
- Backing field (strange name to be just thrown in in the course):
class Employee(val firstName: String, lastName: String = "") {
        var lastName = lastName
                get() {
return field
                }
       }
   Note the get() has to be immediately after the property. And also the special keyword 'field'.
- Similar for the set() method. But note the caller can do: employee.lastName = "abc". No 'setLastName()' call needed.
- You have toplevel constants too just like toplevel functions. So you don't need to declare a constants file for example. val MY_CONSTANT: String = "Text" . Note also the 'const' keyword exists.
- Data class: 'data class Bike(val color: String, val model: String) {}'. You get for free: toString(), equals()+hashCode(), copy() function.
    Requirements: primary constructor has to have at least 1 parameter marked val or var (unlike lastName in previous example). Also: can't be abstract, sealed or inner classes.
- Parameters can be passed in in any order because you can specify the name of the parameter during the call. These are called "named arguments".
- Internal access modifier: private class + internal functions is meaningless, since the private class already restricts it to the file.
- listOf() returns an immutable list of items

Functions

- Simplified function definition: fun myFunction(param1: Int, param2: Int) = "Multiplied = ${param1 * param2}". So the function returns what is after the equals sign. Defining it this way (so not with a {} block body) is called an expression body.
- Returning nothing means specifying Unit (or leaving it out since it is the default)
- Function parameters must have a type specified (even though compiler could derive the type!)
- Variable number of arguments: 'vararg' keyword, e.g: 'fun addUp(vararg amounts: Int)'
- Spread operator: unpacks the array to pass each element separately (used when having to pass to a vararg parameter): addUp(*arrayOfInts). Yes a '*'.
    In Java you could pass in an array of objects to a method with a varargs parameter, but not in Kotlin.
    Another example: 'val newArray = arrayOf(*array1, *array2, anotherElement)'.  If you don't do this, you'll have 2 arrays + one element in 'newArray'!
- Extension functions lets you add functions to classes. Just add the class + a dot (the so-called "receiver type") to a function, e.g: 'fun String.replaceEWithA() {}'. You access the string with 'this' keyword in that method.
    Note it knows since you are putting the extension on String, no need for a String parameter in the new function. 
    Any public field in the class you add the extension function to can be accessed.
    Feels a bit like a "hack" adding random functions to existing classes. People might also get lazy, not coming up with a new class, but just adding it to an existing one because it matched "good enough". Also for a newcomer on the project it is not possible to easily distinct that a method is an
       extension function.
- Function parameters are of type 'val'.
- Inline functions: you can tell the compiler to inline by prefixing function definition with the 'inline' keyword. Usually works best for functions with lambda parameters

Inheritance

- Extend (subclass) a class: 'class RaceBike(): Bike() {}'. Class Bike has to have the 'open' keyword as prefix because default is final (not extendable).
So: 'open class Bike() {}'.  And if you don't want to specify an empty primary constructor: add 'constructor(): super()' to the subclass RaceBike.
- For an abstract class you can remove the 'open' keyword, because 'abstract' implies already that it will be extendable.
- Overriding a method in the subclass requires 'override' in the subclass method and 'open' in the superclass method.
- An 'abstract fun' also needs the 'override' keyword in the subclass. No need for the 'open' keyword here either, it is automatically. Same for the 'override' keyword!
- Subclass (of course) doesn't have to have the same number of parameters for the primary constructor
- Invoking "super.primary constructor" only makes sense/possible if you don't have any primary constructor defined for the parent
- Usually you'll provide a primary constructor and only add secondary ones when absolutely necessary
- In a subclass when creating a secondary constructor, call the constructor (IntelliJ might report it is missing superclass call or similar) like this: 
'constructor(prop1, prop2,..., newParam: Int): this(prop1, prop2,...)'
- Data classes can't be extended (nor be abstract class or inner class)
- Interfaces can be extended with the same syntax: 'interface MyChildInterface: MyParentInterface {}'. Classes can implement multiple interfaces like Java.
- Interfaces can have properties. Which the implementing class has to define too with an override: 'override val number: Int = 30'.
- If you want to set a property's value in an interface, giving it a concrete implementation, you have to define a 'get()' directly below it. Not sure if you'd ever want that in an interface...
- Properties in interfaces don't have the backing field 'field' identifier (which classes do have in the custom get() and set()).
- Making a parameter optional (with default value) in a function in a superclass, doesn't make it an optional parameter in its subclasses.

object keyword

- 'object' keyword is used for: singletons, companion objects, object expressions.
- For singleton there's always only one instance created, very first time you use this class. So I guess not static classes in Java which are created I think at application startup.
     You can access the functions of the singleton, quite normally: MySingleton.someMethod().   Note the singleton class is not constructed first.
- If you want to add e.g functions to a class that are statically accessible (so w/o having to create the class), wrap it with 'companion object {}'.
     The non-private methods in that block are from then on accessible, using special keyword: 'MyClass.Companion.someMethod()'. But then the class is not a singleton but a regular class with a companion object block in it.  You can also give the companion a name. Kotlin also is smart enough you can ommit Companion. This is really much more verbose than the Java 'static' keyword!
- Companion objects can also be used to call private constructors. You can use them to implement factory patterns.
Of course you have to make the primary constructor (if exists) private too then to prevent instantiation outside the factory, e.g: 
class SomeClass private constructor(val input: String) {
companion object: {
fun createBasicSomeClass(val input: String) = SomeClass(input)
}
}
- Object expressions are where in Java you'd use anonymous classes, e.g implementing an interface that needs to be passed as parameter to a function.
Also called anonymous instances.
E.g: 'functionToCall(object: SomeInterface {
override fun functionToImplement(text: String) = "implemented text $text"
}
      '
Note that the object created is not a singleton (confusing, since you use the word 'object' which is also used to create a Singleton)
Also, the object expression code can access local variables in scope (they don't have to be 'val' (or "final" in Java terms), can be 'var' too). So also change that local variable's value!

Enums 

- Use 'enum class' keyword. 
- How annoying: if you add a 'fun' to an enum, you have to add a ';' after the last enum value!! Odd that compiler can't figure it out w/o the ';'...
- Example invoking a function in enum for a given enum value: 'DepartmentEnum.ACCOUNTING.getInfo()'

Importing

- Recommended to (of course) use the same package structure as underlying directory structure. (though it is not a requirement from Kotlin)
- Toplevel functions (in files, not in a class) can be imported like a class
- What if you want to use a function from another module? Then (in IntelliJ) you have to add the dependency to that module first. Will be a regular Maven dependency in Maven I expect.
- Type aliases and extension functions you import the same (they are all top-level anyway)
- You can rename imports too using the 'as' keyword. Useful when 2 third party libraries use the same name for some class.

For loop

- while and do-while is same in Kotlin
- For loop syntax like in Java doesn't exist in Kotlin. 
- Range: values are inclusive. E.g 'val range = 1..5'.  But also: val charRange = 'a' .. 'z'. Because they are comparable. val stringRange = "ABD".."XYZ"
- 'in' keyword for: 'println(3 in range)' is true. But also '"CCCCC" in stringRange' is true! Because the first 'X' is already greater than the first 'C'.
      '"ZZZZZ" in stringRange' is false because the first 'Z' is greater than 'X'
- 'val rangeBackwards = 5..1' is usually not what you want because '5 in rangeBackwards' is false! Because 5 >= 5 but 5 <= 1. So you have to do: '5.downTo(1)'. Not really intuitive!
- You can also provide a step: range.step(2). And '.reversed()'
- To print a range: 'for (i in range) println(i)'.  But not possible for a string-range because it has not iterator. Makes sense of course.
- But for a regular string it is possible: 'for (c in str) println(c)'.
- Loop itself can also step: 'for (i in range step 4) println'. Similar is there a 'downTo' for counting down.
- To not include the last number: use keyword until: 'for (i in 1 until 10) println'.
- Also variables can be used to define the range: 'val range = 1..someStr.length'.
- Negate: 'val notInRange = 33 !in 1..5' evaluates to true. But also: ''e' in "Hello"' is true.
- Index in for loop: 'for (index in myArray.indices) println("Index = $index, value = ${myArray[index]}")'
- Much shorter: 'myArray.forEach { println(it) }'. Or with index: myArray.forEachIndexed {index, value -> println("$value is at array index $index") }
- Loops can given a name: 'myLoopName@ for (i in 1..5)'. When having nested loops, in a subloop of myLoopName, you can say: 'break@myLoopName
    Then you don't further continue running that loop with 'i' anymore either!  Even when having more subloops before hitting the 'break@' statement.
It is almost a goto-statement! Risk of for example spaghetti-code.
- Works similar with 'continue@myLoopName'.

If expression+statement

- 'if' can evaluate to a value! Not possible in Java. You have to put the return value as the last statement in a {} block. In both the if and else block!
- Comparable with the ternary operator in Java: 'val num = if (mycondition) 40 else 60'. You can even write a full 'val num = if () {40} else {60}'.

When expression

- Is the Java 'switch' statement on steroids'
- 'when(condition) {
10 -> println
20 -> println
else -> println("doesnt match")
  '
- No need to add the 'break' statement. So no falling through at all
- More than 1 value possible in the match, e.g: '10,11' it will match on both those values. You can also use ranges. 
- And expressions: 'x + 50 -> println()'. And smart casting: 'when (something) { is String -> println() is BigDecimal -> println()}'
- And also as in the if statement, you can return a value in each branch of the 'when'. Best practice is to *not* return a different type of value in the branches.
- An empty when, looking like an if statement: 'when { 
i < 50 -> println 
i >=50 println 
else println
     }'. Which is much more concise than 3 if statement branches.

Try/Catch

- Reminder: no distinction between checked and non-time exceptions.
- Also the try/catch you can use as expression
- E.g: 'return try { Integer.parseInt(str) } catch (e: NullPointerException) {0} finally { println("hello")}'.
      Note the finally block doesn't return an Int. But that is fine, since the catch block does and the normal block does. So the finally block is not used in the evaluation of a try/catch expression.
- Use case for Nothing return type for functions: as return type for a method that only throws an exception, e.g the NotImplementedYet() exception.

Lambda expressions basics

- You can call them directly using the Kotlin library 'run' function: 'run{println("lambda being run")}'
- 'println(cars.minBy {car: Car -> car.builtYear}'). minBy is a Kotlin collection function built-in. Note you can move the lambda outside () when the last parameter. And if the only parameter, even leave out the (). Note also the 'Car' could be left out, Kotlin can infer the type.
- In this case the compiler can infer even more, so we can use 'it': - 'println(cars.minBy {it.builtYear}'). 
- You can also use a member reference: 'println(cars.minBy(Car::builtYear))'
- And toplevel function calls: 'run(::topLevelFun)'. Where: 'fun topLevelFun(): println("hello")'
- You can access local variables declared before the lambda in the lambda. In Java you can only access final variables in lambdas and anonymous classes.

Lambdas with receivers

- 'with' keyword: 'return with(StringBuilder()) { append("hello") toString() }'.  So can also be written as: 'return with(cars, {println(car)}'.
- So the 'with' turns the parameter you pass it into a receiver. That is how it is called. Inside the lambda you don't have to refer to the receiver object anymore. You can if you want with the 'this' keyword.
- 'apply' keyword is another receiver keyword: 'StringBuilder().apply() { append("hello") }.toString()'.
- A 'return' in an inlined lambda also returns the function it is in, a so called non-local return.
cars.forEach {
if (it.builtYear == 2019) {
                return
}
}
        println("this is not executed when at least 1 car has builtYear 2019")
- You can have a local-return but then you need a label:
cars.forEach returnBlock@ {
if (it.builtYear == 2019) {
                return@returnBlock
}
}
println("this is now also executed even when at least 1 car with builtYear 2019")
- You can also use the label to refer to nested 'apply's and 'with's:
"some text".apply sometext@ {
"another text".aplly {
println(toUpperCase()) // Only converts 'another text'
println(this@sometext.toLowerCase()) // converts the outer 'some text'
}
}
- 'also()' is similar to 'apply()', but you don't need to use 'this'.

Collections: Lists

- All read-only interfaces are covariant: be able to treat a class like its parent (superclass). E.g assign a List of 'Any's to a list of BigDecimals. See next Generics part for more details on this.
- For mutable collections you can't change the collection (of course), like adding elements. Seems usually these are also covariant (because they extend the (immutable) Collection interface which is covariant)
- 'listOf' creates an immutable list. But 'arrayListOf()' creates a mutable list!  See the type of the underlying Java class: first one is Arrays$ArrayList which is inmutable, the 2nd is ArrayList, which is mutable. I guess using 'mutableList' is more informative than using 'arrayListOf'.
- Handy: 'listOfNotNull(a,null,c)' creates an immutable list of only a and c.
- Convert list to array: 'listOf(*arrayOf("a", "b", "c"))'. Note the spread operator '*' there. But you can also do: 'array.toList()'.

Kotlin's added collections functions

- last(), asReversed()
- using the array notation to get an entry at given index: list[5] instead of list.get(5)
- list.getOrNull(): returns null when entry is null. Similar to Optional in Java.
- max(): entry with highest value
- zip(): creates Pair elements.  This is Kotlin Pair so a bit different from Java Pair. For me a bit unintuitive name that 'zip'.
- 'val combinedList = list1 + list2' // So you are really concatenating
- To combine & get no duplicates from 2 lists: 'val noDuplicatesList = list1.union(list2)'. Just like union in SQL.
- To get no duplicates in 1 list: list.distinct()
- Just like in Java, some functions return a new list, others work on the existing list. I expect you get at least an exception when trying the 2nd case on a immutable list.

Maps and destructuring declarations

- 'mapOf<Int, Car>(1 to Car("green", 2012), 2 to Car("yellow", 2013))'. The number is the key of the map. The <Int, Car> is redundant of course.
- And a mutable map: 'mutableMapOf()'.
- In both cases the underlying implementation is LinkedHashMap, so the iteration order is predictable and easy to convert to a Set for example.
- But you can force a hashmap: 'hashMapOf()'.
- Destructuring: 'val (firstValue, secondValue) = Pair(1, "one")'. Kind of "unpacking". And even: 'for ((key, value) : in mutableMap) { println(key) println(value)}'
- To be able to destructure a class, you need to use: component functions
- To add them to your own class use:
class Car(val color: String, val model: String, val year: Int) {
operator fun component1(): color
operator fun component2(): model
operator fun component3(): year
}
    Unclear if these have to have such fixed names componentN().
- Data classes get the component functions already generated (created) for free.

Sets

- To create an immutable set: 'setOf()'. To add an element to a set: 'set.plus(str)'. Duplicates added are ignored. To remove an element: 'set.minus(str)'. (you get a new set when using immutable sets)
- 'drop(nr)': drop the first 3 elements of the set
- Some functions work on the set itself, most return a new set with the function applied.

Other collections functions

- filter(lambda): returns a new instance of the collection
- 'val added10List = myIntsArray.map {it + 10}': adds 10 to each element in myIntsArray and puts that in new list (a java.util.ArrayList)
- You can chain them of course: 'myCarsMap.filter { it.value.builtYear == '2012' }.map { it.value.color = "yellow"}'
- To check for all elements matching a value, use the 'all(lambda)' function, which returns a boolean. And 'any(lambda)' for at least 1 matching.
- count(lambda)
- 'groupBy(lambda)'
- 'sortedBy(lambda)'
- 'toSortedMap()' to sort by key of the map

Sequences

- filter() creates a whole new copy of the collection. Which can be huge. So to prevent too much memory usage, you can use sequences. Similar to Java streams, but re-invented by Kotlin because Android didn't support Java 8 yet back then. To make a map or list a sequence use: 'asSequence().filter()....'
- 2nd case where you need to put a ';': when you have a lambda predicate which you want to put on 1 line, e.g: '.filter { println("$it"); it[0] == 'yellow'}'
- When you have a sequence, the filter(), map() etc return also a sequence, not the list/collection, because they are intermediate operations. So you need a terminal operation in the chain of calls. E.g: 'toList()'. But for example 'find()' stops at the first match. Usually you want to 'map()' late(st) in the chain of calls. Just like Java streams.

Generics

- You always have to specify the generic type in Kotlin in e.g List<String>. So 'List' is not allowed. Also not for 'myList is List'. There you can use: 'myList is List<*>'. That is called the star projection.
- fun <T> myFunction(collection: List<T>) {}
- Or even as extension function: fun <T> List<T>.myFunction() { println(......) }
- fun <T: Number> myFunction(collection: List<T>) to limit to certain subtypes (you specified the Number as upperbound)
- Multiple upperbounds: fun<T> doIt(item1: T, item2: T) where T: CharSequence, T: Apppendable {}
- Note that T accepts the nullable type too!  By default, if no upperbound specified, it is 'Any?'. So nullable! I call this inconsistent with the "try to not support null" goal :)
- You can make it not accept nullable type by specifiying of course: 'T: Any'.
- Just like in Java, at runtime the generic type is erased (not available anymore)
- In Java you can't do 'list instanceof List<String>'. But in Kotlin you can do: 'list is List<String>'. But it can't for a list of type Any.

Reified parameters (generics related)

- Reification: prevents the type from being erased at runtime
- To do that: make the function 'inline' and 'reified T': 
inline fun <reified T> getElements(list: List<Any>): List<T> {
for (element in list) {
if (element is T)  <----------- now compiles!
}
}
   Then call it: val myListOfAny: List<Any> = listOf("string", 1, 20.0f); getElements<BigDecimal(myListOfAny)

Covariance (generics)

- MutableList<Short> is not automatically the same as ImmutableList<Short> at compile time in case of for example function parameters. Immutable collections are covariant, so then the "subtyping" is fine. But if you define the same parameter collection as mutable, the "casting" can't take place (compile error).
- So MutableList is invariant and wants the exact type (you see this also by the 'out' not present at that class's definition; more on that below)
- And so List has the 'out' keyword, so is covariant. So subtyping is preserved. And since it is an immutable interface/class, nothing can change the list anyway. Still some functions have 'T' as parameter while 'out' is there! Like 'contains()'. @UnsafeVariance is then used for such a (in) parameter, telling compiler that the list won't be changed.
- Note that: List is really a class. But List<String> is a type! E.g: Short is a subclass of Number. List<Number> is supertype of List<Short>
A nullable type is a supertype of non-nullable types (a wider type).
- So List<Short> is not by default the same as List<Number>, even though the 2nd is a super-type(!) of the first.
- Mutable collections interface = not covariant. Collections interface = covariant.
- So if you want the sub-typing(!) to be preserved, you have to prefix the type with the 'out' keyword. E.g: 'class Garden<out T: Flower>' 
- That keyword implies you can use that class only in "out position".  By default parameters are of 'in' position. And usually the out position is the return type. By specifying the T as 'out' in the Garden above, you can *only* use it as return type!
So 'fun getFlower(nr: Int): T {}' is fine.  But 'fun waterFlower(flower: T)' won't compile!
  Because otherwise you can pass in a different type than the original Garden was created with (e.g: rose vs daffodil); the compiler can't tell.
Note that Garden is a type, not a class, due to that generics part.
So in/out is protecting the class from invalid updating.
- So if you have a covariant class ('out' keyword), subtyping is preserved
- Constructors don't have in/out parameters. So you can always pass in a covariant class as constructor parameter, since it is at construction type.
But if you specify a var parameter: can't be covariant class as input, because again you could pass in the wrong type, because it generates also 
a setter. But a 'private var' is ok again because nobody outside the class can access it.
- Covariant type: subtyping preserved so you can pass an instance of the type or the subtype as for example a parameter. But you can't change the instance (thus when having the 'out' keyword).
- In Java it would be like this: List<? extends Car>,  where you now accept anything that extends Car. This you see in Java in method declarations only. (a variable of that type you can't anything to)

Contravariance

- Is the opposite of Covariance. With Covariance you are preserving the subtyping (incorrectly said(?): accept all of its sub"classes". Is it classes or types or both?). With covariance you start at a subclass and you want to accept instances of that subclass, or any of its superclasses.
- So you use the keyword 'in': interface FlowerStuff<in T> { fun doSomething(flower: T) }. So if T is a direct or indirect superclass of a class, then the class will match T. And you can't have in that case the T as return type, so 'fun getFLower(): T' does not compile. Again you won't be guaranteed to get the type you want if that would have been allowed. Or said differently: not all flowers are a rose. But all roses (subclass) are a flower (superclass).
- Again (depends on where you looking at from the inheritance tree; some say it is "flipping the inheritance tree for a class": 
Covariance: you want to accept a class and all its subclasses   ("looking down the inheritance tree")
Contravariance: you want to accept a class and all its superclasses ("looking up the inheritance tree")
You are widening a generic type to include a class and its subclasses (covariance), or a class and its subclasses (contravariance).
"declaration site variance", so used during declaring an interface. In Java you only have "use-site variance".
- In Java it would be: List<? super Car>: accepts Car and any of its superclasses. This you see in Java in method declarations only. (a variable of that type you can't read from)

Use-site variance

- Generic types are invariant. So even if 'Ford: Car', you can't call 'Car copyCars(source: MutableList<Car>, dest: MutableList<Car>)' with 'copyCars(mutableListof(Ford(), Ford()), mutableListOf(Ford())'. You have to change the Car type in the copyCars to 'T'.
- And to have 'copyCars(fords, cars)' compile, you have to add 'out' (covariance!) to the 'source: MutableList<out T>'. Because that parameter is not changed in the code, so we can do that. This is called use-site covariance, because we didn't add it to the Car class or its subclass like Ford. You can also add 'in' to the dest parameter, but is not needed in this case.
- Also called type-projection.
- In Java declaration-site variance doesn't exist, so you'd have to add the in/out to each method of the class. But so in Kotlin you can do it at class level, i.e declaration-site variance.

I/O

- Just extension functions added to the java io classes like java.io.File.
- var lines = File("myfile.txt").reader.readLines()
- Or to have the reader resource close automatically at the end: val lines = File("myfile.txt").reader().use {it.readText}
- reader().forEachLine() to read per line
- instead of try with resources in Java, use the 'use()' function, that closes the resource always, even if exception.
- File(".").walkTopDown() for file tree traversal

Java interoperability

Calling Java from kotlin:

- Primitive arrays in the Java method as parameter like int[]: you have to use toIntArray() or intArrayOf().
- Nullability: @Nullable @NotNull are Jetbrains IntelliJ(!) hints that can be used. So they are NOT Kotlin annotations, but you give in IntelliJ the Kotlin compiler these hints. Quite ugly in your code base of course. It might also recognize Lombok's or javax's.
But, you only get an IllegalArgumentException at runtime, so compiler won't complain if you assign null to a field.
E.g: Java: 'void setColor(@NotNull color) {}' and then in Kotlin: 'var car: Car = null' compiles fine, but IAE at runtime because the Kotlin compiler generates the not-null check.
Default when none of these 2 annotations: it is nullable type for Kotlin
- You can directly access the field of the Java class like car.color when the Java class has matching getter + setter methods. Or make the field public.
- Exceptions from java: no need to add a throws to the Kotlin function
- varargs in Java: you can't pass a Kotlin array, you'll have to spread (unpack, *) the array
- void from java method: in Kotlin you'll see Unit
- Use Java Object methods that are not in Kotlin's Any, like the Object.notify() method: (car.anObject as java.lang.Object).notify().
- Static fields and methods in Java: are converted to companion objects, so just like this: Car.myStaticfield and Car.myStaticMethod().
- Single Abstract Methods in Java, e.g the Runnable interface which only has one method run() which you have to implement: you can pass them a Kotlin lambda (just like the Java lambda)

Calling Kotlin from Java:

- call toplevel Kotlin functions: Kotlin compiler creates a class based on the .kt filename. So: 'Car.kt' then you invoke 'Car.kt.myMethod()'.
But you can change that generated classname using the @file:JvmName("myclassname")
- Extension functions you can call from java by prefixing the classname too
- getX() and setX() getters setters are available in Java for 'var' fields
- To be able to directly access fields in Kotlin classes, you have to add the @JvmField on the field in the Kotlin class. Has a few use constrictions
- Companion object access: usually: Car.Companion.myMethod(). If you want to not have to use the Companion part, annotate the companion method with
@JvmStatic fun myMethod() {}
- 'object MySingleton() { fun myMethod() }' will be callable from java as: MySingleton.INSTANCE.myMethod(). Also here you can annotate the method with @JvmStatic to avoid that INSTANCE part.
- For a 'const val' in Kotlin, you don't need that @JvmStatic, it is directly accessible: MySingleton.myConstant
- When passing null to a function in Kotlin that is expecting a non-null type, at runtime the Kotlin will check for null (generated by the Kotlin compiler)
- To have Kotlin functions tell Java code an exception can be thrown, annotate method with: @Throws(IOException::class)
- When default parameter values for parameters in Kotin functions: to have Kotlin generate all combinations for those optional parameters use: @JvmOverloads

Not covered

  • coroutines
  • DSL
  • Spring boot integration, annotations how?
  • Lombok annotations work with Kotlin?
  • How to do unittests
  • How to do integrationtests

Considerations

  • Why are "methods" called functions in Kotlin? Functions sounds less Object Oriented...

Wednesday, September 14, 2022

Connection refused Spring Boot application to MySql in Docker solution

Introduction

The Spring Boot Java application was initially using the H2 embedded database, first the in-memory version (losing all data each run), then the file based version. The database structure is created using Liquibase.

Then the requirement was to have the Spring Boot Java application still not dockerized (just started with mvn spring-boot:run), but have it connect to a MySql 8 database than runs inside a Docker container. Most examples you can find have also the Spring Boot application in a Docker container.
The official Spring documentation only shortly mentions the possibility of running MySql in Docker, but no more details on how to do that: https://spring.io/guides/gs/accessing-data-mysql/

This is how the setup should be working:


I set up that docker container using the Windows 10 WSL 1 shell within IntelliJ (note WSL 2 is now recommended):

docker pull mysql/mysql-server:8.0
docker run --name recipesmysql8 -d mysql/mysql-server:8.0

After changing the One Time Password (OTP) for root, creating the database user, creating a 'recipes' database, this is how docker ps looked:

CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                PORTS                       NAMES
3a507fc66690        mysql/mysql-server:8.0   "/entrypoint.sh mysq…"   6 days ago          Up 6 days (healthy)   3306/tcp, 33060-33061/tcp   recipesmysql8

And MySql is indeed running: docker logs recipesmysql8 shows:

2022-08-31T19:02:05.722742Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 1
2022-08-31T19:02:05.755538Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-31T19:02:06.104813Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-31T19:02:06.416522Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-31T19:02:06.416574Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-31T19:02:06.443572Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-31T19:02:06.443740Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MyS
QL Community Server - GPL.


So all running fine, ports seemed fine. The Spring Boot application configuration for JDBC is this:

spring.datasource.url=jdbc:mysql://localhost:3306/recipes
spring.datasource.username=recipes
spring.datasource.password=mypasswd
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver

But then the mvn spring-boot:run gave this relatively vague error:

com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    ...
   Caused by: java.net.ConnectException: Connection refused: no further information

So unable to connect, but why? It does not seem an incorrect username/password combination, then I would expect some unauthorized/not authenticated type of message.
Then I found this handy command determining what the real IP should be of the machine that the MySql docker runs in:

docker inspect -f '{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq)

I found my MySql container at: /recipesmysql8 - 172.17.0.2
Same result, shorter answer: docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' recipesmysql8
Or even just without the filtering: docker inspect recipesmysql8

So the JDBC configuration I changed to:

spring.datasource.url=jdbc:mysql://172.17.0.2:3306/recipes
spring.datasource.username=recipes
spring.datasource.password=mypasswd
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver

But then the mvn spring-boot:run gave this timeout error:

Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    ...
    Caused by: java.net.ConnectException: Connection timed out: no further information

So a similar error message, but this time a connection timeout. Is maybe the problem connecting from within IntelliJ where I execute the mvn spring-boot:run command, to Docker that runs in WSL?

I tried by disabling the antivirus software and local firewall and WIFI, but those didn't work either, at most a different error message like:

Caused by: java.net.NoRouteToHostException: No route to host: no further information

And even with the mysql shell client does it give an error (bit less clear):

mysql -h 172.17.0.2 -P 3306 --protocol=tcp -u root -p
Enter password:
ERROR 2003 (HY000): Can't connect to MySQL server on '172.17.0.2' (11)

Solution

Then I stopped my container, started a new one with explicitly ports-mapping specified:

docker run --name recipesmysql8v2 -p 3306:3306 -d mysql/mysql-server:8.0

docker ps output:
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                   PORTS
    NAMES
819333cf718e        mysql/mysql-server:8.0   "/entrypoint.sh mysq…"   2 minutes ago       Up 2 minutes (healthy)   0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060-33061/tc
p   recipesmysql8v2

Then a different error message for mysql -h 127.0.0.1 -P 3306 --protocol=tcp -u recipes -p appeared:

ERROR 1130 (HY000): Host '172.17.0.1' is not allowed to connect to this MySQL server

That looked promising.

Note: due to having a new container, I had to reset the OTP for user 'root' again of course. You can find that OTP by issuing docker logs recipesmysqlv2 | grep GENERATED.
Then issue these commands:

docker exec -it your_container_name_or_id bash
mysql -u root -p
Enter password: <enter the one found with the above grep shell command>
ALTER USER 'root'@'localhost' IDENTIFIED BY 'your secret password';

Create the Spring Boot application database again: create database recipes;
Add the Spring Boot application user again: create user 'recipes'@'%' identified by 'L7z$11Oeylh4';

Give that user only the necessary permissions:
grant create, select, insert, delete, update on recipes.* to 'recipes'@'%';

After that, these entries should be in the mysql.user table:
mysql> SELECT host, user FROM mysql.user;
+-----------+------------------+
| host      | user             |
+-----------+------------------+
| %         | recipes          |
| localhost | healthchecker    |
| localhost | mysql.infoschema |
| localhost | mysql.session    |
| localhost | mysql.sys        |
| localhost | root             |
+-----------+------------------+
6 rows in set (0.00 sec)


Note the recipes user entry with host '%', which allows access from any host, which you probably want to make more secure in a production environment. See https://downloads.mysql.com/docs/mysql-secure-deployment-guide-8.0-en.pdf for tips.

Now you should be able to connect from the mysql client prompt in several ways:

mysql -h 127.0.0.1 -P 3306 --protocol=tcp -u recipes -p
mysql -h localhost -P 3306 --protocol=tcp -u recipes -p
mysql -h 0.0.0.0 -P 3306 --protocol=tcp -u recipes -p

Note that the 172.17.0.2 still does not connect! For that to work you'll have to probably add that host to the above mysql.user table (not tried to see if that works).

Instead of mysql client you can also use the standard *nix command telnet to see if at least the port is reachable:

telnet 127.0.0.1 3306
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.


(I don't remember what it showed when the initial issue with the docker container named 'recipesmysql8' was there)

And after starting the Spring Boot application: connection worked and tables and indexes were created!

Note: only from within the docker image can you connect like this: docker exec -it recipesmysql8v2 bash and execute mysql -u root -p after having issued ALTER USER 'root'@'localhost' IDENTIFIED BY 'nR128^n8f3kx';
Because running mysql -h 127.0.0.1 -P 3306 --protocol=tcp -u root -p from within the commandline WSL still gives: ERROR 1045 (28000): Access denied for user 'root'@'172.17.0.1' (using password: YES)

Probable cause: when you only have MySql in a Docker container, it is not in the same "network" as Docker containers, so the port 3306 is not reachable from outside Docker runtime. So you will have to tell Docker how you want to expose the port(s). Another way to solve this is to have the Spring Boot application also as a Docker application; and potentially configure it via docker-compose, see here for an example on how to do this: https://www.javainuse.com/devOps/docker/docker-mysql.



Wednesday, April 6, 2022

Configuring MySQL test-containers in your Spring Boot Java Integration Tests

Introduction

In your Integration Tests (IT) you often try to use the H2 in-memory database, to improve the speed of your integration tests. But on the other hand you want to mimic the production database as much as possible in your integration-tests.

Setting H2 in MySQL database compatibility mode tries to emulate MySQL as much as possible, but only a small subset of the differences are implemented. What for example not works correctly in H2 for JSON fields is that it escapes strings with "". For that reason you usually want to switch to for example starting a Docker database testcontainer in your IT tests, which uses a real MySQL database. With the Java-specific version in https://github.com/testcontainers/testcontainers-java.


Configuration

There are several good-to-know tips when configuring the testcontainers in your ITs.

  1. The simplest configuration is using a datasource URL in the Spring Boot properties file. This has the disadvantage that whatever database name you specify, the testcontainers library still creates a DB named 'test'. So below it will be named 'integration_test_db' you'd think, but it is still named 'test' when the IT runs:

    spring.datasource.url=jdbc:tc:mysql:5.7.32:///integration_test_db?sessionVariables=sql_mode='STRICT_TRANS_TABLES'&TC_MY_CNF=mysql&TC_INITSCRIPT=mysql/init_mysql_integration_tests.sql

    To be able to do everything on the started database, including giving it the name you want, use this URL (or see below the Java version). Notice the user 'root' in the URL:
    spring.datasource.url=jdbc:tc:mysql:5.7.32:///integration_test_db?user=root&password=&sessionVariables=sql_mode='STRICT_TRANS_TABLES'&TC_MY_CNF=mysql&TC_INITSCRIPT=mysql/init_mysql_integration_tests.sql

    Via: https://github.com/testcontainers/testcontainers-java/issues/932

  2. To initialize your database, specify the script via this extra datasource URL variable:

    TC_INITSCRIPT=mysql/init_mysql_integration_tests.sql

    Note the default directory it looks in is ..../resources for the scripts. So the full path is ..../resources/mysql/

  3. An example to prevent GROUP BY error from strict mode, add this in your TC_INITSCRIPT:

    SET GLOBAL sql_mode = 'STRICT_TRANS_TABLES';
    SET SESSION sql_mode = 'STRICT_TRANS_TABLES';


  4. To configure it in the IT Java class itself:

    @RunWith(SpringRunner.class)
    @SpringBootTest(classes = { SomeClassA.class, SomeClassB.class, ApplicationConfiguration.class}, webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
    @TestPropertySource(locations = {
            "classpath:/application-test-mysql.properties" })
    @ContextConfiguration(initializers = {ThisITClass.Initializer.class})

    static class Initializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
            public void initialize(ConfigurableApplicationContext configurableApplicationContext) {
                TestPropertyValues.of(
                        "spring.datasource.url=" + mySQLContainer.getJdbcUrl(),
                        "spring.datasource.username=" + mySQLContainer.getUsername(),
                        "spring.datasource.password=" + mySQLContainer.getPassword()
                ).applyTo(configurableApplicationContext.getEnvironment());
            }
        }

    @ClassRule
    public static MySQLContainer mySQLContainer = new MySQLContainer<>("mysql:5.7.31")
                .withUsername("root") // So now you can do a GRANT too for example
                .withPassword("") // Only possible for user 'root'
                .withEnv("MYSQL_ROOT_HOST", "%")
                .withDatabaseName("integration_test_db") // So this name will now be used, not 'test'
                .withInitScript("mysql/init_mysql_integration_tests.sql")

                ;

  5. The MySQL docker image used in the tests is retrieved from DockerHub https://hub.docker.com/_/mysql

Sunday, December 26, 2021

Sentry in Spring Boot application reports uncaught exception while @RestControllerAdvice is handling exception

Introduction

Even when you have a @RestControllerAdvice configured and you know it gets invoked, it can be that Sentry.io's monitoring library still reports it as an (uncaught) exception. Reason for this is that when you configure Sentry as specified without setting its order in the exception-handler-resolver chain, it's by default set as lowest, so invoked as first in the chain. 

Solution

Thus to fix this, set it to a higher order number. Safest seems to set it to Integer.MAX_VALUE, that way it will always be invoked as the latest.

The most recent Sentry integration examples can be found here for Spring and here for Spring Boot. Maybe you want to consider logging via Logback or Log4J too, see the related Sentry integration examples on the same page.

Another solution is overriding the getOrder() method for older Sentry versions. Below is an example, using the suggesting configuration from this old obsolete Sentry page.

@Configuration
@Slf4j
public class SentryConfig {
    @Bean
    public HandlerExceptionResolver sentryExceptionResolver() {

        return new SentryExceptionResolver() {
            @Override
            public ModelAndView resolveException(HttpServletRequest request,
                    HttpServletResponse response,
                    Object handler,
                    Exception ex) {
                log.info("Sentry resolving this exception: ", ex);
                // You could skip exceptions that are considered client-side errors and return null in that case so it is considered taken care of.
                return super.resolveException(request, response, handler, ex);
            }

            @Override
            public int getOrder() {
                // Ensure other resolver(s) can run first, otherwise this handler is invoked as first and thus reporting an issue for Sentry
                return Integer.MAX_VALUE;
            }
        };
    }
    @Bean
    public ServletContextInitializer sentryServletContextInitializer() {
        return new io.sentry.spring.SentryServletContextInitializer();
    }
}


Similar solutions mentioned here: https://stackoverflow.com/questions/48401974/how-to-have-my-own-error-handlers-before-sentry-in-a-spring-application


Thursday, November 19, 2020

Spring @Scheduled using DynamoDB AWS X-Ray throws SegmentNotFoundException: failed to begin subsegment

Introduction

AWS X-Ray is designed to automatically intercept incoming web requests, see at this introduction.  And also here. 

But when you start your own thread (either via a Runnable or plain new Thread() or @Scheduled), X-Ray cannot initialise itself: there are no web requests to intercept for it. Then it throws an exception like this: 

Suppressing AWS X-Ray context missing exception (SegmentNotFoundException): Failed to begin subsegment named 'AmazonDynamoDBv2': segment cannot be found.

In the above example the distributed DynamoDB Lock Client was used, which uses DDB in its implementation for acquiring and releasing a lock.

Regular web requests were not throwing this X-Ray exception.

Investigation

Amazon explains that crucial bit of knowledge that web requests are "automagically" setting up the X-Ray recorder a bit here. 

But that was not fully explaining it with an example. E.g only adding 

AWSXRay.beginSubsegment("AmazonDynamoDBv2") 

(and ending it) but that didn't fix it. That then gave this exception:

Suppressing AWS X-Ray context missing exception (SubsegmentNotFoundException): Failed to end subsegment: subsegment cannot be found.

My suspicion here is that the lock-client already closed the exactly same named subsegment "AmazonDynamoDBv2".
Also, I'm not creating any worker thread myself, Spring is doing it for me.

Note that you at least can avoid exceptions be thrown by setting environment variable AWS_XRAY_CONTEXT_MISSING   to LOG_ERROR. That will only log the above exception.

Solution

Creating a 'parent' segment and setting the trace entity and the subsegment did the job:

Entity parentSegment = AWSXRay.beginSegment("beginSegmentForSomeScheduledTask");
AWSXRay.getGlobalRecorder().setTraceEntity(parentSegment);
AWSXRay.beginSubsegment("AmazonDynamoDBv2");

I did test only creating the subsegment and only creating and setting the parentSegment. But those raised the exception again.
I did not further investigate whether the "AmazonDynamoDBv2" name of the subsegment is essential.
And of course the matching closeSegment() and closeSubsegment() calls of course; not the reverse order: close the last one begun as first.

This thread pointed me in the right direction. This was a next option I would have tried next: setting up my own filter to run it earlier in the (filter) chain; though the @Scheduled task of course does not have a filter. Another workaround would have been to put the X-Ray logging at a very high level:

logging.level.com.amazonaws.xray = SEVERE

Also helped in explaining is this X-Ray reported issue.

Update: additionally, the error was also due to DynamoDB Lock Client! I did not specify withCreateHeartbeatBackgroundThread() when creating the lock client, but the exception of X-Ray showed that it was trying to send a heartbeat. After explicitly setting withCreateHeartbeatBackgroundThread(false) the exception (and error) regarding segment cannot found was fully fixed.




Sunday, April 5, 2020

PACT provider test with Serverless Java 11 Lambda and JUnit 5 (Jupiter)

Introduction

This blogpost shows how to create a Pact provider test using:
  1. Pact 4.0.0
  2. AWS Serverless Lambda
  3. Java 11
  4. JUnit 5 (Jupiter)

Example Provider Test

Below is the example test class. Note how a mock service is used to simulate the endpoint defined (also) in serverless.yml.

package com.ttlnews.pact.tests.provider;

import au.com.dius.pact.provider.junit.Provider;
import au.com.dius.pact.provider.junit.State;
import au.com.dius.pact.provider.junit.loader.PactBroker;
import au.com.dius.pact.provider.junit.loader.PactBrokerAuth;
import au.com.dius.pact.provider.junit5.HttpTestTarget;
import au.com.dius.pact.provider.junit5.PactVerificationContext;
import au.com.dius.pact.provider.junit5.PactVerificationInvocationContextProvider;
import com.amazonaws.services.lambda.runtime.Context;
import lombok.extern.slf4j.Slf4j;
import net.jcip.annotations.NotThreadSafe;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.TestTemplate;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockserver.integration.ClientAndServer;

import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.mockserver.integration.ClientAndServer.startClientAndServer;
import static org.mockserver.matchers.Times.exactly;
import static org.mockserver.model.Header.header;
import static org.mockserver.model.HttpRequest.request;
import static org.mockserver.model.HttpResponse.response;
import static org.mockserver.model.JsonBody.json;

/**
 * Java 11 serverless provider Pact contract provider implementation using Pact with Junit5 (Jupiter) for provider SERVICE_A.
 * 
 * Dependencies needed:
 *
     <dependency>
     <groupId>au.com.dius</groupId>
     <artifactId>pact-jvm-consumer-junit5</artifactId>
     <version>4.0.0</version>
     <scope>test</scope>
     </dependency>
    
     <dependency>
     <groupId>au.com.dius</groupId>
     <artifactId>pact-jvm-provider-junit5</artifactId>
     <version>4.0.0</version>
     <scope>test</scope>
     </dependency>
 *
 * 
 */
@Provider(SERVICE_A)
// Below environment variables must be set when running this test
@PactBroker(host = "${PACT_HOST}",
        authentication = @PactBrokerAuth(username = "${PACT_HOST_USERNAME}", password = "${PACT_HOST_PASSWD}"))
@NotThreadSafe // Pact contract tests can't seem to handle methods running in parallel, so prevent maven failsafe/surefire plugin to run Pact tests in parallel
@Slf4j
public class ServiceAProviderTest {

    private static final String SERVICE_A = "service-A";
    
    private Context lambdaContext;
    private SomeRepository someRepository;

    private ClientAndServer mockClientAndServer;

    @BeforeEach
    void before(PactVerificationContext context) {

        lambdaContext = mock(Context.class);
        someRepository = new InMemoryMockedRepo();

        mockClientAndServer = startClientAndServer(8888);

        context.setTarget(new HttpTestTarget("127.0.0.1", 8888));

    }

    @AfterEach
    void stopServer() {
        mockClientAndServer.stop();
    }

    // This triggers the defined contract test(s) at the host PACT_HOST
    @TestTemplate
    @ExtendWith(PactVerificationInvocationContextProvider.class)
    void pactVerificationTestTemplate(PactVerificationContext context) {
        context.verifyInteraction();
    }

    @State("Create a new resource A")
    public void shouldCreateResourceA() {

        log.info("Set up state");

        // Given
        // Maybe some more mocking of 'lambdaContext' needed, depending on your case
        
        // Lambda being tested
        CreateResourceALambda createResourceALambda = new CreateResourceA(someRepository);

        CreateResourceALambdaRequest createResourceALambdaRequest = CreateResourceALambdaRequest.builder()
                .someValue("25")
                .build();

        // When
        final LambdaResult lambdaResult = createResourceALambda.executeRequest(createResourceALambdaRequest, lambdaContext);
        final String body = lambdaResult.getBody();
        assertNotNull(body);

        // Prepare the mockserver to return what the lambda returns when it is invoked (set up above in the 'body' variable) 
        // Of course the path to this CreateResourceALambda is defined in the serverless.yml, but that we can't access now. So need to repeat that
        // endpoint here.
        mockClientAndServer.when(
                request()
                        .withMethod("POST")
                        .withPath("/resources/")
                        .withHeaders(

                                header("x-request-trace-id"),  // Any value is fine
                                header("Authorization"),  // Any value is fine
                                header("Content-Type", "application/json")

                        )
                        .withBody(json("{someValue: '25'}")) // Indeed need to use this strange JSON format
                ,
                exactly(1))
                .respond(
                        // Response as defined by the matching Pact consumer test; in this case found at host PACT_HOST
                        response()
                                .withStatusCode(201)
                                .withHeaders(
                                        header("Content-type", "application/json; charset=utf-8"),
                                        header("Authorization") // Any value is fine
                                )
                                .withBody(body)
                );
    }
}
 
Migrating from your Pact tests from JUnit4 to Junit5 can be found here.
 
Note to self: use https://www.opinionatedgeek.com/codecs/htmlencoder to encode code, open the HTML view, and then wrap the code in <pre> open en close tag.

Wednesday, October 2, 2019

Snyk .snyk ignore file with multiple entries format

"Snyk enables you to find, and more importantly fix known vulnerabilities in your open source."


Some vulnerabilities you'd want to ignore, because for example they are (for you) either false positives, or there is no workaround for them (yet). You can specify them via the so-called CLI snyk ignore command, or in the UI manually, or via a .snyk ignore file. An introduction focused on NPM can be found here.
The format with an example for one entry is specified here: https://support.snyk.io/hc/en-us/articles/360000923498-How-can-I-ignore-a-vulnerability-

But what is the format for multiple entries? Most of the time with yaml (-like) formats you might expect a list, so each entry should be prefixed with a dash "-". But that is not the case here. The correct format for specifying multiple entries in the .snyk ignore file is:
version: v1.5.0
ignore:
  'SNYK-JAVA-xxxx-123456':
    - '* > com.abc:def':
      reason: 'Fix in progress'
      expires: '2020-01-05T08:00.00.000Z'
  'SNYK-JAVA-xxxx-678900':
    - '* > org.abc:ghi':
      reason: 'Windows issue, systems run on Linux'
      expires: '2020-02-06T09:00.00.000Z'
 
Note each block after the 'ignore' line is indented with two spaces.

Saturday, July 27, 2019

PACT Consumer Driven Contract Testing: how to allow any body in the response


Consumer Driven Contract testing is a way to ensure that services (such as an API provider and a client) can communicate with each other. Without contract testing, the only way to know that services can communicate is by using expensive and brittle integration tests.
PACT is a contract testing tool.  


For response matching you want to be as loose as possible with the matching for the response (will_respond_with(...)) though. This stops the tests being brittle on the provider side.
 Most of the time you don't care about the exact values of the (JSON or XML) response, but you do care about the types of the values, e.g a string or a number.
In that case you'll be using 'type matching'.

Sadly for JVM matchers there's no such handy method that with one invocation makes sure only types are validated, as is for example with Ruby/Groovy/Javascript/Node: Pact::SomethingLike.
Not much documentation on the Java/JVM matchers can be found at the official Pact site itself. Mostly the examples are for Ruby/Groovy/Javascript/Node.

For  the Java and Virtual Machine integration you'd use pact-jvm with matchers like .stringType() for the body using the PACT DSL or this lambda extension.

But hard to find was how to specify allowing either an empty body in the response or any data in the body in the response, while the consumer does not require a body at all (or in other words: is fine with an empty body or any fields in it).
Solution: for that you need to completely omit the .body() in the consumer contract definition.
If you specify a .body(new PactDslJsonBody()), the contract generated matcher will specify "body" : {}, and therefor requiring an empty body. And if then the provider (test) generates one or more fields in the response, you'll see this as the message in the failing test:


Expected an empty Map but received Map(..... fields added by provider ...)


So full example which accepts an empty body or a body with elements in it:

.consumer("Some Consumer")
.hasPactWith("Some Provider")
.given("a certain state on the provider")
    .uponReceiving("a request for something")
        .path("/hello")
        .method("POST")
.toPact()