Java – Run Gradle tests using multiple Java toolchains

I have a Gradle project that uses the Java version specified by the toolchain API:

val minimumJava = JavaLanguageVersion.of(8)
val maximumJava = JavaLanguageVersion.of(16)
java {
    toolchain {
        languageVersion.set(minimumJava)
        vendor.set(JvmVendorSpec.ADOPTOPENJDK)
    }
}

I want to be able to compile with the minimum supported Java version and then run tests with all JDKs supported by the project.
I tried the following, but apparently only the original tests were executed, all the others were not, even though the required JDK was downloaded and set up correctly:

for (javaVersion in JavaLanguageVersion.of(minimumJava.asInt() + 1)..maximumJava) {
    val base = tasks.test.get()
    val testTask = tasks.register<Test>("testUnderJava${javaVersion.asInt()}") {
        javaLauncher.set(
            javaToolchains.launcherFor {
                languageVersion.set(javaVersion)
            }
        )
        classpath = base.classpath
        testClassesDirs = base.testClassesDirs
        isScanForTestClasses = true
    }
    tasks.test.configure { finalizedBy(testTask) }
}

This is running in a dumb terminal:

❯ TERM=dumb ./gradlew test testUnderJava10 --rerun-tasks --scan
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=on

<<<SNIP>>>

> Task :testClasses

> Task :test
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=on

Gradle Test Executor 4 STANDARD_OUT
    ~~~ Kotest Configuration ~~~
    -> Parallelization factor: 1
    -> Concurrent specs: null
    -> Global concurrent tests: 1
    -> Dispatcher affinity: true
    -> Default test timeout: 600000ms
    -> Default test order: Sequential
    -> Default isolation mode: SingleInstance
    -> Global soft assertions: false
    -> Write spec failure file: false
    -> Fail on ignored tests: false
    -> Spec execution order: SpecExecutionOrder
    -> Remove test name whitespace: false
    -> Append tags to test names: false
    -> Extensions
      - io.kotest.engine.extensions.SystemPropertyTagExtension
      - io.kotest.core.extensions.RuntimeTagExtension
      - io.kotest.engine.extensions.RuntimeTagExpressionExtension


org.danilopianini.template.test.Tests > A greeting should get printed STARTED

org.danilopianini.template.test.Tests > A greeting should get printed STANDARD_OUT
    [:hello=SUCCESS]

    > Task :hello
    Hello from Danilo Pianini

    BUILD SUCCESSFUL in 2s
    1 actionable task: 1 executed


org.danilopianini.template.test.Tests > A greeting should get printed PASSED

<<<Other tests have no output!>>>

> Task :testUnderJava9
> Task :testUnderJava8
> Task :testUnderJava16
> Task :testUnderJava15
> Task :testUnderJava14
> Task :testUnderJava13
> Task :testUnderJava12
> Task :testUnderJava11
> Task :testUnderJava10

BUILD SUCCESSFUL in 23s
36 actionable tasks: 36 executed

<<<SNIP>>>

From build scan , it seems that instead of executing tests, tests executed with JDK8. I’m confused, the docs says this should be simple:

tasks.register<Test>("testsOn14") {
    javaLauncher.set(javaToolchains.launcherFor {
        languageVersion.set(JavaLanguageVersion.of(14))
    })
}

I think I’ve found the root cause of the problem I’m experiencing, and I’ll post a solution in case anyone else is having a similar issue.
I have the following test configuration:

tasks.test {
    useJUnitPlatform()
    testLogging {
        showStandardStreams = true
        showCauses = true
        showStackTraces = true
        events(*org.gradle.api.tasks.testing.logging.TestLogEvent.values())
        exceptionFormat = org.gradle.api.tasks.testing.logging.TestExceptionFormat.FULL
    }
}

A task named test is being directed to . useJunitPlatform() This setting is not automatically propagated to all subsequent Test tasks (of course). So, in this case, the solution is simply to use:

tasks.withType<Test> { 
    // Same configuration as above
}

Update 2022-03-16

I decided to create a multi-JVM testing plugin for Gradle so that all the test tasks were created and fewer boilerplate files were needed across projects.