New output directories in IntelliJ IDEA 2017 causes problems in Gradle tasks

To be able to add build cache, Gradle 4 introduced new compilation-output directory layout. Since version 4, every language present in your source set has its own output directory. This means that your Java file main/java/JavaClass.java will be compiled into build/classes/java/main/JavaClass.class and your Kotlin file will end up in build/classes/kotlin/main/KotlinClass.class. That is the reason, why Gradle's SourceSetOutput.getClassesDir() method, returning a single path became deprecated and a new method getClassesDirs() returning collection was added.

Unfortunately IDEA does not support multiple output directories per source set and Gradle's change caused some problems. To address that, JetBrains changed an existing output directory to new one that differs from Gradle's so there wouldn't be any interference.

Problem

The problem is, sometimes you need to access classes compiled by IDEA from a Gradle task. For example if you are doing an instrumentation or some kind of other byte code manipulation. In my case I am using ActiveJDBC and Ebean ORMs, which both needs to instrument compiled classes. Of course, I could use javaagent but that's not the point.

So, when I launch an application from IDEA, the process is following:

1. Classes are compiled by IDEA, into out/classes/... directory (formerly it was build/classes/...).
2. Gradle instrumentation task is executed. The task looks for compiled classes in build/classes/... but none is found.
3. Application is started and fails shortly after with an error message "Are you sure, your classes has been instrumented?".

Solution #1 (insufficient in some cases)

To solve the problem, Vladislav Soronka of JetBrains suggested adding idea plugin into Gradle build script. The plugin allows to reconfigure IDEA's output directories.

apply plugin: 'idea'

idea {
    module {
        outputDir file('build/classes/main')
        testOutputDir file('build/classes/test')
    }
}

Unfortunately this sets the same output directory for classes and resources. So instead of having build/classes directory for your class files and build/resources for your resources, you will end up with everything in the build/classes. Of course, former build/resources directory is not goint to be put on classpath of Java application started by IDEA.

In my case this caused problems with JavaScript and CSS styles preprocessing, which I run as another Gradle task.

Solution #2 (final)

Best solution I was able to come with is to create a parametrized build. That means I've modified my instrumentation and resources preprocessing scripts to use one set of output directories when executed by Gradle command line client, and other set of directories when started from IDEA. In my case, I've used project properties to determine whether the build was started from IDEA. Environment variables instead of project properties can be used as well.

First of all, I've created idea.gradle build script. The script contains a simple resolve method. Gradle does not allow functions to be shared between build scripts, so I had to define it as a closure in an extension property.

// IDEA 2017 changed output directories from Gradle compliant build/... to
// out/... because of compatibility issues with Gradle 4. Because of that, the
// instrumentation, less compilation or JavaScript routes generation doesn't
// work.
//
// To fix the issue we need to tell postprocessing scripts where to find
// compiled classes and/or output resources (because of classpath). This
// can be done by passing -Pidea2017 argument to Gradle.
//
// gradle -Pidea2017 instrumentModels
//
ext {
    resolveOutputPath = { project, sourceSet ->
        gradleOutput = project.sourceSets.main.output
        idea2017OutputPath = project.file('out/production').toPath()
        boolean idea2017 = project.hasProperty('idea2017')

        switch (sourceSet) {
            case "classes":
                return idea2017 ? idea2017OutputPath.resolve('classes') : gradleOutput.classesDir.toPath()
            case "resources":
                return idea2017 ? idea2017OutputPath.resolve('resources') : gradleOutput.resourcesDir.toPath()
                break;
            default:
                throw new IllegalArgumentException("Unknown source set " + sourceSet)
        }
    }
}

Then, in my instrumentation build script I defined output directory simply by calling the closure. This is a snippet from my ActiveJDBC instrumentation build script:

apply from 'idea.gradle'

Instrumentation instrumentation = new Instrumentation()
instrumentation.outputDirectory = resolveOutputPath(project, "classes")
...

Finally, I specified project property in a "before launch" run configuration.

Gradle 4 and the future

As it was mentioned before, Gradle 4 introduced multiple output directories. That means: a method project.sourceSets.main.output.classesDir is currently marked as deprecated and will be removed soon. Some kind of refactoring towards new classesDirs method will be necessary in the future. For now everything should work without problems.

Shared test sources in Gradle multi-module project

Basic assumptions

Let's suppose following project structure.

:root-module
|    +--- main source set
|    \--- test source set
\--- :module1
     |    +--- main source set
     |    \--- test source set
     \--- :core-module
              +--- main source set
              \--- test source set

Shared test-utility classes shouldn't be placed in test source set

In some legacy projects I've seen utility classes to be located in test source set. This is problematic in multi-project applications because Gradle does not pull test sources to a dependent module. So module1 won't "see" classes from core-module's test source set. This makes sense because you don't want to have module1's test-classpath flooded with core-modules's test classes.

Note: IDEA 14 does not respect this rule and it pull test sources to a dependent module. This behaviour has been fixed lately.

Test-utility classes shouldn't be placed in main source set

For obvious reasons you don't want to pollute main source set with test classes. In main source set there should be production code only. Tests are usually executed on application's build phase so there is no need to put them into final build.

In Spring application test classes shouldn't be located in component-scanned package

This is mandatory for Spring developers who uses component scan. In some cases you may want to create annotated class used exclusively by tests.

IDEA places all source sets marked as sources to classpath. So if you place your test-only annotated class to a package scanned by Spring your application context will ends up polluted.

Shared test classes should be kept in module where it belongs

Because of previous point some people have tendency to put shared test classes to a stand-alone module. Although core-module-test (main) -> core-module (main), core-module (test) -> core-module-test (main) is legal construct in Gradle you don't want to complicate your project's dependency structure by creating "unnecessary" modules.

Module dependency resolution basics in Gradle and IDEA 14 and 16

Gradle

Gradle allows you to define dependency between two module configurations.

// Declared in root-module
dependencies {
    // Current module's testCompile configuration depends on another module's default configuration.
    testCompile project(':module1')
    // Current module's configuration depends on another module's test configuration.
    testCompile project(path: ':core-module', configuration: 'test')
}

Dependency to a default configuration of another module pulls module's dependencies and sources to a dependent module. Dependency to a test configuration pulls only dependencies without sources. So in our case root-module won't see any core-module's test classes.

IDEA 14

In contrast to Gradle the IDEA 14 pulls all sources to a dependent module. Even test sources.

If you open Gradle project IDEA will behave oddly. It will let you run tests that Gradle won't even compile. Unfortunately IDEA 14 doesn't have any feature that can be used to fix this so you have to think about it when you will write your test.

IDEA 16

Latest version of IDEA does not pull test sources to a dependent module which is consistent behaviour to Gradle. It also allows you to define dependency to a specific source set. You just have to choose create separate module per source set option when importing Gradle project. It has been added lately to harmonize IDEA's dependency resolution with Gradle.

Using shared test sources by dependent module

Declaring dependency to a module's output directory

In projects having utility test-classes in test source set there is usual hot-fix to add testCompile dependency to other module's output directory. So in our example the module1's tests will depend on core-module's compiled test classes.

// Declared in module1
dependencies {
  testCompile project(':core-module').sourceSets.test.output
}

This is quick solution but it has some cons:

• In your IDE you have to rebuild shared classes on every change to refresh dependencies. Remember IDEA 16 does not pull test sources.
• It's against the requirements in first chapter of this article.

Placing utility classes to a custom source set

The good solution is to create custom source set for shared test-classes and declare dependencies to this source set from dependent modules. This approach allows you to keep classes in modules where it belongs still separated from main and test sources. This approach is used by Gradle project itself.

Implementation is quite straightforward. First create script plugin that will add new testFixtures source set and appropriate configurations to a module.

Then apply script plugin on a module and declare necessary dependencies.

// Declared in core-module
apply from: 'testFixtures.gradle'

dependencies {
  testFixturesCompile('junit:junit:4.12')
}

Finally add dependency to the newly created source set.

// Declared in module1
dependencies {
    testCompile project(path: ':core-module', configuration: 'testFixturesUsageCompile')
}

Note for IDEA users: There is a small drawback. If you open the project without create separate module per source set option your custom-source set will be available in main source set. If you use a shared test-class in main source set IDEA won't complain about it but Gradle won't be able to compile. So you have to be careful about what you are using.

Gradle multi-project using cross dependent modules

I have a framework which consists of several modules. On top of this framework I'd like to build various projects where each project can use one or more modules of this framework. In real world these projects can represent framework's implementations for customers. Framework's modules can depend on each other. For this task I decided to use Gradle build system.

Part of this article is an example project on GitHub.

Setting up the build

I've created three modules that are shared by two root projects. Project1 and project2. Second project doesn't use one of the modules as it shown by following diagram. Both root projects has very similar build scripts so I will focus on project1 for the rest of this article.

:project1
+--- :module1
|    \--- :core-module
\--- :module2
     \--- :core-module

:project2
+--- :module1
     \--- :core-module

Root project's build script contains buildscript block which introduces plugin dependency and other common configuration used by Gradle when compiling/building project. The build script applies spring-boot-multi-project on the root project. This plugin is wrapper of the Gradle Spring Boot plugin and enhances it's functionality. The plugin also adds new task that generates serialized version of dependency graph. We will get to the dependency graph later.

There are three modules (sub-projects). A core module which is Spring Boot project and two modules (module1 and module2) which depends on core module and extends its functionality. These modules forms the framework. The framework is just a collection of libraries so it can't work as a standalone application.

Core module depends on Spring Boot starter project so to be compiled it needs to have compile dependency configuration set to spring-boot-starter. To be able to test this module it also need to have declared testCompile dependency on spring-boot-starter-test.

Note that module's build scripts does not contain repository configuration or does not apply any plugin. Repository configuration is included in spring-boot-multi-project plugin which is applied in root project's build script. This plugin adds Maven Central Repository and JitPack repository to every module including root project.

Module1 depends on core module. And because core module contains Spring Boot project it's not necessary to declare Spring Boot compile dependency in module1. Compile dependencies are transitive so all dependencies from core module are going to be placed on classpath of module1 too.

This rule doesn't apply to testCompile dependencies. When a module (subproject in Gradle's terminology) is tested it's tested separately. This means module is first compiled with all it's dependencies (not root project's dependencies) and it's own testCompile dependencies (not a root project's or subproject's dependencies) and then tests are executed. This means three things.

1. You need to add necessary testCompile dependencies to each subproject's build script.
2. If a module depends on a resource which is not added in compile dependencies you need to add this resource as a testCompile dependency to build script of the module. For example the resource can be a configuration file located in root project. To add directory as a testCompile dependency you can use file command testCompile files("path/to/resource").
3. If you want to use some shared class in your tests you will need to place it to main source set or a new source set and declare a dependency to it. In this example a source set called testFixtures is created in core module. Support for testFixtures source set is added in spring-boot-multi-project plugin.

Of course Gradle's multi-project build has to contain settings.gradle file which tells Gradle where all modules (in Gradle's terminology sub projects) are located.

Project dependencies in application

Sometimes it is useful to know module dependencies in the application. For example if you want to execute data model updating scripts you will need to know which script should run first (core project's) and which last (root project's which depends on everything else).

To do this you can use project dependencies file generated by discoverProjectDependencies task. With this graph you can easily sort resources on classpath in certain order as it is shown in ProjectDependencyManager service.

Running the project

To start projects or to build it you can experiment with tasks provided by Gradle in project1 or project2 directories. For example:

gradle discoverProjectDependencies - to serialize project dependencies into a file
gradle bootRun - to run the application
gradle build - to build jar file
gradle test - to run all tests

If you'd like to open project in IntelliJ's IDEA use the import Gradle project functionality instead of simple open project. Import project will open all framework modules as a project modules so you'll have all required source codes available via project panel. You can also use IDEA's Gradle plugin to easily modify build scripts.

When running project in IDEA please make sure that your run configuration does have root project selected in use classpath of module option.

Be aware of if you don't add any source code into Gradle project then Gradle won't generate empty build directories for this project. Unfortunately it adds these non-existing build directories to a classpath when bootRun or a simmilar task is executed. Invalid paths breaks Java's class loader and it causes problems with loading libraries which usually ends up with java.io.FileNotFoundException: class path resource [] cannot be resolved to URL because it does not exist error message. This can happen in early stages of a project when you create an empty module with some static content but without any Java code.