Creating a Shaded Jar

Java applications of a certain complexity often run into dependency conflicts due to their nature of bringing in many external dependencies, each with their own set of transitive dependencies. One way of resolving this issue is to introduce a 'shaded Jar' - a single jar file including the library code, as well as the code of all of its dependencies, with the dependency package names renamed into a different namespace to avoid conflicts.

The following guide will shade your application using apache shade plugin. This plugin can be used to create a self-contained version of the application, which is a single Uber JAR that contains the application package and all of its dependency packages (both immediate and transitive dependencies). The same plugin can be used to relocate the conflicting dependency packages in this Uber JAR to prevent their path names from conflicting with those that ADB bring into the class-path. The resulting JAR after applying relocation to Uber JAR is called Shaded JAR.

Creating Uber JAR, example

To create Uber version of your application and to relocate conflicting Jackson dependency in the Uber JAR, add the following shaded plugin entry under <build><plugins> section of application POM.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <version>3.0.0</version>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>shade</goal>
      </goals>
      <configuration>
        <transformers>
          <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
            <mainClass><!-- YOUR_APPLICATION_MAIN_CLASS --></mainClass>
          </transformer>
          <!--Transforms META-INF/services (essential if you relocate com.azure classes)-->
          <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
        </transformers>
        <finalName>${project.arifactId}-${project.version}-shaded</finalName>
          <filters>
            <filter>
              <artifact>*:*</artifact>
              <excludes>
              <exclude>META-INF/maven/**</exclude>
              <exclude>META-INF/*.SF</exclude>
              <exclude>META-INF/*.DSA</exclude>
              <exclude>META-INF/*.RSA</exclude>
            </excludes>
          </filter>
        </filters>
        <relocations>
          <relocation>
           <pattern>com.fasterxml.jackson</pattern>
           <shadedPattern>org.example.shaded.com.fasterxml.jackson</shadedPattern>
          </relocation>
          <relocation>
            <!--Environment like Databricks may bring its own version of azure-core which may be incompatible with your Azure client libraries.
                Relocate azure-core to avoid collisions with it-->
            <pattern>com.azure</pattern>
            <shadedPattern>org.example.shaded.com.azure</shadedPattern>
          </relocation>

          <!--In Databricks 10.2 you nay also need to relocate reactor netty packages-->
          <relocation>
            <pattern>io.netty</pattern>
            <shadedPattern>ms.shaded.io.netty</shadedPattern>
          </relocation>
          <relocation>
            <pattern>reactor</pattern>
            <shadedPattern>ms.shaded.reactor</shadedPattern>
          </relocation>
        </relocations>
      </configuration>
    </execution>
  </executions>
</plugin>

then run:

mvn package

This will generate a shaded version of the application in target directory. For example: if your application POM has following maven co-ordinates defined:

<artifactId>adb-app</artifactId>
<version>1.0-SNAPSHOT</version>

then the shaded jar will be named as:

adb-app-1.0-SNAPSHOT-shaded.jar

This name is derived from the finalName entry in shaded plugin ${project.arifactId}-${project.version}-shaded.

The Jackson relocation configuration defined in the plugin will result in renaming the com.fasterxml.jackson package in the Uber JAR to com.microsoft.shaded.com.fasterxml.jackson and update all references to the classes from the original package.

Shading specific library

If you have a conflict within your application, shading dependency into the application JAR would not resolve a version conflict - it only forces a single shaded version of that dependency for all components.

In this case, you can create a new separate module wrapping library causing conflict directly or though one of its transitive dependencies and shading (relocating) conflicting dependency into it. The following steps show an example of shading and relocating Jackson libraries under a new JAR:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <version>${maven-shade-plugin-version}</version>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>shade</goal>
      </goals>
      <configuration>
        <!--Create shaded JAR only-->
        <shadedArtifactAttached>false</shadedArtifactAttached>
        <!--Remove original replaced dependencies-->
        <createDependencyReducedPom>true</createDependencyReducedPom>
        <!--Promotes transitive dependencies of removed dependencies to direct-->
        <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
        <relocations>
          <relocation>
            <pattern>com.fasterxml.jackson</pattern>
            <shadedPattern>org.example.shaded.com.fasterxml.jackson</shadedPattern>
          </relocation>
        </relocations>
      </configuration>
    </execution>
  </executions>
</plugin>

Shading and minimizing specific library

If you want to reduce the third library dependency shaded classes in the shaded jar, which means only shading the classes your library used, for the unused part will be dropped.

For this scenario, you can use a shading execution (id is minimize-shade) to only minimize the third dependency, use bcpkix-lts8on as example, use another shading execution (id is full-shade) to shade other specific library and exclude the bcpkix-lts8on that has been minimized, then the final jar will include the smallest shaded classes from bcpkix-lts8on. Here is the detail information of the example:

<dependencies>
  <dependency>
    <groupId>org.bouncycastle</groupId>
    <artifactId>bcpkix-lts8on</artifactId>
    <version>2.73.6</version>
    <optional>true</optional>
  </dependency>
  <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-json</artifactId>
    <version>1.3.0</version>
    <optional>true</optional>
  </dependency>
</dependencies>

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <version>3.6.0</version>
  <executions>
    <execution>
      <id>minimize-shade</id>
      <goals>
        <goal>shade</goal>
      </goals>
      <configuration>
        <shadeTestJar>false</shadeTestJar>
        <createSourcesJar>true</createSourcesJar>
        <shadeSourcesContent>true</shadeSourcesContent>
        <minimizeJar>true</minimizeJar>
        <artifactSet>
          <includes>
            <include>org.bouncycastle:*</include>
          </includes>
        </artifactSet>
        <filters>
          <filter>
            <artifact>org.bouncycastle:*</artifact>
            <excludes>
              <exclude>META-INF/*.SF</exclude>
              <exclude>META-INF/*.DSA</exclude>
              <exclude>META-INF/*.RSA</exclude>
              <exclude>META-INF/services/java.security.Provider</exclude>
            </excludes>
          </filter>
        </filters>
        <relocations>
          <relocation>
            <pattern>org.bouncycastle</pattern>
            <shadedPattern>com.azure.security.keyvault.jca.implementation.shaded.org.bouncycastle</shadedPattern>
          </relocation>
        </relocations>
        <transformers>
          <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
        </transformers>
      </configuration>
    </execution>
    <execution>
      <id>full-shade</id>
      <goals>
        <goal>shade</goal>
      </goals>
      <configuration>
        <shadeTestJar>false</shadeTestJar>
        <createSourcesJar>true</createSourcesJar>
        <shadeSourcesContent>true</shadeSourcesContent>
        <artifactSet>
          <excludes>
            <exclude>org.bouncycastle:*</exclude>
          </excludes>
        </artifactSet>
        <filters>
          <filter>
            <artifact>com.azure:azure-json</artifact>
            <excludes>
              <exclude>META-INF/*.SF</exclude>
              <exclude>META-INF/*.RSA</exclude>
            </excludes>
          </filter>
        </filters>
        <relocations>
          <relocation>
            <pattern>com.azure.json</pattern>
            <shadedPattern>com.azure.security.keyvault.jca.implementation.shaded.com.azure.json</shadedPattern>
          </relocation>
        </relocations>
        <transformers>
          <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
        </transformers>
      </configuration>
    </execution>
  </executions>
</plugin>