[SonarQube] Cannot attach javaagent from plugin

[IkenoXamos]

Leveraging the Community Branch Plugin requires both installing a plugin as well as attaching the provided jar as a javaagent to both the web server process as well as the compute engine process.

I have used the below properties.

override.yaml:

persistence:
  enabled: true
plugins:
  install:
  - "https://github.com/mc1arke/sonarqube-community-branch-plugin/releases/download/${plugin_version}/sonarqube-community-branch-plugin-${plugin_version}.jar"
sonarProperties:
  sonar.ce.javaAdditionalOpts: "-javaagent:/opt/sonarqube/extensions/plugins/sonarqube-community-branch-plugin-${plugin_version}.jar=ce"
  sonar.web.javaAdditionalOpts: "-javaagent:/opt/sonarqube/extensions/plugins/sonarqube-community-branch-plugin-${plugin_version}.jar=web"

where ${plugin_version} is being substituted for 1.8.1.

While standing up, I see that the plugin is downloaded by the install-plugins init-container:
$ kubectl logs -c install-plugins sonarqube-sonarqube-649bd6ff78-75z99 -n sonar

2022-02-03 22:36:46 URL:https://objects.githubusercontent.com/github-production-release-asset-2e65be/173807249/6e8b73be-c19b-4acb-ba28-6188de5870e8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220203T223645Z&X-Amz-Expires=300&X-Amz-Signature=283c90f8a8ea0133698a568fc80293a9160d497b093fbc8ed45a7ddc8811b4d9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=173807249&response-content-disposition=attachment%3B%20filename%3Dsonarqube-community-branch-plugin-1.8.1.jar&response-content-type=application%2Foctet-stream [12045563/12045563] -> "/opt/sonarqube/extensions/downloads/sonarqube-community-branch-plugin-1.8.1.jar" [1]

However, the resulting pod fails with:
$ kubectl logs sonarqube-sonarqube-649bd6ff78-75z99 -n sonar

2022.02.03 22:36:48 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
2022.02.03 22:36:48 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:44633]
2022.02.03 22:36:48 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch
2022.02.03 22:36:48 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
warning: no-jdk distributions that do not bundle a JDK are deprecated and will be removed in a future release
2022.02.03 22:36:51 INFO  es[][o.e.n.Node] version[7.16.2], pid[25], build[default/tar/2b937c44140b6559905130a8650c64dbd0879cfb/2021-12-18T19:42:46.604893745Z], OS[Linux/5.4.172-90.336.amzn2.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/11.0.11/11.0.11+9]
2022.02.03 22:36:51 INFO  es[][o.e.n.Node] JVM home [/opt/java/openjdk]
2022.02.03 22:36:51 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseG1GC, -Djava.io.tmpdir=/opt/sonarqube/temp, -XX:ErrorFile=../logs/es_hs_err_pid%p.log, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djna.tmpdir=/opt/sonarqube/temp, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=COMPAT, -Des.enforce.bootstrap.checks=true, -Xmx512m, -Xms512m, -XX:MaxDirectMemorySize=256m, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/sonarqube/elasticsearch, -Des.path.conf=/opt/sonarqube/temp/conf/es, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=false]
2022.02.03 22:36:52 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2022.02.03 22:36:52 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2022.02.03 22:36:52 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2022.02.03 22:36:52 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2022.02.03 22:36:52 INFO  es[][o.e.p.PluginsService] no plugins loaded
2022.02.03 22:36:52 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/opt/sonarqube/data (/dev/nvme2n1)]], net usable_space [9.6gb], net total_space [9.7gb], types [ext4]
2022.02.03 22:36:52 INFO  es[][o.e.e.NodeEnvironment] heap size [512mb], compressed ordinary object pointers [true]
2022.02.03 22:36:52 INFO  es[][o.e.n.Node] node name [sonarqube], node ID [cMoGVIhIQe-UuHvlIJhBeQ], cluster name [sonarqube], roles [data_frozen, master, remote_cluster_client, data, data_content, data_hot, data_warm, data_cold, ingest]
2022.02.03 22:36:56 INFO  es[][o.e.t.NettyAllocator] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
2022.02.03 22:36:56 INFO  es[][o.e.d.DiscoveryModule] using discovery type [zen] and seed hosts providers [settings]
2022.02.03 22:36:57 INFO  es[][o.e.g.DanglingIndicesState] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
2022.02.03 22:36:57 INFO  es[][o.e.n.Node] initialized
2022.02.03 22:36:57 INFO  es[][o.e.n.Node] starting ...
2022.02.03 22:36:58 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:44633}, bound_addresses {127.0.0.1:44633}
2022.02.03 22:36:58 INFO  es[][o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2022.02.03 22:36:58 INFO  es[][o.e.c.c.Coordinator] setting initial configuration to VotingConfiguration{cMoGVIhIQe-UuHvlIJhBeQ}
2022.02.03 22:36:58 INFO  es[][o.e.c.s.MasterService] elected-as-master ([1] nodes joined)[{sonarqube}{cMoGVIhIQe-UuHvlIJhBeQ}{E_FlMQkFTMyLyzCSavBFPw}{127.0.0.1}{127.0.0.1:44633}{cdfhimrsw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{sonarqube}{cMoGVIhIQe-UuHvlIJhBeQ}{E_FlMQkFTMyLyzCSavBFPw}{127.0.0.1}{127.0.0.1:44633}{cdfhimrsw}]}
2022.02.03 22:36:58 INFO  es[][o.e.c.c.CoordinationState] cluster UUID set to [mpFSnJH3Q8mM0mupsHIheA]
2022.02.03 22:36:58 INFO  es[][o.e.c.s.ClusterApplierService] master node changed {previous [], current [{sonarqube}{cMoGVIhIQe-UuHvlIJhBeQ}{E_FlMQkFTMyLyzCSavBFPw}{127.0.0.1}{127.0.0.1:44633}{cdfhimrsw}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
2022.02.03 22:36:58 INFO  es[][o.e.h.AbstractHttpServerTransport] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2022.02.03 22:36:58 INFO  es[][o.e.n.Node] started
2022.02.03 22:36:58 INFO  es[][o.e.g.GatewayService] recovered [0] indices into cluster_state
2022.02.03 22:36:58 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2022.02.03 22:36:58 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/sonarqube]: /opt/java/openjdk/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube/temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -javaagent:/opt/sonarqube/extensions/plugins/sonarqube-community-branch-plugin-1.8.1.jar=web -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/sonar-application-8.9.6.50800.jar:/opt/sonarqube/lib/jdbc/postgresql/postgresql-42.2.19.jar org.sonar.server.app.WebServer /opt/sonarqube/temp/sq-process15592240587763802852properties
Error opening zip file or JAR manifest missing : /opt/sonarqube/extensions/plugins/sonarqube-community-branch-plugin-1.8.1.jar
Error occurred during initialization of VM
agent library failed to init: instrument
2022.02.03 22:36:58 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [web]: 1
2022.02.03 22:36:58 INFO  app[][o.s.a.SchedulerImpl] Process[web] is stopped
2022.02.03 22:36:58 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 143
2022.02.03 22:36:58 INFO  app[][o.s.a.SchedulerImpl] Process[es] is stopped
2022.02.03 22:36:58 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

Notably, the 3 lines

Error opening zip file or JAR manifest missing : /opt/sonarqube/extensions/plugins/sonarqube-community-branch-plugin-1.8.1.jar
Error occurred during initialization of VM
agent library failed to init: instrument

Seems to indicate that the plugin jar file is not available for instrumenting. My best guess is that the additional Java opts are applied before the plugin has been received, so therefore fails to find it.

If I remove the sonarProperties configuration, the helm chart deploys successfully, and the plugin is shown as installed, however it does not function as intended, since sonarqube was not instrumented.

[mkostelcev]

Hello, man! I solved this problem. You need to set this

sonarProperties:
  sonar.ce.javaAdditionalOpts: "-javaagent:./lib/common/sonarqube-community-branch-plugin-${plugin_version}.jar=ce"
  sonar.web.javaAdditionalOpts: "-javaagent:./lib/common/sonarqube-community-branch-plugin-${plugin_version}.jar=web"

Problem in folder, while sonarqube is starting up, folder extensions/plugins does not contain any file. Only after startup and initialization (and this happens after setting the javaagent parameter)

but lib/common already contains this plugin on current stage of initialization (if it's copied correctly, of course). And, if you refer to the right path that exists, everything will start and work :)

Enjoy!