pyimagej image.init() FileNotFoundError: [Errno 2] No such file or directory: ~.m2/repository/io/scif/scifio-labeling/0.3.1/scifio-labeling-0.3.1.jar

[benwilks]

Hi,
I'm having trouble running pyimagej image.init() installed with conda/mamba in a docker container.

mamba create -n pyimagej -c conda-forge python=3.8 pyimagej openjdk=11 ipykernel
wget https://downloads.imagej.net/fiji/latest/fiji-linux64.zip 
unzip fiji-linux64.zip 
rm fiji-linux64.zip

Importing imagej works but when i run:

ij = imagej.init(os.path.join(base_dir, 'Fiji.app'), mode='headless')

I get the following error related to

FileNotFoundError: No such file or directory: ~/.m2/repository/io/scif/scifio-labeling/0.3.1/scifio-labeling-0.3.1.jar' -> ~/.jgo/io.scif/scifio-labeling/0.3.1/64d37f756ce9b31af4b9413862ddb4fdae7bd1d10bdfadd417d693b8642fc2e3/scifio-labeling-0.3.1.jar

-------------------------------------------------

Local path to ImageJ2 installation given: /home/jovyan/code/image-analysis/imagej/Fiji.app
Added 507 JARs to the Java classpath.
Adding jars from endpoints ['io.scif:scifio-labeling:0.3.1', 'net.imglib2:imglib2-imglyb:1.1.0']
Using endpoints ['io.scif:scifio-labeling:0.3.1', 'net.imglib2:imglib2-imglyb:1.1.0']
Getting verbose level: 2
Getting verbose level: 2
Returning expanded coordinate io.scif:scifio-labeling:0.3.1.
Returning expanded coordinate net.imglib2:imglib2-imglyb:1.1.0.
First time start-up may be slow. Downloaded dependencies will be cached for shorter start-up times in subsequent executions.
Executing: ('/opt/conda/envs/pyimagej/bin/mvn', '-B', '-f', '/home/jovyan/.jgo/io.scif/scifio-labeling/0.3.1/64d37f756ce9b31af4b9413862ddb4fdae7bd1d10bdfadd417d693b8642fc2e3/pom.xml', 'dependency:resolve', '-X')
Relevant mvn output: [INFO]    io.scif:scifio-labeling:jar:0.3.1:compile -- module io.scif.labeling [auto]
Linking source /home/jovyan/.m2/repository/io/scif/scifio-labeling/0.3.1/scifio-labeling-0.3.1.jar to target /home/jovyan/.jgo/io.scif/scifio-labeling/0.3.1/64d37f756ce9b31af4b9413862ddb4fdae7bd1d10bdfadd417d693b8642fc2e3/scifio-labeling-0.3.1.jar with link_type auto
Linking source /home/jovyan/.m2/repository/io/scif/scifio-labeling/0.3.1/scifio-labeling-0.3.1.jar to target /home/jovyan/.jgo/io.scif/scifio-labeling/0.3.1/64d37f756ce9b31af4b9413862ddb4fdae7bd1d10bdfadd417d693b8642fc2e3/scifio-labeling-0.3.1.jar with link_type hard
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[7], line 1
----> 1 ij = imagej.init(os.path.join(base_dir, 'Fiji.app'), mode='headless')

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/imagej/__init__.py:1200, in init(ij_dir_or_version_or_endpoint, mode, add_legacy, headless)
   1197     raise EnvironmentError("Sorry, the interactive mode is not available on macOS.")
   1199 if not sj.jvm_started():
-> 1200     success = _create_jvm(ij_dir_or_version_or_endpoint, mode, add_legacy)
   1201     if not success:
   1202         raise RuntimeError("Failed to create a JVM with the requested environment.")

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/imagej/__init__.py:1424, in _create_jvm(ij_dir_or_version_or_endpoint, mode, add_legacy)
   1421 sj.config.endpoints.extend(original_endpoints)
   1423 try:
-> 1424     sj.start_jvm()
   1425 except subprocess.CalledProcessError as e:
   1426     # Check to see if initialization failed due to "un-managed"
   1427     # imagej-legacy
   1428     err_lines = []

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/scyjava/_java.py:190, in start_jvm(options)
    188     endpoints = endpoints[:1] + sorted(endpoints[1:])
    189     _logger.debug("Using endpoints %s", endpoints)
--> 190     _, workspace = jgo.resolve_dependencies(
    191         "+".join(endpoints),
    192         m2_repo=scyjava.config.get_m2_repo(),
    193         cache_dir=scyjava.config.get_cache_dir(),
    194         manage_dependencies=scyjava.config.get_manage_deps(),
    195         repositories=repositories,
    196         verbose=scyjava.config.get_verbose(),
    197         shortcuts=scyjava.config.get_shortcuts(),
    198     )
    199     jpype.addClassPath(os.path.join(workspace, "*"))
    201 # HACK: Try to set JAVA_HOME if it isn't already.

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/jgo/jgo.py:712, in resolve_dependencies(endpoint_string, cache_dir, m2_repo, link_type, update_cache, force_update, manage_dependencies, repositories, shortcuts, verbose)
    709 relevant_jars.append(jar_file_in_workspace)
    711 try:
--> 712     link(
    713         os.path.join(m2_repo, *g.split("."), a, version, jar_file),
    714         jar_file_in_workspace,
    715         link_type=link_type,
    716     )
    717 except FileExistsError:
    718     # Do not throw exception if target file exists.
    719     pass

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/jgo/jgo.py:222, in link(source, link_name, link_type)
    220 except OSError as e:
    221     if e.errno != 18:
--> 222         raise e
    223 try:
    224     return link(source=source, link_name=link_name, link_type="soft")

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/jgo/jgo.py:219, in link(source, link_name, link_type)
    217 elif link_type.lower() == "auto":
    218     try:
--> 219         return link(source=source, link_name=link_name, link_type="hard")
    220     except OSError as e:
    221         if e.errno != 18:

File /opt/conda/envs/pyimagej/lib/python3.8/site-packages/jgo/jgo.py:214, in link(source, link_name, link_type)
    212     return os.symlink(source, link_name)
    213 elif link_type.lower() == "hard":
--> 214     return os.link(source, link_name)
    215 elif link_type.lower() == "copy":
    216     return shutil.copyfile(source, link_name)

FileNotFoundError: [Errno 2] No such file or directory: '/home/jovyan/.m2/repository/io/scif/scifio-labeling/0.3.1/scifio-labeling-0.3.1.jar' -> '/home/jovyan/.jgo/io.scif/scifio-labeling/0.3.1/64d37f756ce9b31af4b9413862ddb4fdae7bd1d10bdfadd417d693b8642fc2e3/scifio-labeling-0.3.1.jar'

Running ImageJ doctor doesn't show any problems

import imagej.doctor
imagej.doctor.checkup()

Checking Python:

--> Python executable = /opt/conda/envs/pyimagej/bin/python

Checking environment:
--> CONDA_PREFIX = /opt/conda/envs/pyimagej
--> Python executable matches Conda environment.

Checking Python dependencies:
--> jgo: /opt/conda/envs/pyimagej/lib/python3.8/site-packages/jgo/__init__.py
--> scyjava: /opt/conda/envs/pyimagej/lib/python3.8/site-packages/scyjava/__init__.py
--> imglyb: /opt/conda/envs/pyimagej/lib/python3.8/site-packages/imglyb/__init__.py
--> pyimagej: /opt/conda/envs/pyimagej/lib/python3.8/site-packages/imagej/__init__.py

Checking Maven:
--> Maven executable = /opt/conda/envs/pyimagej/bin/mvn
$ mvn -v
Apache Maven 3.9.3 (246d85a6ebec21fd45a0f9abd2a99906f92a4707)
Maven home: /opt/conda/envs/pyimagej/opt/maven
Java version: 11.0.15-internal, vendor: Oracle Corporation, runtime: /opt/conda/envs/pyimagej/lib/jvm
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.1.36-unraid", arch: "amd64", family: "unix"

Checking Java:
--> JAVA_HOME = /opt/conda/envs/pyimagej/lib/jvm
--> Java executable = /opt/conda/envs/pyimagej/bin/java
$ java -version
openjdk version "11.0.15-internal" 2022-04-19
OpenJDK Runtime Environment (build 11.0.15-internal+0-adhoc..src)
OpenJDK 64-Bit Server VM (build 11.0.15-internal+0-adhoc..src, mixed mode)

Great job! All looks good.

[ctrueden]

@benwilks Sorry for the slow reply. Something went sideways trying to download the scifio-labeling library from Maven, and then when PyImageJ (via jgo) tried to include it in the environment, it didn't work because the scifio-labeling-0.3.1.jar was not in the expected location of ~/.m2/repository. I wonder if this is because your Maven installation (which looks like it came from conda-forge) downloaded the library elsewhere without jgo understanding that.

With your pyimagej environment active, you can try to run:

$ mvn -X dependency:get -Dartifact=org.scijava:scijava-common:2.94.2 | grep url:
      url: file:///home/curtis/.m2/repository/
      url: https://repo.maven.apache.org/maven2

We want to see where the local .m2/repository resides. As you can see, mine is in my home directory, but maybe yours is somewhere else?

But you know, now that I wrote all this out, it actually might be something else going wrong: is it possible that Docker does not like the attempt to hard-link? From your log above, the attempt is:

ln /home/jovyan/.m2/repository/io/scif/scifio-labeling/0.3.1/scifio-labeling-0.3.1.jar' '/home/jovyan/.jgo/io.scif/scifio-labeling/0.3.1/64d37f756ce9b31af4b9413862ddb4fdae7bd1d10bdfadd417d693b8642fc2e3/scifio-labeling-0.3.1.jar`

which only works when the source and destination are on the same physical disk partition. Maybe Docker does not like hard links in general? I'm not sure, but you can try adding the following configuration step to your script:

echo "[settings]" >> ~/.jgorc
echo "links = soft" >> ~/.jgorc

which will change jgo's preferred behavior from hard links (ln) to soft/symlinks (ln -s).

Happy to help advise further if you are still stuck. Let me know whether any of the above ideas enable any progress.

[ctrueden]

One other thing: I just stumbled over #280, which is likely to bite you right now since you are also trying to build a wrapped local Fiji installation. In the meantime, you might want to consider using sc.fiji:fiji:2.14.0 instead as your endpoint, rather than curling down fiji-linux64.zip, as long as you don't need to customize your installation with additional plugins afterward.

[ctrueden]

Finally, if you have sufficient Python programming experience, and you are able to figure this out, it would be awesome if you could think about how to improve the imagej.doctor to catch this situation in future in case other people have the same problem.

[zhozhe]

I have also come across this error. Please refer to the following for the error.

The cause is that I set <localRepository>D:/Maven-Repository</localRepository> in the settings.xml under the .m2 folder.
This led to the inability of jgo.py to find my local repository, resulting in this error.

D:\anaconda3\envs\pyimagej\python.exe "C:\Users\21250\Documents\Tencent Files\2125019370\FileRecv\StarCraft2-RL-main\superbtest.py" 
Traceback (most recent call last):
  File "C:\Users\21250\Documents\Tencent Files\2125019370\FileRecv\StarCraft2-RL-main\superbtest.py", line 3, in <module>
    ij = imagej.init("D://Fiji.app")
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\anaconda3\envs\pyimagej\Lib\site-packages\imagej\__init__.py", line 1211, in init
    success = _create_jvm(ij_dir_or_version_or_endpoint, mode, add_legacy)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\anaconda3\envs\pyimagej\Lib\site-packages\imagej\__init__.py", line 1449, in _create_jvm
    sj.start_jvm()
  File "D:\anaconda3\envs\pyimagej\Lib\site-packages\scyjava\_jvm.py", line 138, in start_jvm
    _, workspace = jgo.resolve_dependencies(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\anaconda3\envs\pyimagej\Lib\site-packages\jgo\jgo.py", line 708, in resolve_dependencies
    link(
  File "D:\anaconda3\envs\pyimagej\Lib\site-packages\jgo\jgo.py", line 221, in link
    raise e
  File "D:\anaconda3\envs\pyimagej\Lib\site-packages\jgo\jgo.py", line 218, in link
    return link(source=source, link_name=link_name, link_type="hard")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\anaconda3\envs\pyimagej\Lib\site-packages\jgo\jgo.py", line 213, in link
    return os.link(source, link_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 3] 系统找不到指定的路径。: 'C:\\Users\\21250\\.m2\\repository\\net\\imglib2\\imglib2-imglyb\\1.1.0\\imglib2-imglyb-1.1.0.jar' -> 'C:\\Users\\21250\\.jgo\\net.imglib2\\imglib2-imglyb\\1.1.0\\0414d8274ab28cfc94250aa08011fc25de85a945899109fb7b2eca48a45f0983\\imglib2-imglyb-1.1.0.jar'
进程已结束，退出代码为 1

Solution: Modify line 17 in the file D:\anaconda3\envs\pyimagej\Lib\site-packages\scyjava\config.py to _m2_repo = "D:/Maven-Repository" instead of _m2_repo = pathlib.Path.home() / ".m2" / "repository". This should resolve the issue.

[ctrueden]

@zhozhe Thanks for the report and workaround!

It should also be possible to override the m2 repository without changing the code as follows:

import scyjava.config
scyjava.config.set_m2_repo("D:/Maven-Repository")

If it doesn't work for you, please let me know!

P.S. in looking into this, I noticed some jankiness at the scyjava and jgo levels around this issue; see scijava/scyjava#69 for details.

[ctrueden]

@benwilks Did you ever overcome the problems and get PyImageJ working in Docker? I'd like to close this issue if possible of course, but if there's something that needs to be done on the PyImageJ side, let us know!