No such file or directory: '/allen/programs/celltypes/workgroups/em-connectomics/danielk/em_lens_correction/test_data/_metadata_20180220175645_247488_8R_tape070A_05_20180220175645_reference_0_.json'

[zikai1]

Hi, thanks for providing such a useful tool.

When I try to study the asap-modules documentation and run
python3 -m asap.mesh_lens_correction.do_mesh_lens_correction
using the default example setting in do_mesh_lens_correction.py,

I enconter the following error:
No such file or directory:'/allen/programs/celltypes/workgroups/emconnectomics/danielk/em_lens_correction/test_data/_metadata_20180220175645_247488_8R_tape070A_05_20180220175645_reference_0_.json.

Dose it mean I have to provide my own data to use do_mesh_lens_correction.py? Is it possible to get the above test_data or could you please make the _metadata_20180220175645_247488_8R_tape070A_05_20180220175645_reference_0_.json a publicly available demo file, thank you very much! Since I am a newer in EM image processing, and thus your help will be quite appreciated.

[RussTorres]

Hello,

Thanks for reaching out. While you'll eventually want to collect your own highly-overlapping lens correction sets as described in the elife article https://doi.org/10.7554/eLife.76534 when using this module for your own data, you should be able to try out using it on some of our publicly available raw data.

We have a somewhat large upload of montage acquisitions and their associated lens corrections on data dryad (though the link below is currently broken, we are fixing it):
https://doi.org/10.5061/dryad.qjq2bvqhr

And I just uploaded a single lens correction reference set (~2GiB) to zenodo that can be used specifically for working with the lens correction:
https://doi.org/10.5281/zenodo.7600575

Additionally, the mesh_lens_correction available in asap-modules is a wrapper (including e.g. SIFT point matching) around the functionality found in the em_stitch repo https://github.com/AllenInstitute/em_stitch . We have a jupyter notebook example there showing how to use this tool with one of the lens corrections available in those uploads and hopefully shedding some light on how we use the metadata files in that repo.
https://github.com/AllenInstitute/em_stitch/blob/master/docs/notebooks/em_stitch_derive_lenscorrection.ipynb

Hope this helps!
Russel

[zikai1]

Hi,
Thanks for the timely response. I will follow your instructions.

Thank you again for providing such a useful tool.

[zikai1]

@RussTorres
Hi Professor Russel Torres,

I have test the data 21617_R1_166_T5_15_20201231140731_20201231140731 you shared in the em_stitch repo, and it works! However, when I test it by python -m asap.mesh_lens_correction.do_mesh_lens_correction in ASAP repo, it failed......

The error I encountered is:
renderapi.errors.ClientScriptError: client_script call ['/home/zhaomingyang/render/render-ws-java-client/src/main/scripts/run_ws_client.sh', '5G', 'org.janelia.render.client.ImportJsonClient', '--baseDataUrl', 'http://192.168.200.121:8080/render-ws/v1', '--owner', 'myzhao', '--project', 'lens_corr', '--stack', 'raw_lens_stack', '/tmp/tmptdbrqtj8.json'] failed

I imitate the example in asap.mesh_lens_correction.do_mesh_lens_correction to set my own --input_json file :

{
    "render": {
        "host": "192.168.200.121",
        "port": 8080,
        "owner": "myzhao",
        "project": "lens_corr",
        "client_scripts": "home/zhaomingyang/render/render-ws-java-client/src/main/scripts",
        "memGB": "2G"
    },
    "regularization": {
        "default_lambda": 1e5,
        "translation_factor": 1e-5,
        "lens_lambda": 1e-5
    },
    "input_stack": "raw_lens_stack",
    "output_stack": "lens_corrected_stack",
    "overwrite_zlayer": "True",
    "close_stack": "True",
    "z_index": 10,
    "metafile": "/home/zhaomingyang/data/21617_R1_166_T5_15_20201231140731_20201231140731/_metadata_20201231140731_21617_R1_166_T5_15_20201231140731_reference_0_.json",
    "match_collection": "raw_lens_matches",
    "nfeature_limit": 20000,
    "output_dir": "/home/zhaomingyang/data/tmp",
    "outfile": "lens_out.json",
    "output_json": "./mesh_lens_output.json",
    "mask_coords": [
        [ 0,100],
        [100,0],
        [3840, 0],
        [3840,3840],
        [0,3840]
    ],
    "mask_dir": "/home/zhaomingyang/data/masks_for_stacks"
}

I guess the error is due to the wrong setting of client_scripts in render. The client_scripts in example is "client_scripts": "/allen/aibs/pipeline/image_processing/volume_assembly/render-jars/production/scripts", BUT How to set my own correct client_scripts?

Thank you very much for your kind help, Professor Russel Torres.

Best wishes,
Mingyang

[RussTorres]

Apologies for missing this -- I believe your last comment is resolved since I see evidence of client scripts found in AllenInstitute/BigFeta#25 . To clarify it here, these scripts are from render, and are essentially pointing where to find run_ws_client.sh. Here is a permalink to the scripts in the render repo:
https://github.com/saalfeldlab/render/tree/773f3e12fd29a63e8ebb277eea8a9710ae94c6b9/render-ws-java-client/src/main/scripts