Ingest 1000 soils data generated at JGI

[aclum]

Deliverable this task is associated with

GSP 2024 deliverable

RACI

Tag people in their roles

• Responsible: @mbthornton-lbl, @mflynn-lanl
• Accountable: @hubin-keio, @aclum
• Consulted: @aclum
• Informed: @aclum

Describe the the task

• stage in data for Gs0156736/JGI proposal ID 508306. There are 16 projects. We need to stage raw, filtered results and assembly. Annotation and binning will be re-done to be consistent with running annotation 5.2 on the genewiz data.

Criteria for completion

• Workflow execution activity records exist for ReadQcAnalysisActivity and MetagenomeAssembly with corresponding data objects including raw data as has_input for ReadQcAnalysisActivity

• 1000 soil - process metaG data: JGI & Genewiz #613

JGI File Staging steps:

• jgi_file_metadata.py
• file_restoration.py
• globus_file_transfer.py

Estimate people time

• [Hours or days of people time. 1 person, 4 hours]

Completion Date (Goal)

• March 27

Target Sprint Start & End Dates

• Start: March 11
• End: April 5

Tag Blocker/Contingent upon issues

• [Tagg issues]

[aclum]

Actively in progress based on slack messages, moving to the next sprint.

[mflynn-lanl]

Data files have been downloaded from JGI to Perlmutter and can be found at: /global/homes/n/nmdcda/m3408/aim2/dev/1000_soils/1000_soils_analysis_projects/

[mbthornton-lbl]

@mflynn-lanl Thank you for doing this! Does this mean we can close it as done?

[mflynn-lanl]

I think we should wait for it to be processed, just in case there are any problems

…

________________________________ From: Michael Thornton ***@***.***> Sent: Thursday, April 4, 2024 2:33 PM To: microbiomedata/issues ***@***.***> Cc: Flynn, Mark C ***@***.***>; Mention ***@***.***> Subject: [EXTERNAL] Re: [microbiomedata/issues] Ingest 1000 soils data generated at JGI (Issue #632) @mflynn-lanl<https://urldefense.com/v3/__https://github.com/mflynn-lanl__;!!Bt8fGhp8LhKGRg!E87CXDjgTBvz8nk-Kyipu2FMwfUtQ2juUBTlqTygek0Ss8FbyLe9ZIFMa3RZu0jEPrCEmniLY64gNzdWrKKPN8hg$> Thank you for doing this! Does this mean we can close it as done? — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/microbiomedata/issues/issues/632*issuecomment-2038156769__;Iw!!Bt8fGhp8LhKGRg!E87CXDjgTBvz8nk-Kyipu2FMwfUtQ2juUBTlqTygek0Ss8FbyLe9ZIFMa3RZu0jEPrCEmniLY64gNzdWrPRnbTEb$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AEXIUF2IP5NOFKIKRRCYFKLY3W2IBAVCNFSM6AAAAABEWXIYI6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZYGE2TMNZWHE__;!!Bt8fGhp8LhKGRg!E87CXDjgTBvz8nk-Kyipu2FMwfUtQ2juUBTlqTygek0Ss8FbyLe9ZIFMa3RZu0jEPrCEmniLY64gNzdWrBEXAWVw$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[mbthornton-lbl]

Data staging work completed.

Moving to In Review pending successful completion of #613

[aclum]

@mbthornton-lbl there is code in the nmdc-automation repo which should make the workflow execution activities and data objects for the JGI files. We need this for readsqc and assembly, annotation and binning will be run by NMDC so it has the same version as the genewiz data.
I think the code is nmdc_automation/run_process/run_import.py
I believe this is the corresponding required config file
https://github.com/microbiomedata/nmdc_automation/blob/main/configs/import.yaml

[aclum]

1000 soils jgi gold mappings.xlsx
We have one missing omics processing set record, see #665 but otherwise I believe this the first sheet on this excel is the format that run_import.py needs.

[ssarrafan]

@aclum can I remove this from the GSP/ECR board?

[aclum]

@ssarrafan yes.

[ssarrafan]

@mbthornton-lbl will be continuing to work on this in the next sprint per Slack message. Moving over.

[aclum]

Moving to the next sprint since this sprint is focused on re-iding.

[aclum]

backlog until re-iding is done.

[aclum]

was previously blocked on microbiomedata/nmdc_automation#274
Current blocker
microbiomedata/nmdc_automation#280

[aclum]

Blockers are resolved, i added a manfiest tsv file to the /global/homes/n/nmdcda/m3408/aim2/dev/1000_soils/1000_soils_analysis_projects/, this is ready for work.

[aclum]

some issues arose trying to import these records
microbiomedata/nmdc_automation#333
microbiomedata/nmdc_automation#332
microbiomedata/nmdc_automation#331

[aclum]

This is in a mixed state, 1/2 the projects where imported and 1/2 had their data_generation_set records updated but workflow_execution_set and corresponding data_object_set records were not generated.