Cutover from workflows API to DAG Launcher

[smohiudd]

Description

The workflows API is used to ingest datasets using Airflow DAGs. With SM2A we have access to the native airflow endpoints which can be used instead of the workflows. API. This ticket involves documenting the differences between Workflows API and SM2A endpoints and any changes needed in SM2A to ingest datasets similar to Workflows.

Acceptance Criteria

• Verify ingestion process using DAG Launcher/SM2A and parity with Workflows API

[smohiudd]

Proposal: use simple lambda as proxy for SM2A /dags/{dag_id}/dagRuns instead of using workflows api. Use eventbridge lambda as example with dag name and auth: https://github.com/NASA-IMPACT/veda-data-airflow/blob/dev/sm2a/infrastructure/functions/s3_event_bridge_to_sfn_execute/lambda_function.py#L13

[amarouane-ABDELHAK]

Another option may be to create an SM2A role using Airflow Role-Based Access Control (RBAC) with limited access to the Discover DAG. The allowed users would only be able to see the Discover DAG in the Airflow UI, trigger it using the form, and view the execution of each task. We can control who is assigned as a DAG Launcher using GitHub teams. In the future, we could use another OIDC tool, such as AWS Cognito or Keycloak.

DAGs to add RBAC role for are:

• VEDA-Discover
• VEDA-Collection-Pipeline
• Veda-Dataset-Pipeline

[amarouane-ABDELHAK]

Work will be documented here #269