Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update pr.yml to open veda-config pr #216

Open
wants to merge 68 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
599c066
feat: update pr.yml to open veda-config pr and publish collections
botanical Dec 17, 2024
3d4b2d0
feat: add test dataset-config
botanical Dec 17, 2024
af17bf4
fix: add debugging
botanical Dec 17, 2024
608b690
fix: add more debugging statements
botanical Dec 17, 2024
865928c
fix: add environment to job
botanical Dec 17, 2024
8cb9631
fix: testing
botanical Dec 17, 2024
a86093e
fix: add debugging for directory
botanical Dec 17, 2024
f351a1a
fix: add github debugging
botanical Dec 17, 2024
4376637
fix: add logs to create-mdx-files step
botanical Dec 17, 2024
5e5a1a6
fix: add step to output files
botanical Dec 17, 2024
b256e48
fix: update ls path
botanical Dec 17, 2024
1013d4d
fix: replace git clone with checkout
botanical Dec 17, 2024
3c7b34b
fix: typo
botanical Dec 17, 2024
94f6e4a
fix: merge steps
botanical Dec 17, 2024
3eb2059
fix: add environment
botanical Dec 17, 2024
63886d1
fix: try to copy generated file to new directory in new repo
botanical Dec 18, 2024
31c3aed
fix: add modifications for copying untracked file
botanical Dec 18, 2024
dbf55f4
fix: create source directory
botanical Dec 18, 2024
4d18d7c
fix: update copying commands
botanical Dec 18, 2024
91dc819
fix: update command
botanical Dec 18, 2024
788ca5f
fix: add debugging for copying file
botanical Dec 18, 2024
5ef917f
fix: add more logging, remove extraneous copy
botanical Dec 18, 2024
f3fa2b7
fix: remove ls path
botanical Dec 18, 2024
9184260
fix: attempt to create pr
botanical Dec 18, 2024
ff68de3
fix: update pr creation command to specify org/repo directly
botanical Dec 18, 2024
41c9dd1
fix: add debugging
botanical Dec 18, 2024
189221f
fix: check branch
botanical Dec 18, 2024
a5ba677
fix: change token reference
botanical Dec 18, 2024
98fd0ca
fix: add branch debugging
botanical Dec 18, 2024
44ba78b
fix: add debugging token access step
botanical Dec 18, 2024
87dcf47
fix: try another syntax for token supplying
botanical Dec 18, 2024
ad26820
fix: update token commands
botanical Dec 18, 2024
5b1846d
fix: more debugging target repo
botanical Dec 18, 2024
618d743
fix: try checking out target repo another way
botanical Dec 18, 2024
54c93e2
fix: typo
botanical Dec 18, 2024
ecfc08d
fix: remove prev debugging token step
botanical Dec 18, 2024
921bf57
fix: change cloning step
botanical Dec 18, 2024
8bab675
fix: cd into cloned repo
botanical Dec 18, 2024
0fc433c
fix: simplify
botanical Dec 18, 2024
3ec9ea2
fix: update untracked file copying step
botanical Dec 18, 2024
d37fb9b
fix: typo
botanical Dec 18, 2024
b830338
fix: cd into cloned repo
botanical Dec 18, 2024
b006851
fix: reorder steps, see if it fixes issue
botanical Dec 18, 2024
378c91f
fix: print working directory
botanical Dec 18, 2024
1ee2549
fix: condense steps
botanical Dec 18, 2024
d45aa7e
fix: add commands to add and commit file
botanical Dec 18, 2024
578119a
fix: modify commit message, add logs
botanical Dec 18, 2024
19a9c73
fix: authenticate git
botanical Dec 18, 2024
21bc63b
fix: add other steps to update comment in veda-data pr
botanical Dec 18, 2024
1eb663e
fix: add debugging to new step
botanical Dec 18, 2024
a84ebe9
fix: update env var setting
botanical Dec 18, 2024
a914ca2
fix: update comment-id references
botanical Dec 18, 2024
ebedabd
fix: update output to include commentId
botanical Dec 18, 2024
3d5548a
fix: echo comment_id
botanical Dec 18, 2024
094c917
fix: update comment id extraction
botanical Dec 19, 2024
302c90a
fix: add anayeaye changes
botanical Dec 19, 2024
3002432
fix: clean up logging
botanical Dec 19, 2024
4613805
fix: update step name
botanical Dec 19, 2024
d2b8503
fix: update to include github actor and try to extract collection
botanical Dec 19, 2024
10cfebd
fix: reformat file
botanical Dec 19, 2024
bd3fcd3
fix: update to handle list of collections
botanical Dec 19, 2024
d1d6cd2
fix: escape special chars
botanical Dec 19, 2024
86d7422
fix: add more cleaning to collection id
botanical Dec 19, 2024
f7ae3c5
fix: add debugging for collection id extraction
botanical Dec 19, 2024
cc0d8be
fix: try another approach for updating gh body
botanical Dec 19, 2024
c6dfa31
fix: update pr_url
botanical Dec 19, 2024
795a031
fix: cleanup by extracting pr title and body
botanical Dec 19, 2024
1c72609
fix: use -e flag for echo to fix newlines
botanical Dec 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 123 additions & 14 deletions .github/workflows/pr.yml
Original file line number Diff line number Diff line change
@@ -30,6 +30,7 @@ jobs:
environment: staging
outputs:
publishedCollections: ${{ steps.publish-collections.outputs.success_collections }}
commentId: ${{ steps.init-comment.outputs.COMMENT_ID }}
steps:
- uses: actions/checkout@v4

@@ -128,16 +129,17 @@ jobs:

# Track successful publications
all_failed=true
success_collections=()
declare -a success_collections=()
status_message='### Collection Publication Status
'

for file in "${ADDED_FILES[@]}"; do
for file in ${ADDED_FILES}; do
echo $file
if [ -f "$file" ]; then
dataset_config=$(jq '.' "$file")
collection_id=$(jq -r '.collection' "$file")

echo "Publishing $collection_id"
response=$(curl -s -w "%{http_code}" -o response.txt -X POST "$publish_url" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AUTH_TOKEN" \
@@ -171,7 +173,7 @@ jobs:
fi

# Output only successful collections to be used in subsequent steps
echo "success_collections=$(IFS=','; echo "${success_collections[*]}")" >> $GITHUB_OUTPUT
echo "success_collections=${success_collections[*]}" >> $GITHUB_OUTPUT

# Update PR comment
CURRENT_BODY=$(gh api -H "Authorization: token $GITHUB_TOKEN" /repos/${{ github.repository }}/issues/comments/$COMMENT_ID --jq '.body')
@@ -203,14 +205,15 @@ jobs:
** ❌ The workflow run failed. [See logs here]($WORKFLOW_URL)**"
gh api -X PATCH -H "Authorization: token $GITHUB_TOKEN" /repos/${{ github.repository }}/issues/comments/$COMMENT_ID -f body="$UPDATED_BODY"

create-mdx-files:
create-mdx-files-and-open-pr:
runs-on: ubuntu-latest
environment: staging
needs: publish-new-datasets
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Use output from dataset-publication-and-configuration
- name: Use output from publish-new-datasets
run: |
echo "The output from the previous step is: ${{ needs.publish-new-datasets.outputs.publishedCollections }}"

@@ -220,19 +223,125 @@ jobs:
PUBLISHED_COLLECTION_FILES: ${{ needs.publish-new-datasets.outputs.publishedCollections }}
run: |
echo $PUBLISHED_COLLECTION_FILES
collection_ids=""
pip install -r ./scripts/requirements.txt
for file in "${PUBLISHED_COLLECTION_FILES[@]}"
do
python3 ./scripts/generate-mdx.py "$file"
for file in ${PUBLISHED_COLLECTION_FILES}; do
collection_id=$(python3 ./scripts/generate-mdx.py "$file")
collection_id=$(echo "$collection_id" | sed 's/^["\s]*//;s/["\s]*$//')
echo "Processed collection ID: $collection_id"
collection_ids="$collection_ids$collection_id,"
done
# Remove trailing comma
collection_ids=${collection_ids%,}
echo "Final collection_ids: $collection_ids"
echo "collection_ids=${collection_ids}" >> $GITHUB_ENV

open-veda-config-pr:
runs-on: ubuntu-latest
needs: create-mdx-files
steps:
- name: Open veda-config PR
- name: Set up Variables
run: |
echo "NO-OP. Placeholder for future job that will open a Pull Request in veda-config for a dashboard preview for the new/changed datasets."
echo "VEDA_CONFIG_REPO=${{ vars.VEDA_CONFIG_REPO_ORG }}/${{ vars.VEDA_CONFIG_REPO_NAME }}" >> $GITHUB_ENV

- name: Clone veda-config repository
run: |
git clone https://github.com/${{ env.VEDA_CONFIG_REPO }}.git
ls

- name: Copy untracked mdx files to veda-config
run: |
echo "Copying untracked .mdx files to veda-config repository"
ls ./ingestion-data/dataset-mdx/
mkdir -p datasets
find ingestion-data/dataset-mdx/ -name '*.mdx' -exec cp {} veda-config/datasets/ \;

- name: Create veda-config PR with changes
id: create-pr
env:
GITHUB_TOKEN: ${{ secrets.VEDA_CONFIG_REPO_ACCESS_TOKEN }}
COMMENT_ID: ${{ needs.publish-new-datasets.outputs.commentId }}
PUBLISHED_COLLECTION_FILES: ${{ steps.publish-collections.outputs.success_collections }}
run: |
cd veda-config
git config --global user.name "github-actions[bot]"
git config --global user.email "github-actions[bot]@users.noreply.github.com"
git remote set-url origin https://${{ secrets.VEDA_CONFIG_REPO_ACCESS_TOKEN }}@github.com/${{ env.VEDA_CONFIG_REPO }}

files_string=$(IFS=$'\n'; echo "${PUBLISHED_COLLECTION_FILES[*]}")
hash=$(echo -n "$files_string" | md5sum | cut -d ' ' -f 1)
NEW_BRANCH="add-dataset-$hash"

git fetch origin
if git ls-remote --exit-code --heads origin $NEW_BRANCH; then
git push origin --delete $NEW_BRANCH
fi
git checkout -b $NEW_BRANCH

git status
git add .
git commit -m "feat: add MDX files for dataset(s) [Automated workflow]"
git push origin $NEW_BRANCH

# Convert the comma-separated list into bullet points
collection_bullet_points=""
IFS=',' read -ra IDs <<< "$collection_ids"

# Extract the first collection ID
first_collection_id="${IDs[0]}"
for id in "${IDs[@]}"; do
collection_bullet_points+="- $id\n"
done

pr_title="Add dataset(s) - $first_collection_id [Automated PR by ${{ github.actor }}]"
body="### Add dataset(s) - $first_collection_id [Automated PR by ${{ github.actor }}]\n\n$collection_bullet_points"

echo "$body"
PR_URL=$(GITHUB_TOKEN=${{ secrets.VEDA_CONFIG_REPO_ACCESS_TOKEN }} gh pr create -R ${{ env.VEDA_CONFIG_REPO }} -H $NEW_BRANCH -B develop --title "$pr_title" --body "$(echo -e "$body")")

echo "PR_URL=$PR_URL" >> $GITHUB_OUTPUT
echo "PR creation succeeded!"

# Updates the comment with a link to the above PR
- name: Update PR comment with PR creation result
if: success()
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
COMMENT_ID: ${{ needs.publish-new-datasets.outputs.commentId }}
run: |
PR_URL=${{ steps.create-pr.outputs.PR_URL }}
CURRENT_BODY=$(gh api -H "Authorization: token $GITHUB_TOKEN" /repos/${{ github.repository }}/issues/comments/$COMMENT_ID --jq '.body')
UPDATED_BODY="$CURRENT_BODY

**A PR has been created with the dataset configuration: 🗺️ [PR link]($PR_URL)**"
gh api -X PATCH -H "Authorization: token $GITHUB_TOKEN" /repos/${{ github.repository }}/issues/comments/$COMMENT_ID -f body="$UPDATED_BODY"

- name: Update PR comment on PR creation failure
if: failure() && steps.create-pr.outcome == 'failure'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
COMMENT_ID: ${{ needs.publish-new-datasets.outputs.commentId }}
run: |
CURRENT_BODY=$(gh api -H "Authorization: token $GITHUB_TOKEN" /repos/${{ github.repository }}/issues/comments/$COMMENT_ID --jq '.body')
UPDATED_BODY="$CURRENT_BODY

**Failed ❌ to create a PR with the dataset configuration. 😔 **"
gh api -X PATCH -H "Authorization: token $GITHUB_TOKEN" /repos/${{ github.repository }}/issues/comments/$COMMENT_ID -f body="$UPDATED_BODY"

# If the workflow fails at any point, the PR comment will be updated
- name: Update PR comment on overall workflow failure
if: failure() && steps.create-pr.outcome != 'failure'
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
COMMENT_ID: ${{ needs.publish-new-datasets.outputs.commentId }}
run: |
WORKFLOW_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
CURRENT_BODY=$(gh api -H "Authorization: token $GITHUB_TOKEN" /repos/${{ github.repository }}/issues/comments/$COMMENT_ID --jq '.body')
UPDATED_BODY="$CURRENT_BODY

# Output WORKFLOW_URL to logs for verification
echo "Workflow URL: $WORKFLOW_URL"

** ❌ The workflow run failed. [See logs here]($WORKFLOW_URL)**"
gh api -X PATCH -H "Authorization: token $GITHUB_TOKEN" /repos/${{ github.repository }}/issues/comments/$COMMENT_ID -f body="$UPDATED_BODY"

echo "Updated Comment Body: $UPDATED_BODY"

publish-to-prod-on-pr-merge:
if: ${{ github.event_name == 'pull_request' && github.event.action == 'closed' && github.event.pull_request.merged == true }}
31 changes: 31 additions & 0 deletions ingestion-data/staging/dataset-config/test.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"collection": "hls-swir-falsecolor-composite-THIRD-TEST",
"title": "HLS SWIR FalseColor Composite",
"spatial_extent": {
"xmin": -156.75,
"ymin": 20.80,
"xmax": -156.55,
"ymax": 20.94
},
"temporal_extent": {
"startdate": "2023-08-08T00:00:00Z",
"enddate": "2023-08-08T23:59:59Z"
},
"data_type": "cog",
"license": "CC0-1.0",
"description": "HLS falsecolor composite imagery using Bands 12, 8A, and 4.",
"is_periodic": false,
"time_density": "day",
"sample_files": [
"s3://veda-data-store-staging/maui-fire/Lahaina_HLS_2023-08-08_SWIR_falsecolor_cog.tif",
"s3://veda-data-store-staging/maui-fire/Lahaina_HLS_2023-08-13_SWIR_falsecolor_cog.tif"
],
"discovery_items": [
{
"discovery": "s3",
"prefix": "maui-fire/",
"bucket": "veda-data-store-staging",
"filename_regex": "(.*)SWIR_falsecolor(.*).tif$"
}
]
}
3 changes: 3 additions & 0 deletions scripts/generate-mdx.py
Original file line number Diff line number Diff line change
@@ -125,3 +125,6 @@ def safe_open_w(path):
)
with safe_open_w(output_filepath) as ofile:
ofile.write(new_content)

collection_id = input_data["collection"]
print(collection_id)