Tasking manager fastapi #6649

prabinoid · 2024-12-02T11:49:46Z

What type of PR is this? (check all applicable)

Related Issue

Related discussions:

#6264

https://community.openstreetmap.org/t/migrating-tasking-manager-from-flask-to-fastapi-and-psycopg2-to-asyncpg/116543

Describe this PR

This PR includes the TM fastapi migration and covers the commits from the initial phases of the migration.

This PR is made for the code review of the migration and is NOT be merged currently.

Signed-off-by: eternaltyro <[email protected]>

- Build multi-stage container images on specific targets - in this case 'prod' - Better naming for jobs Signed-off-by: eternaltyro <[email protected]>

- Fix outputs format - Experiment with metadata parsing in different ways - Add labels back to container metadata Signed-off-by: eternaltyro <[email protected]>

Signed-off-by: eternaltyro <[email protected]>

…managers assignment fixed and message module fixes

…nd favourites fixes

…organisation manager fixed

… aware * fix: All Teams Pagination fixed * fix: Json response for task split small area error message * fix: Timestamps serialization in messages, comments, and notifications

…ct messages.

…ment.

* Project creation date for creation date sorting * Email services fixed * User statistics for validation time spent.

* Image upload in comment section. * Last updated utc serialization in my tasks of my contributions

…nct task

* Project mapping type filter fixed.

* Partners CRUD * Mapswipe and statistics. * Project partnerships. * Request user display name made None if not authenticated. * Routes updated for partners, partnerships and statistics.

sonarcloud · 2024-12-02T11:51:07Z

Quality Gate failed

Failed conditions
1 Security Hotspot
E Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

spwoodcock · 2024-12-02T15:43:32Z

Dockerfile

@@ -28,7 +25,7 @@ FROM base as extract-deps
 RUN pip install --no-cache-dir --upgrade pip
 WORKDIR /opt/python
 COPY pyproject.toml pdm.lock README.md /opt/python/
-RUN pip install --no-cache-dir pdm==2.18.1
+RUN pip install --no-cache-dir pdm==2.8.0


Small thing, but it might be worth updating pdm to 2.18.1 here to avoid downgrading on merge

spwoodcock · 2024-12-02T16:46:30Z

backend/api/annotations/resources.py

+router = APIRouter(
+    prefix="/projects",
+    tags=["projects"],
+    dependencies=[Depends(get_db)],


I'm not sure this is needed here. Some endpoints may not need a db connection and this would create one anyway. I'm not sure how you would access the database connection variable like this anyway

spwoodcock · 2024-12-02T16:48:08Z

backend/api/annotations/resources.py

+    """
+    Get all task annotations for a project
+    ---
+    tags:


Entirely optional, but you could always delete this docstrings. Fastapi generates the OpenAPI docs using typing anyway, so it's a bit redundant. It also means you need to remember to update the docstring if you update variables

spwoodcock · 2024-12-02T16:51:47Z

backend/api/issues/resources.py

+    category_dto = await MappingIssueCategoryService.get_mapping_issue_category_as_dto(
+        category_id, db
+    )
+    return category_dto.model_dump(by_alias=True)


This will work, but the preferred approach to a manual model dump is probably using response_model on the endpoint decorator. This will handle serialisation and dumping

spwoodcock · 2024-12-02T16:56:25Z

backend/api/notifications/resources.py

+    sort_by = request.query_params.get("sortBy", "date")
+    sort_direction = request.query_params.get("sortDirection", "desc")
+    message_type = request.query_params.get("messageType", None)
+    from_username = request.query_params.get("from")


This is an interesting approach for endpoint query params. It should work, but typically you would define query params as function arguments and they would be parsed as such. If there is confusion, it's possible to explicitly define arguments as fastapi.Query

spwoodcock · 2024-12-02T16:57:32Z

backend/api/organisations/campaigns.py

+        async with db.transaction():
+            await CampaignService.create_campaign_organisation(
+                organisation_id, campaign_id, db
+            )
            message = "campaign with id {} assigned for organisation with id {}".format(


F-strings are a bit nicer to read, but not a big deal 👍

spwoodcock · 2024-12-02T16:59:35Z

backend/api/system/applications.py

+    return token.model_dump(by_alias=True)
+
+
+@router.patch("/authentication/applications/{application_key}/")


Using a PATCH method here seems a bit out of place, but maybe I am missing something

spwoodcock · 2024-12-02T17:01:19Z

That's a lot of code changed! Well done on this - lots of great work 🙌

Just a few minor comments

kaditya97 and others added 30 commits November 27, 2024 09:56

feat: initial setup for fastapi

812d8fd

fix: basic setup

4e73655

feat: added pyinstrument for api performance testing

151d878

feat: asyncio implementation

04dd738

fix: minor import issues

93e0f08

feat: new docker files with pdm integration

5af39f7

fix: projects all api

4fb15fb

fix: countries api working

63d5255

Fix backend container name on ECS

9f438a1

Signed-off-by: eternaltyro <[email protected]>

Fix image name

f5d2a6c

Signed-off-by: eternaltyro <[email protected]>

Make deploy job depend on completion of build job

c535e98

Signed-off-by: eternaltyro <[email protected]>

Test image output from build job

cc8dfa4

Signed-off-by: eternaltyro <[email protected]>

Fix image link

5219dcd

Signed-off-by: eternaltyro <[email protected]>

Fix interpolation in workflow

769a6aa

Signed-off-by: eternaltyro <[email protected]>

Move to new Task Definition family

877a506

Signed-off-by: eternaltyro <[email protected]>

Fix github actions output step

3a389ce

Signed-off-by: eternaltyro <[email protected]>

Better naming and targeted build

121332b

- Build multi-stage container images on specific targets - in this case 'prod' - Better naming for jobs Signed-off-by: eternaltyro <[email protected]>

Fix output format

9b3e6a7

- Fix outputs format - Experiment with metadata parsing in different ways - Add labels back to container metadata Signed-off-by: eternaltyro <[email protected]>

Fix syntax error in workflow

5131dce

Signed-off-by: eternaltyro <[email protected]>

Fix metadata parse

1e0de89

Signed-off-by: eternaltyro <[email protected]>

Build metadata fixes

bfa8850

Signed-off-by: eternaltyro <[email protected]>

Move shell variables to workflow step env

6e2bb82

Signed-off-by: eternaltyro <[email protected]>

Fix image_name for push and cleanup Dockerfile

735603a

Signed-off-by: eternaltyro <[email protected]>

Suppress apt output properly

206712b

Signed-off-by: eternaltyro <[email protected]>

Fix output ref

765ca15

Signed-off-by: eternaltyro <[email protected]>

Bug fixes

e5c2a17

Signed-off-by: eternaltyro <[email protected]>

fix: add json to ignored types in config

ef02f9d

fix: frontend update

d25f557

hot-fix: migrations error fix for multiple entry

87c5386

feat: Campaigns get api for fastapi app

7de60b5

prabinoid and others added 23 commits November 27, 2024 10:34

All messages, organisation retrieve and task activities fixes

cf1507c

Project and notification duplication handled, organisation teams and …

b1e0f08

…managers assignment fixed and message module fixes

Messages, teams, comments, notification modules and project filters a…

0c14df4

…nd favourites fixes

Teams managers and members assignment and retrieve query fixed

f2c6705

Task dto last updated attribute made timezone aware and add teams by …

bce7821

…organisation manager fixed

Project clone and transfer project ownership refactored

8c38fe5

Added root validator for project info dto

4aa995f

Split task url and geom fixed

473a5cf

Project, organisation teams deletion and other fixes

0c10901

Http exception handler in the app for token expiration handling

a917b38

Register user, retrieve org by id and slug and user interests refactored

22fd980

feat(backend): Timestamps made utcnow and serialization done with utc…

54e0d42

… aware * fix: All Teams Pagination fixed * fix: Json response for task split small area error message * fix: Timestamps serialization in messages, comments, and notifications

fix: Join team by request

fc495ce

Task and project card last contributions fixed and list dto for proje…

920bb5c

…ct messages.

fix: Project search and deletion.

a6b81c0

fix: clear priority areas, send message after validation, project com…

3495697

…ment.

fix: Update teams and formatting

11e8d49

* Team messaging background task,

f3ae9dc

* Project creation date for creation date sorting * Email services fixed * User statistics for validation time spent.

fix: Invalid token exception in TokenAuthBackend

d41b8d9

* Send message to contributors and background task.

c372e4c

* Image upload in comment section. * Last updated utc serialization in my tasks of my contributions

Revert user task fixed and get latest task activity now returns disti…

9d2021f

…nct task

* Send message after comment, parse usernames and mentions.

a007d5d

* Project mapping type filter fixed.

* Rebased against develop.

0dc540a

* Partners CRUD * Mapswipe and statistics. * Project partnerships. * Request user display name made None if not authenticated. * Routes updated for partners, partnerships and statistics.

prabinoid requested a review from spwoodcock December 2, 2024 11:49

prabinoid self-assigned this Dec 2, 2024

github-actions bot added scope: backend scope: infrastructure labels Dec 2, 2024

spwoodcock reviewed Dec 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tasking manager fastapi #6649

Tasking manager fastapi #6649

prabinoid commented Dec 2, 2024

sonarcloud bot commented Dec 2, 2024

spwoodcock Dec 2, 2024

spwoodcock Dec 2, 2024

spwoodcock Dec 2, 2024

spwoodcock Dec 2, 2024

spwoodcock Dec 2, 2024

spwoodcock Dec 2, 2024

spwoodcock Dec 2, 2024

spwoodcock commented Dec 2, 2024

		return token.model_dump(by_alias=True)


		@router.patch("/authentication/applications/{application_key}/")

Tasking manager fastapi #6649

Are you sure you want to change the base?

Tasking manager fastapi #6649

Conversation

prabinoid commented Dec 2, 2024

What type of PR is this? (check all applicable)

Related Issue

Describe this PR

sonarcloud bot commented Dec 2, 2024

Quality Gate failed

spwoodcock Dec 2, 2024

Choose a reason for hiding this comment

spwoodcock Dec 2, 2024

Choose a reason for hiding this comment

spwoodcock Dec 2, 2024

Choose a reason for hiding this comment

spwoodcock Dec 2, 2024

Choose a reason for hiding this comment

spwoodcock Dec 2, 2024

Choose a reason for hiding this comment

spwoodcock Dec 2, 2024

Choose a reason for hiding this comment

spwoodcock Dec 2, 2024

Choose a reason for hiding this comment

spwoodcock commented Dec 2, 2024