Dynamic Pagination in CPT Dashboard #135

MVarshini · 2024-11-19T16:54:38Z

Type of change

Description

The [PROD]_JOBS_API accepts size and offset params

Related Tickets & Documents

Related Issue # PANDA-651
Closes #

Checklist before requesting a review

I have performed a self-review of my code.
If it is a core feature, I have added thorough tests.

Testing

Please describe the System Under Test.
Please provide detailed steps to perform tests related to this code change.
How were the fix/results from this change verified? Please provide relevant screenshots or results.

jaredoconnell · 2024-11-20T16:25:59Z

backend/app/api/v1/commons/hce.py

        return jobs
-    return jobs
+    return ({'data':jobs, 'total': response['total']})


Why do we have the output format differ between the empty and non-empty case?
It looks like this results in the need to have if statements for the empty case in several spots.

@MVarshini maybe we can just have this check at one place from where this virtual function is being invoked?

Updated the non-empty case too.

jaredoconnell · 2024-11-20T16:30:50Z

backend/app/api/v1/endpoints/cpt/cptJobs.py

@@ -37,7 +37,11 @@
            },)
 async def jobs(start_date: date = Query(None, description="Start date for searching jobs, format: 'YYYY-MM-DD'", examples=["2020-11-10"]),
               end_date: date = Query(None, description="End date for searching jobs, format: 'YYYY-MM-DD'", examples=["2020-11-15"]),
-               pretty: bool = Query(False, description="Output contet in pretty format.")):
+               pretty: bool = Query(False, description="Output contet in pretty format."),


Typo: content

jaredoconnell · 2024-11-20T17:15:48Z

backend/app/api/v1/endpoints/cpt/maps/hce.py

@@ -2,8 +2,8 @@
 from datetime import date

 ################################################################
-# This will return a DataFrame from HCE required by the CPT
-#  endpoint, it contians the following columns:
+# This will return a Dictionary with from HCE required by the CPT


The inclusion of "with" here seems odd.

jaredoconnell · 2024-11-20T17:25:37Z

backend/app/services/search.py

+                        # return await self.scan_indices(self.new_es, self.new_index, query, timestamp_field, start_date, end_date, size)
+                    # else:


Why is this commented out?

I'm concerned that this change might inadvertently impact other scenarios, even though I've verified it against most cases. If it doesn't affect any other cases then these lines can be taken out or it can be reverted.

dbutenhof

I didn't get all the way through, but here's a bunch ...

backend/app/api/v1/commons/hce.py

backend/app/api/v1/endpoints/cpt/cptJobs.py

dbutenhof · 2024-11-20T20:43:17Z

backend/app/api/v1/endpoints/cpt/cptJobs.py

            except Exception as e:
                print(f"Error fetching data for product {product}: {e}")
-
+
+    num = 0 if totalJobs is None else int(totalJobs)


totalJobs is an int and it's not optional, so either the parameter type hint is wrong, or most of this statement isn't meaningful. Why is it checking for None? Why is it converting to int?

I'm not really sure what totalJobs does here. The relationship of num and total is a bit odd; a comment would probably help to clarify the intent.

num is 0 if totalJobs is passed in as 0 or None. If totalJobs is 0 (not None) then we compute total by summing across products; and then we set totalJobs to the max of those two, and report it as the response["total"] ... what purpose does this serve? I guess I'm missing something here...

backend/app/api/v1/endpoints/cpt/cptJobs.py

dbutenhof · 2024-11-20T20:57:47Z

backend/app/api/v1/endpoints/cpt/maps/hce.py

+        df["ciSystem"] = "Jenkins"
+        df["testName"] = df["product"] + ":" + df["test"]
+        df["product"] = df["group"] 
+        df["jobStatus"] = df['result'].apply(lambda x: "SUCCESS" if x == 'PASS' else "FAILURE")


Mixing double-quote and single-quote indices isn't really a good idea. (If you ran black it would convert all the single quoted strings to double-quotes.) While Python allows both, mixing them in a module (much less in a line) is unnecessarily confusing.

And, as a general maintainability/readability suggestion to the maintainers: after all the outstanding PRs are merged and stabilized, it'd be a great idea to

run black and Prettier on all Python and JavaScript/CSS files, and

add a CI lint check GitHub Action that'll fail a PR if it breaks the style rules. (Of course adding additional lint checks with flake8/ES lint/CodeQL would be a great idea too.)

Submit this as a PR with no functional changes; it might be large, (depending on how bad the code is), but it'd be straightforward to review.

Will follow it!

backend/app/api/v1/endpoints/cpt/maps/ocp.py

backend/app/api/v1/endpoints/ocm/ocmJobs.py

backend/app/api/v1/endpoints/ocp/ocpJobs.py

frontend/src/components/organisms/Pagination/index.jsx

dbutenhof · 2024-11-21T20:43:30Z

backend/app/api/v1/endpoints/cpt/maps/hce.py

+        df["ciSystem"] = "Jenkins"
+        df["testName"] = df["product"] + ":" + df["test"]
+        df["product"] = df["group"] 
+        df["jobStatus"] = df['result'].apply(lambda x: "SUCCESS" if x == 'PASS' else "FAILURE")


And, as a general maintainability/readability suggestion to the maintainers: after all the outstanding PRs are merged and stabilized, it'd be a great idea to

run black and Prettier on all Python and JavaScript/CSS files, and

add a CI lint check GitHub Action that'll fail a PR if it breaks the style rules. (Of course adding additional lint checks with flake8/ES lint/CodeQL would be a great idea too.)

Submit this as a PR with no functional changes; it might be large, (depending on how bad the code is), but it'd be straightforward to review.

dbutenhof

Lots of good cleanup, but I think you added a few bugs while you were at it. 😆

dbutenhof · 2024-12-02T19:58:51Z

backend/app/api/v1/commons/hce.py

    if len(jobs) == 0:
-        return jobs
+        return({'data':jobs, 'total': 0})
    return ({'data':jobs, 'total': response['total']})


Is this necessary? if len(jobs) is 0 doesn't that mean len(tasks) was 0? And doesn't that mean that response["total"] is 0?

Also, you don't need to parenthesize the dict constructor -- return {"data": jobs, "total": response["total"]} would work fine.

Yeah, is this necessary? Is the response value total not present when the number of jobs is zero?

dbutenhof · 2024-12-02T20:00:32Z

backend/app/api/v1/commons/ocp.py

@@ -27,7 +27,7 @@ async def getData(start_datetime: date, end_datetime: date, size:int, offset:int
    tasks = [item['_source'] for item in response["data"]]
    jobs = pd.json_normalize(tasks)
    if len(jobs) == 0:
-        return jobs
+        return ({'data':jbs, 'total': 0})


I think that 'data':jbs is a typo. There isn't actually a jbs in scope, is there?

dbutenhof · 2024-12-02T20:01:05Z

backend/app/api/v1/commons/quay.py

@@ -27,7 +27,7 @@ async def getData(start_datetime: date, end_datetime: date, size, offset, config
    tasks = [item['_source'] for item in response['data']]
    jobs = pd.json_normalize(tasks)
    if len(jobs) == 0:
-        return jobs
+        return ({'data':jbs, 'total': 0})


jbs => jobs

dbutenhof · 2024-12-02T20:02:21Z

backend/app/api/v1/endpoints/ocp/ocpJobs.py

@@ -12,7 +12,7 @@

 @router.get('/api/v1/ocp/jobs',
            summary="Returns a job list",
-            description="Returns a list of jobs in the specified dates of requested size. \
+            description="Returns a list of jobs in the specified dates. \


Maybe "in the specified date range"?

MVarshini added 3 commits October 25, 2024 23:28

Page Size and offset

4baf573

pagination params

9385645

Date filter

5abc6f8

MVarshini self-assigned this Nov 19, 2024

MVarshini requested review from vishnuchalla and chentex November 19, 2024 16:54

MVarshini changed the base branch from main to revamp November 20, 2024 13:39

MVarshini requested review from dbutenhof and jaredoconnell November 20, 2024 13:39

jaredoconnell reviewed Nov 20, 2024

View reviewed changes

dbutenhof reviewed Nov 20, 2024

View reviewed changes

vishnuchalla mentioned this pull request Nov 21, 2024

Dynamic pagination in OCP tab #118

Closed

7 tasks

dbutenhof reviewed Nov 21, 2024

View reviewed changes

MVarshini added 2 commits November 30, 2024 18:39

review comments

58ca8df

commented line in Pagination/index

065e2ba

dbutenhof requested changes Dec 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic Pagination in CPT Dashboard #135

Dynamic Pagination in CPT Dashboard #135

MVarshini commented Nov 19, 2024

jaredoconnell Nov 20, 2024

vishnuchalla Nov 22, 2024

MVarshini Nov 30, 2024

jaredoconnell Nov 20, 2024

jaredoconnell Nov 20, 2024

jaredoconnell Nov 20, 2024

MVarshini Nov 30, 2024

dbutenhof left a comment

dbutenhof Nov 20, 2024

dbutenhof Nov 20, 2024

dbutenhof Nov 21, 2024

MVarshini Nov 30, 2024

dbutenhof Nov 21, 2024

dbutenhof left a comment

dbutenhof Dec 2, 2024

jaredoconnell Dec 2, 2024

dbutenhof Dec 2, 2024

dbutenhof Dec 2, 2024

dbutenhof Dec 2, 2024

		# return await self.scan_indices(self.new_es, self.new_index, query, timestamp_field, start_date, end_date, size)
		# else:

Dynamic Pagination in CPT Dashboard #135

Are you sure you want to change the base?

Dynamic Pagination in CPT Dashboard #135

Conversation

MVarshini commented Nov 19, 2024

Type of change

Description

Related Tickets & Documents

Checklist before requesting a review

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbutenhof left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbutenhof left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment