Added pagination to the aws_ec2_ami_shared table list operation and optimized the table to perform batch operation with given image IDs and owner IDs

[ParthaI]

Integration test logs

Logs

N/A

Example query results

Results

Earlier(Making single API calls per Image ID):

> WITH instances AS (
  SELECT
    instance_id,
    instance_type,
    account_id,
    tags ->> 'Name' AS instance_name,
    _ctx ->> 'connection_name' AS account_name,
    instance_state,
    region,
    image_id
  FROM
    aws_zero_intg_test.aws_ec2_instance
)
SELECT DISTINCT
  aws_ec2_ami_shared.image_id as image_id,
  aws_ec2_ami_shared.owner_id as image_owner_id,
  aws_ec2_ami_shared.image_owner_alias as image_owner_name,
  instances.instance_name,
  instances.account_name,
  instances.region,
  aws_zero_intg_test.aws_ec2_ami_shared.name as image_name
FROM
  instances
LEFT JOIN aws_zero_intg_test.aws_ec2_ami_shared ON aws_zero_intg_test.aws_ec2_ami_shared.image_id=instances.image_id
WHERE aws_zero_intg_test.aws_ec2_ami_shared.image_owner_alias != 'amazon'
  AND aws_zero_intg_test.aws_ec2_ami_shared.image_owner_alias != 'self'
+----------+----------------+------------------+---------------+--------------+--------+------------+
| image_id | image_owner_id | image_owner_name | instance_name | account_name | region | image_name |
+----------+----------------+------------------+---------------+--------------+--------+------------+
+----------+----------------+------------------+---------------+--------------+--------+------------+

Time: 16.9s. Rows returned: 0. Rows fetched: 14 (1 cached). Hydrate calls: 13. Scans: 9.

Scans:
  1) aws_ec2_instance.aws_zero_intg_test: Time: 2.7s. Fetched: 8. Hydrates: 8.
  2) aws_ec2_ami_shared.aws_zero_intg_test: Time: 139ms. Fetched: 1. Hydrates: 1. Quals: image_owner_alias<>'amazon', image_owner_alias<>'self', image_id=ami-04c5f154a6c2fec00.
  3) aws_ec2_ami_shared.aws_zero_intg_test: Time: 127ms. Fetched: 1. Hydrates: 1. Quals: image_owner_alias<>'amazon', image_owner_alias<>'self', image_id=ami-0b2106aac2a665c89.
  4) aws_ec2_ami_shared.aws_zero_intg_test: Time: 125ms. Fetched: 1. Hydrates: 1. Quals: image_owner_alias<>'amazon', image_owner_alias<>'self', image_id=ami-067c21fb1979f0b27.
  5) aws_ec2_ami_shared.aws_zero_intg_test: Time: 124ms. Fetched: 1. Hydrates: 1. Quals: image_owner_alias<>'amazon', image_owner_alias<>'self', image_id=ami-0ded8326293d3201b.
  6) aws_ec2_ami_shared.aws_zero_intg_test: Time: 124ms. Fetched: 1. Hydrates: 1. Quals: image_owner_alias<>'amazon', image_owner_alias<>'self', image_id=ami-08df646e18b182346.
  7) aws_ec2_ami_shared.aws_zero_intg_test: Time: 9ms. Fetched: 1 (cached). Hydrates: 0. Quals: image_owner_alias<>'amazon', image_owner_alias<>'self', image_id=ami-067c21fb1979f0b27.

The changes in this PR(Makes only a single API call by taking all the image IDs):

> WITH instances AS (
  SELECT
    instance_id,
    instance_type,
    account_id,
    tags ->> 'Name' AS instance_name,
    _ctx ->> 'connection_name' AS account_name,
    instance_state,
    region,
    image_id
  FROM
    aws_zero_intg_test.aws_ec2_instance
),
all_image_ids as (
  SELECT
    json_agg(image_id)::jsonb AS image_ids  -- Cast to jsonb
  FROM
    instances
)

select 
* 
from 
aws_zero_intg_test.aws_ec2_ami_shared as s,
all_image_ids
 where s.image_ids = all_image_ids.image_ids
+-----------------------------------------------------+-----------------------+-----------+------------+------------------------------------------------------------+---------------------------+--------------+--------------------------->
| name                                                | image_id              | state     | image_type | image_location                                             | creation_date             | architecture | description               >
+-----------------------------------------------------+-----------------------+-----------+------------+------------------------------------------------------------+---------------------------+--------------+--------------------------->
| al2023-ami-2023.1.20230725.0-kernel-6.1-x86_64      | ami-0ded8326293d3201b | available | machine    | amazon/al2023-ami-2023.1.20230725.0-kernel-6.1-x86_64      | 2023-07-26T04:16:54+05:30 | x86_64       | Amazon Linux 2023 AMI 2023>
| aws-intg-test                                       | ami-0b2106aac2a665c89 | available | machine    | 260848204312/aws-intg-test                                 | 2023-07-26T10:51:25+05:30 | x86_64       | DO NOT DELETE             >
| al2023-ami-2023.2.20230920.1-kernel-6.1-x86_64      | ami-067c21fb1979f0b27 | available | machine    | amazon/al2023-ami-2023.2.20230920.1-kernel-6.1-x86_64      | 2023-09-19T03:47:10+05:30 | x86_64       | Amazon Linux 2023 AMI 2023>
| amzn2-ami-kernel-5.10-hvm-2.0.20220606.1-x86_64-gp2 | ami-08df646e18b182346 | available | machine    | amazon/amzn2-ami-kernel-5.10-hvm-2.0.20220606.1-x86_64-gp2 | 2022-06-15T01:24:19+05:30 | x86_64       | Amazon Linux 2 Kernel 5.10>
| old-docker-test-server                              | ami-04c5f154a6c2fec00 | available | machine    | 260848204312/old-docker-test-server                        | 2023-09-26T20:28:18+05:30 | x86_64       | old-docker-test-server    >
+-----------------------------------------------------+-----------------------+-----------+------------+------------------------------------------------------------+---------------------------+--------------+--------------------------->

Time: 7.6s. Rows returned: 5. Rows fetched: 13. Hydrate calls: 15. Scans: 2.

Scans:
  1) aws_ec2_instance.aws_zero_intg_test: Time: 4.8s. Fetched: 8. Hydrates: 0.
  2) aws_ec2_ami_shared.aws_zero_intg_test: Time: 180ms. Fetched: 5. Hydrates: 15. Quals: image_ids=[
    "ami-08df646e18b182346",
    "ami-04c5f154a6c2fec00",
    "ami-0518b9a76d645f07e",
    "ami-0518b9a76d645f07e",
    "ami-0b2106aac2a665c89",
    "ami-067c21fb1979f0b27",
    "ami-067c21fb1979f0b27",
    "ami-0ded8326293d3201b"
].

Note: Notice the SCANS for the queries.

[github-actions]

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.