Struggling to use github_my_repository with a lot of repos (50k+)

[chris-garrett]

Describe the bug
Running select * from github.github_my_repository causes timeouts and rate limits:

SQL Error [HV000]: ERROR: rpc error: code = Unknown desc = github: non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits\",\n  \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to

Steampipe version (steampipe -v)
Steampipe v1.0.0

Plugin version (steampipe plugin list)

+-----------------------------------------------+---------+-------------+
| Installed                                     | Version | Connections |
+-----------------------------------------------+---------+-------------+
| hub.steampipe.io/plugins/turbot/github@latest | 1.0.0   | github      |
| hub.steampipe.io/plugins/turbot/jira@latest   | 1.0.0   | jira        |
+-----------------------------------------------+---------+-------------+

To reproduce
Run the command above.

Expected behavior
Results are returned from query.

Additional context
I did a bit of poking around and I think that the plugin is pulling a large number of forks that I have nothing to do with. I'm attached to the EpicGames repo and it looks like it's trying to pull 50k+ forks from there?

I'm also surprised that I'm hitting rate limits at all. There is a well known 5k request per hour limit I would have expected this plugin to use that limit as a sane default.

[misraved]

Apologies for the delayed response on this issue @chris-garrett!!

GitHub rate limits are extremely harsh and they often return errors.

We do have a GitHub issue where we have some discussions/suggestions on how one might be able to control the rate limits - #398

Unfortunately, we haven't been able to get to a meaningful solution yet, but as a workaround, you could try out Pipes which should be able to handle large datasets.

[misraved]

Please feel free to add your observations in #398, your input would be greatly appreciated and will help us address the matter more effectively. Thank you!