rate-limit makes the plugin unusable #398

electriquo · 2024-01-04T08:32:19Z

Describe the bug
github rate-limit make the plugin unusable

ERROR: rpc error: code = Unknown desc = github: non-200 OK status code: 403 Forbidden body: "{\n "documentation_url": "https://docs.github.com/en/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits\",\n "message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID"\n}" (SQLSTATE HV000)

and prone to getting banned by github

Continuing to make requests while you are rate limited may result in the banning of your integration.

Steampipe version (steampipe -v)
Example: v0.21.2

Plugin version (steampipe plugin list)
Example: v0.39.0

Expected behavior
github specifies exactly how to handle these status code, we should honor by implementing it.

If you exceed your primary rate limit, you will receive a 403 or 429 response, and the x-ratelimit-remaining header will be 0. You should not retry your request until after the time specified by the x-ratelimit-reset header.

If you exceed a secondary rate limit, you will receive a 403 or 429 response and an error message that indicates that you exceeded a secondary rate limit. If the retry-after response header is present, you should not retry your request until after that many seconds has elapsed. If the x-ratelimit-remaining header is 0, you should not retry your request until after the time, in UTC epoch seconds, specified by the x-ratelimit-reset header. Otherwise, wait for at least one minute before retrying. If your request continues to fail due to a secondary rate limit, wait for an exponentially increasing amount of time between retries, and throw an error after a specific number of retries.

Continuing to make requests while you are rate limited may result in the banning of your integration.

Additional context

The text was updated successfully, but these errors were encountered:

electriquo · 2024-01-16T10:52:18Z

@cbruno10 maybe you could add more information

ParthaI · 2024-02-26T06:17:28Z

Hello, @electriquo. Sorry for the delayed response.

Before we proceed with the fixes, I have a couple of questions, could you please take a look?

How many rows are you attempting to retrieve with your query?
Could you share the specific query that's triggering the rate-limit error?

I attempted to replicate the error by querying the github_issue and github_pull_request tables, but I was unable to reproduce it. I tried the query select count(*) from github_workflow where repository_full_name in (select name_with_owner from github_my_repository) multiple times, this query returns the result successfully all the time.

+-------+
| count |
+-------+
| 893   |
+-------+

Thank you!

electriquo · 2024-02-26T08:40:06Z

@ParthaI Install the github modules (steampipe-mod-github-compliance, steampipe-mod-github-insights, steampipe-mod-github-sherlock) and navigate to the GitHub Default Branch Protection Report.

I have more than 2000 active (non-archived) repositories.

ParthaI · 2024-02-26T13:19:03Z

Thanks, @electriquo, for the detailed information, I will give it a try to reproduce the issues, and make the necessary changes.

ParthaI · 2024-03-07T08:58:14Z

Hello, @electriquo, following our previous discussion, I set up my local environment to try and replicate the error.

Despite my attempts, I couldn't reproduce the rate limit error. The compliance/insight mods are functioning correctly across approximately 370 repositories in our organization.

For your information, the plugin initially used the REST API to generate results, which was prone to errors, particularly those related to rate limits. As a remedy, we've implemented GraphQL query support.

Additionally, I experimented with the plugin code in my local environment, making around 2300 API calls, and it worked without issues, completing in 1,671.1 seconds.

Query Result:

> select count(*) from github_my_repository
+-------+
| count |
+-------+
| 11871 |
+-------+

Time: 1,671.1s. Rows fetched: 11,871. Hydrate calls: 0.

Thanks!

electriquo · 2024-03-07T14:24:41Z

@ParthaI You were experimenting, which is a reproduction best effort. Yet, it does not say that there is no issue :)
Which information/logs do you need to get to the bottom of it?

As you can see in the description, the rate limit error is clear.

ParthaI · 2024-03-08T04:54:29Z

Hello @electriquo,

Thank you for your patience and for highlighting the issue once more. Indeed, the rate-limit error you mentioned is a significant concern, when navigating to the GitHub Default Branch Protection Report, given the extensive number of active (non-archived) repositories you manage, exceeding 2000.

In an effort to meticulously replicate this scenario, I conducted tests involving over 2000 API calls within a single query. Regrettably, these tests did not trigger the same rate-limit error, which suggests the situation might be influenced by specific conditions or a huge number of repositories in the organization.

Understanding the importance of accurately diagnosing and resolving this issue, I'd like to delve deeper. The dashboard you referred to leverages the github_my_repository table to fetch and display data. To assist us further in isolating the cause, could you kindly confirm if executing the query select * from github_my_repository directly in your Steampipe environment results in the same rate-limit error? This information will be invaluable in our investigation.

Your cooperation and insights are greatly appreciated!

electriquo · 2024-03-10T07:17:45Z

@ParthaI

To assist us further in isolating the cause, could you kindly confirm if executing the query select * from github_my_repository directly in your Steampipe environment results in the same rate-limit error?

No issue here but when I open my-steampipe-host.com/github_insights.dashboard.default_branch_protection_report I get the limit-rate error.

ParthaI · 2024-03-12T06:24:08Z

Thank you, @electriquo, for conducting those tests and sharing your findings. Replicating the rate-limit error appears to be challenging with less set of data set in our environment.

According to the documentation, using a Personal Access Token (PAT) allows for 5000 API calls within a certain timeframe. However, running the GitHub Insight Mods seems to exceed this limit in your setup, likely due to the large number of repositories (around 2000) in your environment.

In comparison, our environment hosts approximately 330 repositories. Before executing the GraphQL query, our initial rate limit was 5000. After executing select count(*) from github_my_repository, this limit dropped to 4923, indicating that approximately 77 API calls were made (per page result limit value is set to 5) when querying the github_my_repository table.

Here's the GraphQL query to check the rate limit:

query {
  viewer {
    login
  }
  rateLimit {
    limit
    remaining
    used
    resetAt
  }
}

You can find more details on how rate limits are calculated for GraphQL queries in the GitHub documentation.

Upon reviewing the GitHub insights mod, specifically the github_insights.dashboard.default_branch_protection_report, it seems the github_my_repository table is queried multiple times, which could contribute to reaching the rate limit.

I'm further investigating the plugin and the insight mod to understand this behavior better and will update you with any progress.

Thank you once more for your assistance!

electriquo · 2024-03-12T10:09:05Z

using a Personal Access Token (PAT) allows for 5000 API calls within a certain timeframe.

To clear any doubts, I am using a dedicate PAT; meaning that only Steampipe is using this PAT. Thus, the rate-limit must come when using Steampipe only.

Could it be that not only the amount of repositories are the factor but the amount of metadata (such as a big Git history/commits)?

ParthaI · 2024-03-12T11:17:44Z

Could it be that not only the amount of repositories are the factor but the amount of metadata (such as a big Git history/commits)?

Absolutely, @electriquo, you're correct. However, it's worth noting that within the Steampipe Dashboard, API calls are not initiated until you actively click on any of the hyperlinks.

electriquo · 2024-03-12T13:46:26Z

My repositories are reach, they are big with many metadata.
How do you advise to address it or is this plugin is only for a small scale and thin repositories?

ParthaI · 2024-03-12T15:31:58Z

Hello @electriquo,

I hope this message finds you well. I am currently working on this issue aimed at reducing the number of API calls while quering the table github_my_repository I have made a few changes in the GitHub Plugin and the GitHub Insight Mod. Unfortunately, I have yet to finalize an effective solution and find myself in need of a bit more information. When you have a moment, could you kindly assist me by checking on a few details?

Could you please share how long the query select * from github_my_repository takes to execute in your current setup? If you could follow these steps, it would be immensely helpful:
- Execute the command steampipe query.
- Enable timing with .timing on.
- Run the query select * from github_my_repository.
I have recently committed some updates to the GitHub plugin repository on the branch update-limit-for-github-my-repo-table and to the GitHub Insight mod on the branch updated-query-to-hit-cache. Could I trouble you to fetch these updates to your local setup and build the plugin from there?

Please note, these modifications are specifically for the GitHub Default Branch Protection Report dashboard. My goal is to ensure these changes perform effectively, especially with larger datasets.

If you're willing to test the scenario, here are the steps:

Clone the repository with git clone https://github.com/turbot/steampipe-plugin-github.git.
Change the directory with cd steampipe-plugin-github.
Switch to the relevant branch with git checkout update-limit-for-github-my-repo-table.
Compile the code with make.
Start the service with steampipe service start.
Return to the root directory.
Clone another repository with git clone https://github.com/turbot/steampipe-mod-github-insights.git.
Change the directory with cd steampipe-mod-github-insights.
Switch to the appropriate branch with git checkout updated-query-to-hit-cache.
Start the server with powerpipe server.
Finally, navigate to the GitHub Default Branch Protection Report.

Your feedback on these changes would be invaluable to me.

Thank you very much for your cooperation and assistance in this matter.

ParthaI · 2024-03-19T04:46:37Z

Hi @electriquo, have you had the chance to try it out yet?

electriquo · 2024-03-19T10:11:46Z

@ParthaI Sorry, will try it soon.

electriquo · 2024-03-19T11:56:35Z

Could you please share how long the query select * from github_my_repository takes to execute in your current setup?

$ steampipe query
Welcome to Steampipe v0.22.1
For more information, type .help
> .timing on
> select * from github_my_repository

Error: github: non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits\",\n  \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 279A:5B4F:18264BC:2CAC1C2:65F97C91.\"\n}" (SQLSTATE HV000)

Time: 47.1s. Rows fetched: 250. Hydrate calls: 17,250.

Could I trouble you to fetch these updates to your local setup and build the plugin from there?
...
here are the steps:

Where is the install command for update-limit-for-github-my-repo-table plugin and updated-query-to-hit-cache mod?
What am I missing?

ParthaI · 2024-03-19T12:26:49Z

$ steampipe query
Welcome to Steampipe v0.22.1
For more information, type .help
>.timing on
>select * from github_my_repository

Error: github: non-200 OK status code: 403 Forbidden body: "{\n "documentation_url": "https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits\",\n "message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 279A:5B4F:18264BC:2CAC1C2:65F97C91."\n}" (SQLSTATE HV000)

Time: 47.1s. Rows fetched: 250. Hydrate calls: 17,250.

I believe as per our previous conversation it was working fine.

Where is the install command for update-limit-for-github-my-repo-table plugin and updated-query-to-hit-cache mod?
What am I missing?

Please follow the following steps:

Steampie Plugin changes:

Clone the repository with git clone https://github.com/turbot/steampipe-plugin-github.git.
Change the directory with cd steampipe-plugin-github.
Switch to the relevant branch with git checkout update-limit-for-github-my-repo-table.
Run the command make.
Start the service with steampipe service start.

Steampipe Github Insights Mod:

Clone the repository to a directory git clone https://github.com/turbot/steampipe-mod-github-insights.git.
Change the directory with cd steampipe-mod-github-insights.
Switch to the branch with git checkout updated-query-to-hit-cache.
Start the server with powerpipe server.
Finally, navigate to the GitHub Default Branch Protection Report.

electriquo · 2024-03-24T08:09:29Z

@ParthaI Haven't forgot, didn't find time this :(

cbruno10 · 2024-03-26T19:49:31Z

Hey @electriquo , I was able to reproduce the behaviours you were seeing (albeit with a much lower repository count) with select * from github_my_repository.

I looked into the plugin code, and found our error handling code does retry secondary rate limit errors (

steampipe-plugin-github/github/errors.go

Lines 39 to 43 in ec93282

    
           // v4 secondary rate limit 
        
           if strings.Contains(err.Error(), "You have exceeded a secondary rate limit.") { 
        
           	plugin.Logger(ctx).Debug("github_errors.shouldRetryError", "abuse_rate_limit_error", err) 
        
           	return true 
        
           }

), with the plugin's retry config set to use an exponential backoff algorithm with max 10 retries (

steampipe-plugin-github/github/errors.go

Lines 54 to 62 in ec93282

    
           func retryConfig() *plugin.RetryConfig { 
        
           	return &plugin.RetryConfig{ 
        
           		ShouldRetryErrorFunc: shouldRetryError, 
        
           		MaxAttempts:          10, 
        
           		BackoffAlgorithm:     "Exponential", 
        
           		RetryInterval:        1000, 
        
           		CappedDuration:       30000, 
        
           	} 
        
           }

); this is probably still too aggressive based on how many API calls we'd be making for that many repos.

If you're directly running a query, one suggestion is to only select the columns you want to retrieve.

If you're running a mod dashboard/benchmark/control, you could also try using rate limiters to slow down Steampipe.

For instance, you can try adding this into your github.spc:

# stay well under the 100 hydrate/list/get functions concurrently based on limits in https://docs.github.com/en/graphql/overview/rate-limits-and-node-limits-for-the-graphql-api#secondary-rate-limits
plugin "github" {
  limiter "github_global_concurrency" {
    max_concurrency = 30 
  }
}

And then see if that helps with running the dashboards or benchmarks you were trying before.

I think select * from my_repository will still be tough for the plugin to complete even with rate limiters though based on how GitHub checks for secondary rate limits, as they have some internal timeouts and we request a lot of data as part of the query when selecting all columns.

For instance, with the limiter settings above, I avoided getting the secondary abuse rate limits, but eventually got this unhelpful 502 Bad Gateway error from them with no additional information:

Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug.

I don't believe a lot of our dashboards run this type of query though.

electriquo · 2024-03-27T07:32:05Z

@cbruno10

found our error handling code does retry secondary rate limit errors

Does it mean that you are working on a fix and I should wait for a new release?

For instance, you can try adding this into your github.spc:

# stay well under the 100 hydrate/list/get functions concurrently based on limits in https://docs.github.com/en/graphql/overview/rate-limits-and-node-limits-for-the-graphql-api#secondary-rate-limits
plugin "github" {
  limiter "github_global_concurrency" {
    max_concurrency = 30 
  }
}

Where in the docs did you find about github_global_concurrency and max_concurrency?

cbruno10 · 2024-03-27T14:10:33Z

@electriquo

We are looking into if we can improve the error handling for rate limit errors, but we're not actively working on an identified fix and don't have a schedule on when it will be released.

I'd suggest still trying to use a limiter and seeing if that helps your executions.

In the example I sent, the github_global_concurrency is the name I gave the limiter resource, but general documentation can be found at https://steampipe.io/docs/guides/limiter.

electriquo · 2024-03-27T16:42:46Z

@cbruno10

We are looking into if we can improve the error handling for rate limit errors, but we're not actively working on an identified fix and don't have a schedule on when it will be released.

Thanks

In the example I sent, the github_global_concurrency is the name I gave the limiter resource, but general documentation can be found at steampipe.io/docs/guides/limiter.

In the future, how one may know the variables that can be set in a plugin to handle concurrency and rate limiting?
If you would not have mentioned the structure and the variable, I guess that most won't know about it :|
That's why I asked

Where in the docs did you find about github_global_concurrency and max_concurrency?

Maybe it should be documented in steampipe.io/docs/guides/limiter.

I was able to reproduce the behaviours

you can try adding this into your github.spc:

# stay well under the 100 hydrate/list/get functions concurrently based on limits in https://docs.github.com/en/graphql/overview/rate-limits-and-node-limits-for-the-graphql-api#secondary-rate-limits
plugin "github" {
  limiter "github_global_concurrency" {
    max_concurrency = 30 
  }
}

If you apply the block in github.spc, did it mitigate the rate-limiting which you were able to reproduce?

cbruno10 · 2024-04-09T13:59:27Z

@electriquo With that block in my github.spc, yes I did see the rate-limiting was resolved. My query execution time increased, but I was able to get all of the data for my repos (though this number is much lower than the amount of repos you have).

Were you able to give the limiter a try, either with max concurrency of 30 (or a different number)? If so, did you see any more consistency in getting results back?

Also, we should probably mention rate limiters and at least link to the doc on steampipe.io, which has some examples, so we'll add that to our backlog to see where that section belongs.

electriquo · 2024-04-09T14:23:27Z

Were you able to give the limiter a try, either with max concurrency of 30 (or a different number)?

It is on my list for tomorrow :)

If so, did you see any more consistency in getting results back?

Promise to keep you posted

Also, we should probably mention rate limiters and at least link to the doc on steampipe.io, which has some examples, so we'll add that to our backlog to see where that section belongs.

Awesome, thanks you :)

electriquo · 2024-04-10T07:08:47Z

@cbruno10

$ steampipe --version
Steampipe v0.22.2

$ steampipe plugin list --output json | jq -r '.installed[].name'
hub.steampipe.io/plugins/turbot/[email protected]

$ powerpipe mod list --output json | jq -r '.[].dependency_path'
github.com/turbot/[email protected]
github.com/turbot/[email protected]
github.com/turbot/[email protected]

$ cat ~/.steampipe/config/github.spc
connection "github" {
  plugin = "hub.steampipe.io/plugins/turbot/[email protected]"
}

plugin "github" {
  limiter "github_global_concurrency" {
    max_concurrency = 30
  }
}

When I navigate to the GitHub Default Branch Protection Report dashboard, the rate-limit error still pops.

But if I repeat #398 (comment) does not return a rate-limit error message

$ steampipe query
Welcome to Steampipe v0.22.2
For more information, type .help
> .timing on
> select count(*) from github_my_repository
...
Time: 7.6s. Rows fetched: 1,088. Hydrate calls: 0.

And then navigating to the GitHub Default Branch Protection Report dashboard, the rate-limit error does not pops but besides the repository list, all other columns are empty :(

I look at look at the plugin log, and I found that it seems not to use (but do honor) the max concurrency connections. Here are some log lines

2024-04-10 07:16:43.590 UTC [INFO]  waitForPluginLoad: hub.steampipe.io/plugins/turbot/[email protected] is now loaded and ready (0x40007cd880)
2024-04-10 07:16:43.593 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403165: Plugin execute table name: github_my_repository quals: NONE (1712733403165)
2024-04-10 07:16:43.593 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403165: Executing for 1 connection: github
2024-04-10 07:16:43.593 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403165: Setting max concurrent connections to 25
2024-04-10 07:16:43.593 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403165: resolveRateLimiterScopeValues (github-1712733403165)
2024-04-10 07:16:43.593 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403165: resolvedRateLimiterConfig: no rate limiters (github-1712733403165)
2024-04-10 07:16:51.016 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403520: rowData chan returned nil - wait for rows to complete (github-1712733403165)
2024-04-10 07:16:51.017 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403520: streamRows - nil row, stop streaming (github-1712733403165)
2024-04-10 07:16:51.023 UTC [INFO]  steampipe-plugin-github.plugin: [INFO]  1712733403520: Plugin execute table: github_my_repository closing error chan and output chan  (1712733403165)

Also tried with max_concurrency = 10, to check whether it will still be honored and/or resolve the issue by examining the log file and the dashboard, but everything was the same :(

electriquo · 2024-04-14T09:37:49Z

@cbruno10 Hi, any insights?

electriquo · 2024-05-05T07:54:49Z

@cbruno10 Hello, do you have any new insights?

electriquo · 2024-05-07T07:28:20Z

@cbruno10 @ParthaI Hello, did we lost all momentum?

misraved · 2024-05-17T06:38:50Z

Apologies for the radio silence on this issue @electriquo !!

I think the reason why you didn't see the limiter tags getting honored could be indicative of a caching issue.

Could you please retry the queries by launching a fresh instance of Steampipe and killing any old instances?

Command that you could try before running steampipe query:

steampipe service stop --force
pkill -f steampipe

electriquo · 2024-05-18T07:40:48Z

Could you please retry the queries by launching a fresh instance of Steampipe and killing any old instances?

@misraved That's what I always did :)

I think the reason why you didn't see the limiter tags getting honored could be indicative of a caching issue.

Seems like firing in all directions rather than base things on data.
Can we get to the bottom of it or should I just drop the GitHub plugin as a whole?

misraved · 2024-05-20T07:33:42Z

Thanks for the clarification @electriquo !!

We are actively exploring options to enhance the rate-limiting capabilities within our plugin. Currently, there is a limited selection of solutions that offer a straightforward method for managing errors originating from the API.

electriquo · 2024-05-20T09:33:45Z

@misraved Could you please follow #398 (comment)?

cbruno10 · 2024-05-23T19:38:29Z

Hi @electriquo , sorry for losing track of this thread.

For verifying if your limiters are being used, you can run select * from steampipe_plugin_limiter (taken from Viewing and Overriding Limiters). If you run that query, are you able to see your limiter with the correct settings?
While using a limiter, do you have issues with queries, or primarily with the dashboards?
- If running dashboards, which ones have you tried beside GitHub Default Branch Protection Report? And have they all given you errors?
- Dashboards in general run a lot of queries simultaneously, so it'll be more common to hit rate limiters when running dashboards

electriquo · 2024-05-24T14:24:44Z

@cbruno10

The rate-limit error should be handled generally regardless the component that causes it, especially when there is a clear protocol for rate-limit. Specifically, GitHub has Rate limits for the REST API.

are you able to see your limiter with the correct settings?

No.
I followed your guidance to set github_global_concurrency and executed the query. As you can see below, GitHub is not in the list.

$ steampipe query "select distinct name from steampipe_plugin_limiter"
+---------------------------------------------------+
| name                                              |
+---------------------------------------------------+
| aws_servicequotas_list_service_quotas             |
| aws_servicequotas_list_aws_default_service_quotas |
| aws_servicequotas_list_tags_for_resource          |
+---------------------------------------------------+

Why would you expect to see the limiter appear here when the plugin log clearly states so?

Dashboards in general run a lot of queries simultaneously, so it'll be more common to hit rate limiters when running dashboards

From your words, I understand that dashboards are useless :(

If running dashboards, which ones have you tried beside GitHub Default Branch Protection Report? And have they all given you errors?

Received the same error across few GitHub's dashboards.

cbruno10 · 2024-05-24T18:02:39Z

@electriquo

If I understand your first point above correctly, we do handle rate-limit errors today (for v3 and v4 APIs). For instance, in https://github.com/turbot/steampipe-plugin-github/blob/main/github/errors.go, we have a function to determine when to retry based on the error code or error message we get back and use a reset time (if available in the response), e.g., https://github.com/turbot/steampipe-plugin-github/blob/main/github/errors.go#L22-L37. For v4/GraphQL API calls, we handle them in

steampipe-plugin-github/github/errors.go

Lines 39 to 43 in 8399ae8

    
           // v4 secondary rate limit 
        
           if strings.Contains(err.Error(), "You have exceeded a secondary rate limit.") { 
        
           	plugin.Logger(ctx).Debug("github_errors.shouldRetryError", "abuse_rate_limit_error", err) 
        
           	return true 
        
           }

; unfortunately, we don't get back more info from the SDK on retry times, but we do use exponential backoff when retrying requests. If I missed your original point though, please let me know.

I was curious if the limiter would show up in that query result, as I wasn't sure in what context the plugin log you were referencing was used in. For instance, in the Steampipe plugin SDK, the log message is sent from https://github.com/turbot/steampipe-plugin-sdk/blob/5d869679770d34049cf492182213514c1f5669f2/plugin/env.go#L29-L36, but Steampipe under the hood does a lot of connection management (even before limiters existed), so it's good to check what the introspection table says as well.
Our current dashboards may not scale well based on how many repositories you have access to. For instance, we've tested them with users who have access to 400+ repositories, but definitely not 1k+ (that's a lot)! Our goal is to make dashboards load correctly for your environment too, so we're interested in figuring out Steampipe configuration + dashboard updates to make this possible.

Looking at your limiter configurations, can you please update it to:

plugin "hub.steampipe.io/plugins/turbot/[email protected]" {
  limiter "github_global_concurrency" {
    max_concurrency = 30
  }
}

Since you're using a specific plugin version, the plugin reference needs to reference that version.

Afterward, can you please restart Steampipe, steampipe service restart and then run select * from steampipe_plugin_limiter and see if the limiter then shows up?

If it does, can you please try running the single query you shared above first and see if that still executes correctly? If so, can you please then run the dashboard again? You may still get throttling errors, and if so, can you please try lowering max_concurrency to 20 (and then 10)?

Thanks!

electriquo · 2024-05-25T06:24:05Z

@cbruno10 Although I am cooperating, I am not the Steampipe QA team :)
I've done whatever I was asked until now, but feel that there is firing to all directions in hope to hit something, and Steampipe do not need me for it - Steampipe team can and should reproduce everything on their own to save this back and forth writing.

Since you're using a specific plugin version, the plugin reference needs to reference that version.

This doesn't sound correct for me, just another shot in the dark.

After you took the toll to create and environment to:

Reproduce the issue
Examine the logs
Check and test proposed solutions

I'd be happy to continue and assist.

cbruno10 · 2024-05-28T14:01:13Z

@electriquo Thanks for all of your testing efforts and sharing so far, we appreciate all of the info you've shared! We try to reproduce users' environments as much as we can, but sometimes that's difficult as we don't have direct access to see how many repositories, branches, users, branch protection settings, etc., that are in those environments, so we're not able to reproduce them exactly. We do have a GitHub organization with a fairly large number of repositories, but definitely not as large as the one you're querying, so any information you're able to share is helpful for us.

In regards to the rate limiter, I was able to reproduce the rate limiter not loading when a specific plugin version was installed, but once I installed GitHub plugin v0.39.1 and created a limiter I shared above:

plugin "hub.steampipe.io/plugins/turbot/[email protected]" {
  limiter "github_global_concurrency" {
    max_concurrency = 30
  }
}

Then I could see my limiter being loaded and used by the plugin.

electriquo · 2024-05-29T10:12:34Z

I was able to reproduce the rate limiter not loading when a specific plugin version was installed

This holds. Had to set the max_concurrency to 1.
I only wonder why does the version must be specified when the same file already declares which plugin (and version) to use.

jmreicha · 2024-06-16T20:55:22Z

I ran into this issue today. I have access to 3800 repos and 10 orgs but I’m only interested in a handful of those orgs for querying.

I haven't tried the limiter yet but my first inclination was to see if there was any way to scope down the repos/orgs that get queried in the config file, similar to how the AWS plugin handles it.

I think that coupled with the limiter would cover my use case.

electriquo · 2024-06-17T05:10:35Z

@jmreicha No matter what I did, I couldn't make this work as I was always facing the rate limit.

cbruno10 · 2024-06-21T14:04:44Z

Hi @jmreicha , are you running standalone queries or benchmarks/dashboards from mods? If running queries, there are ways to structure the queries sometimes to reduce the number of requests and work better within the rate limit. For the mods, I think we can add better filtering capabilities to optimize the queries, so if there are specific ones you're interested in (along with any you're interested in @electriquo ), please let us know and we can have a look at those first.

electriquo · 2024-06-21T20:47:33Z

Hi @cbruno10, Thank you.

I cannot tell what would I like to see since I cannot explore the plugin/mods due to the rate limit. Thus, I cannot share much about the usage

if there are specific ones you're interested in, please let us know and we can have a look at those first.

jmreicha · 2024-06-22T18:37:51Z

Hey @cbruno10

are you running standalone queries or benchmarks/dashboards from mods?

I noticed the issue when I was playing around with dashboards, specifically the organization best practices. That would be my main use case probably. I was also triggering the rate limit pretty easily doing queries but I will play around with that more and see if I can make better queries.

github-actions · 2024-08-21T23:31:52Z

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

electriquo · 2024-08-22T05:39:56Z

Is it really stale or there is no fix?

misraved · 2024-08-26T07:10:59Z

Hi @electriquo

Thank you for your patience and understanding. We have explored various options to bypass the strict rate limits imposed by the GitHub API, but unfortunately, we have not been able to find a viable solution that would allow us to overcome these restrictions effectively.

However, we would recommend looking into Turbot Pipes, which might offer a more scalable approach to managing these limitations. Turbot Pipes is designed to handle API interactions efficiently and could provide the flexibility needed for your use case.

Please let us know if you have any other questions or if there is anything else we can assist you with.

electriquo · 2024-08-26T07:28:16Z

@misraved Thank you for the effort :)

cbruno10 · 2024-09-04T12:59:55Z

I ran into this issue today. I have access to 3800 repos and 10 orgs but I’m only interested in a handful of those orgs for querying.

I haven't tried the limiter yet but my first inclination was to see if there was any way to scope down the repos/orgs that get queried in the config file, similar to how the AWS plugin handles it.

I think that coupled with the limiter would cover my use case.

@jmreicha That's a lot of repos!

I like the idea of scoping down the repos and orgs for queries. I don't know if we'd add that capability to the GitHub config file, but instead maybe in the GitHub Compliance mod itself. For instance, we could add variables that allow users to specify which orgs and repos they want to include in the control/dashboard queries, and then those are added to the where clause. There may be other columns we can filter API calls on as well.

I don't have any specific timeline on when we'd add this capability, but it is on our radar to see how to make the mod more usable at larger scale.

jmreicha · 2024-09-04T17:55:40Z

That sounds pretty reasonable, thanks for the update.

github-actions · 2024-11-03T23:32:02Z

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

electriquo added the bug Something isn't working label Jan 4, 2024

misraved assigned ParthaI Feb 13, 2024

github-actions bot added the stale No recent activity has been detected on this issue/PR and it will be closed label Aug 21, 2024

github-actions bot removed the stale No recent activity has been detected on this issue/PR and it will be closed label Aug 22, 2024

github-actions bot added the stale No recent activity has been detected on this issue/PR and it will be closed label Nov 3, 2024

misraved removed the stale No recent activity has been detected on this issue/PR and it will be closed label Nov 18, 2024

misraved mentioned this issue Nov 21, 2024

Struggling to use github_my_repository with a lot of repos (50k+) #461

Closed

rate-limit makes the plugin unusable #398

rate-limit makes the plugin unusable #398

Comments

electriquo commented Jan 4, 2024 • edited Loading

electriquo commented Jan 16, 2024

ParthaI commented Feb 26, 2024

electriquo commented Feb 26, 2024

ParthaI commented Feb 26, 2024

ParthaI commented Mar 7, 2024

electriquo commented Mar 7, 2024 • edited Loading

ParthaI commented Mar 8, 2024 • edited Loading

electriquo commented Mar 10, 2024

ParthaI commented Mar 12, 2024 • edited Loading

electriquo commented Mar 12, 2024

ParthaI commented Mar 12, 2024

electriquo commented Mar 12, 2024 • edited Loading

ParthaI commented Mar 12, 2024

ParthaI commented Mar 19, 2024

electriquo commented Mar 19, 2024

electriquo commented Mar 19, 2024

ParthaI commented Mar 19, 2024

electriquo commented Mar 24, 2024

cbruno10 commented Mar 26, 2024

electriquo commented Mar 27, 2024

cbruno10 commented Mar 27, 2024

electriquo commented Mar 27, 2024

cbruno10 commented Apr 9, 2024

electriquo commented Apr 9, 2024

electriquo commented Apr 10, 2024 • edited Loading

electriquo commented Apr 14, 2024

electriquo commented May 5, 2024

electriquo commented May 7, 2024 • edited Loading

misraved commented May 17, 2024

electriquo commented May 18, 2024

misraved commented May 20, 2024

electriquo commented May 20, 2024

cbruno10 commented May 23, 2024 • edited Loading

electriquo commented May 24, 2024 • edited Loading

cbruno10 commented May 24, 2024

electriquo commented May 25, 2024 • edited Loading

cbruno10 commented May 28, 2024

electriquo commented May 29, 2024 • edited Loading

jmreicha commented Jun 16, 2024 • edited Loading

electriquo commented Jun 17, 2024

cbruno10 commented Jun 21, 2024

electriquo commented Jun 21, 2024 • edited Loading

jmreicha commented Jun 22, 2024

github-actions bot commented Aug 21, 2024

electriquo commented Aug 22, 2024

misraved commented Aug 26, 2024

electriquo commented Aug 26, 2024

cbruno10 commented Sep 4, 2024

jmreicha commented Sep 4, 2024

github-actions bot commented Nov 3, 2024

electriquo commented Jan 4, 2024 •

edited

Loading

electriquo commented Mar 7, 2024 •

edited

Loading

ParthaI commented Mar 8, 2024 •

edited

Loading

ParthaI commented Mar 12, 2024 •

edited

Loading

electriquo commented Mar 12, 2024 •

edited

Loading

electriquo commented Apr 10, 2024 •

edited

Loading

electriquo commented May 7, 2024 •

edited

Loading

cbruno10 commented May 23, 2024 •

edited

Loading

electriquo commented May 24, 2024 •

edited

Loading

electriquo commented May 25, 2024 •

edited

Loading

electriquo commented May 29, 2024 •

edited

Loading

jmreicha commented Jun 16, 2024 •

edited

Loading

electriquo commented Jun 21, 2024 •

edited

Loading