Limit data collection to public organizations/groups

[bzg]

https://code.gouv.fr/public/#/repos collects data from repositories of public GitHub organizations and public GitLab groups.

My understanding is that https://repos.ecosyste.ms collects data from all groups, public and private.

Can we configure repos so that it only considers public groups?

If so, can we spare the need for (GitLab) tokens?

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[bzg]

cc @simkim

[andrew]

You'll still need a public token for GitLab as the api doesn't allow read access even for public data without a token, unlike gitea and github

[andrew]

Right now repos will try to crawl a whole forge, and if it's given a token that can see private repos I suspect it may find some, although I've not tested that, we can definitely add a check to reject repositories that are have a private flag.

Groups/orgs are called "owners" in the repos service.

[bzg]

Groups/orgs are called "owners" in the repos service.

Good to know, thanks.

we can definitely add a check to reject repositories that are have a private flag.

Yes, that will be useful.

You'll still need a public token for GitLab as the api doesn't allow read access even for public data without a token

Are you sure? This script collects metadata from GitLab instances without the need for a token. Or maybe I misunderstand what is the token needed for exactly?

[andrew]

Are you sure? This script collects metadata from GitLab instances without the need for a token. Or maybe I misunderstand what is the token needed for exactly?

I'm not 100% on that, will need to double check, but I remember some GitLab API endpoints needing a token, I can't recall which ones though, it was a little while ago that I set that up.

[bzg]

Okay, thanks.

Because we want to crawl a lot for GitLab forges and because obtaining/renewing tokens can be a chore, we would love to have an option to crawl GitLab forges without tokens, even if it means that we don't get all the data we have when crawling with a token.

[simkim]

fully agree on the usefullness to ignore repo flagged as private.

[andrew]

PR to ignore private repos over here: #492

[andrew]

I've merged #492 now