feat/enterpriseportal: import data from dotcom #63858

bobheadxi · 2024-07-16T18:31:20Z

Adds a background job that can periodically import subscriptions, licenses, and Cody Gateway access from dotcom.

Note that subscriptions and licenses cannot be deleted, so we don't need to worry about that. Additionally licenses cannot be updated, so we only need to worry about creation and revocation.

The importer can be configured with DOTCOM_IMPORT_INTERVAL - if zero, the importer is disabled.

Closes https://linear.app/sourcegraph/issue/CORE-216

Test plan

DOTCOM_IMPORT_INTERVAL=10s sg start dotcom

Look for service.importer logs. Play around in https://sourcegraph.test:3443/site-admin/dotcom/product/subscriptions/ to create and edit subscriptions, licenses, and Cody Gateway access. Watch them show up in the database:

psql -d sourcegraph
sourcegraph# select * from enterprise_portal_susbscriptions;
sourcegraph# select * from enterprise_portal_susbscription_licenses;
sourcegraph# select * from enterprise_portal_cody_gateway_access;

bobheadxi · 2024-07-16T18:31:34Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @bobheadxi and the rest of your teammates on Graphite

cmd/enterprise-portal/internal/database/importer/importer.go

Co-authored-by: James Cotter <[email protected]>

…63925) This change follows https://github.com/sourcegraph/sourcegraph/pull/63858 by making the Cody Access APIs _read_ from the Enterprise Portal database, instead of dotcomdb, using the data that we sync from dotcomdb into Enterprise Portal. As part of this, I also expanded the existing "compatibility" test suite that compares the result of our dotcomdb queries against the existing GraphQL resolvers in dotcom to also compare the results of our new Cody Access APIs, to validate that they return the same access. > [!WARNING] > There is one behavioural change, which is that hashes of _expired licenses_ will no longer be valid as access tokens. This shouldn't be an issue if customers use zero-config (implied access token from their license key) - I will do some outreach before rolling this out. Subsequent PRs will implement write APIs. Part of https://linear.app/sourcegraph/issue/CORE-218 Part of https://linear.app/sourcegraph/issue/CORE-160 ## Test plan Integration and unit tests at various layers

Implements the only Cody Access write API we have right now, the ability to apply overrides to Cody Gateway rate limits, which we'll have in Enterprise Portal with https://github.com/sourcegraph/sourcegraph/pull/63858 A lot of the diff is a new generated mock for the Cody Access store so that we can write some simple unit tests. With this addition, all initially proposed Cody Gateway access APIs are implemented. Part of https://linear.app/sourcegraph/issue/CORE-218 Closes https://linear.app/sourcegraph/issue/CORE-160 ## Test plan Unit and existing integration tests

…tal (#64090) Upgrades the self-contained Cody Gateway modal to only depend on Enterprise Portal to read/write Cody Gateway access. I also made some minor tweaks to parsing empty form inputs, and also made the "Cody Gateway access token" collapsed by default, with warnings to discourage its use (opted not to remove entirely since it may be useful for dev/debugging) Depends on: - https://github.com/sourcegraph/sourcegraph/pull/63926 - https://github.com/sourcegraph/sourcegraph/pull/63925 - https://github.com/sourcegraph/sourcegraph/pull/63858 Closes https://linear.app/sourcegraph/issue/CORE-218 ## Test plan ``` sg start dotcom ``` Edit the Cody Gateway access stuff, toggling it up and down. Cross-reference with the `enterprise_portal_cody_gateway_access` table.

…e portal DB (#63927) This change follows https://github.com/sourcegraph/sourcegraph/pull/63858 by making the existing subscriptions APIs _read_ from the Enterprise Portal database, instead of dotcomdb, using the data that we sync from dotcomdb into Enterprise Portal. Part of https://linear.app/sourcegraph/issue/CORE-156 Part of https://linear.app/sourcegraph/issue/CORE-158 ## Test plan Updated unit and integration tests

#63959) This change follows https://github.com/sourcegraph/sourcegraph/pull/63858 by making the _all_ subscriptions APIs read and write to the Enterprise Portal database, instead of dotcomdb, using the data that we sync from dotcomdb into Enterprise Portal. With this PR, all initially proposed subscriptions APIs are at least partially implemented. Uses hexops/valast#27 for custom `autogold` rendering of `utctime.Time` Closes https://linear.app/sourcegraph/issue/CORE-156 Part of https://linear.app/sourcegraph/issue/CORE-158 ## Test plan - [x] Unit tests on API level - [x] Adapters unit testing - [x] Simple E2E test: https://github.com/sourcegraph/sourcegraph/pull/64057

cla-bot bot added the cla-signed label Jul 16, 2024

bobheadxi mentioned this pull request Jul 16, 2024

feat/enterpriseportal: support DevOnly in list subscriptions #63837

Merged

bobheadxi force-pushed the 07-15-feat_enterpriseportal_support_devonly_in_list_subscriptions branch from 5061fd6 to aa59edf Compare July 16, 2024 18:32

bobheadxi force-pushed the 07-16-feat_enterpriseportal_import_data_from_dotcom branch 2 times, most recently from c60dc8b to 3dba2ad Compare July 16, 2024 20:18

bobheadxi mentioned this pull request Jul 16, 2024

fix/enterpriseportal: drop old gorm fk constraints #63864

Merged

bobheadxi force-pushed the 07-15-feat_enterpriseportal_support_devonly_in_list_subscriptions branch from aa59edf to 8ff52ad Compare July 16, 2024 22:15

bobheadxi force-pushed the 07-16-feat_enterpriseportal_import_data_from_dotcom branch from 3dba2ad to 1443224 Compare July 16, 2024 22:15

bobheadxi force-pushed the 07-15-feat_enterpriseportal_support_devonly_in_list_subscriptions branch from 8ff52ad to a52025d Compare July 16, 2024 22:24

bobheadxi force-pushed the 07-16-feat_enterpriseportal_import_data_from_dotcom branch from 1443224 to 6b7eeb3 Compare July 16, 2024 22:24

bobheadxi force-pushed the 07-15-feat_enterpriseportal_support_devonly_in_list_subscriptions branch from a52025d to 10e57d2 Compare July 16, 2024 22:25

bobheadxi force-pushed the 07-16-feat_enterpriseportal_import_data_from_dotcom branch from 6b7eeb3 to 299816d Compare July 16, 2024 22:25

bobheadxi force-pushed the 07-15-feat_enterpriseportal_support_devonly_in_list_subscriptions branch from 10e57d2 to b990a28 Compare July 16, 2024 22:51

bobheadxi force-pushed the 07-16-feat_enterpriseportal_import_data_from_dotcom branch from 299816d to 2fe83b1 Compare July 16, 2024 22:51

bobheadxi force-pushed the 07-15-feat_enterpriseportal_support_devonly_in_list_subscriptions branch from b990a28 to 3f5c924 Compare July 16, 2024 22:52

bobheadxi force-pushed the 07-16-feat_enterpriseportal_import_data_from_dotcom branch from 2fe83b1 to 8f88f6c Compare July 16, 2024 22:52

bobheadxi force-pushed the 07-15-feat_enterpriseportal_support_devonly_in_list_subscriptions branch from 3f5c924 to b732ac4 Compare July 16, 2024 23:06

bobheadxi force-pushed the 07-16-feat_enterpriseportal_import_data_from_dotcom branch 2 times, most recently from 26a724c to f8d2c14 Compare July 16, 2024 23:20

bobheadxi marked this pull request as ready for review July 16, 2024 23:23

bobheadxi requested a review from a team July 16, 2024 23:23

jac approved these changes Jul 17, 2024

View reviewed changes

cmd/enterprise-portal/internal/database/importer/importer.go Outdated Show resolved Hide resolved

bobheadxi force-pushed the 07-15-feat_enterpriseportal_support_devonly_in_list_subscriptions branch from b732ac4 to 00147a6 Compare July 17, 2024 21:17

bobheadxi force-pushed the 07-16-feat_enterpriseportal_import_data_from_dotcom branch from 2e28f48 to f580d6f Compare July 17, 2024 21:17

bobheadxi mentioned this pull request Jul 18, 2024

feat/enterpriseportal: use database for reading Cody Gateway access #63925

Merged

bobheadxi force-pushed the 07-16-feat_enterpriseportal_import_data_from_dotcom branch 2 times, most recently from 783afdd to 2a6279d Compare August 2, 2024 19:24

bobheadxi mentioned this pull request Aug 2, 2024

feat/enterpriseportal: add instance category #64253

Merged

bobheadxi and others added 8 commits August 5, 2024 13:07

feat/enterpriseportal: import data from dotcom

8ac3cbe

add option to disable importer

c4d74b2

bazel configure

93fee8d

continue on invalid tag

1cf4da6

Co-authored-by: James Cotter <[email protected]>

improve ergonomics of database.NewNullString

f3afafe

import user display name

227c23f

bazel

c6d2578

add trace annotation for skipped subscription

8586fee

bobheadxi force-pushed the 07-16-feat_enterpriseportal_import_data_from_dotcom branch from 2a6279d to 8586fee Compare August 5, 2024 20:07

bobheadxi added 4 commits August 6, 2024 14:12

bazel

61f2df9

stable test

61ae86f

order

b8e35bf

add subscription import condition event

330ff47

bobheadxi mentioned this pull request Aug 7, 2024

feat/enterpriseportal: license expiry checker #64317

Merged

2 tasks

bobheadxi added 6 commits August 7, 2024 09:32

lock import worker

d66979c

bazel

49864f1

no-op tryacquire on low intervals

edc8077

instrument redislock

266283a

fix lint problem

3c96a22

fix test

8f18f2f

bobheadxi merged commit eedc12e into main Aug 7, 2024
14 checks passed

bobheadxi deleted the 07-16-feat_enterpriseportal_import_data_from_dotcom branch August 7, 2024 18:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat/enterpriseportal: import data from dotcom #63858

feat/enterpriseportal: import data from dotcom #63858

bobheadxi commented Jul 16, 2024 •

edited

Loading

bobheadxi commented Jul 16, 2024 •

edited

Loading

feat/enterpriseportal: import data from dotcom #63858

feat/enterpriseportal: import data from dotcom #63858

Conversation

bobheadxi commented Jul 16, 2024 • edited Loading

Test plan

bobheadxi commented Jul 16, 2024 • edited Loading

bobheadxi commented Jul 16, 2024 •

edited

Loading

bobheadxi commented Jul 16, 2024 •

edited

Loading