trestle import - error 403

[dfkunstler]

Describe the bug

The trestle import -f ... command described in the tutorial fails with HTTP 403 error:

Error while importing OSCAL file: Cache update failure for https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json. Please confirm the file is json and not html: GET returned code 403: https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json.

curl and wget both successfully retrieve the file at that URL without any parameters. trestle imports the file from the local FS without complaint.

To Reproduce

Steps to reproduce the behavior:

1. trestle import -f https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json -o mycatalog

Expected behavior

I expect the import to succeed as it did with input file on local file system

Screenshots / Logs.

% trestle import -f https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json -o mycatalog -vvvvv
trestle.core.remote.cache:219 DEBUG: Initializing HTTPSFetcher
trestle.core.remote.cache:58 DEBUG: Initializing FetcherBase
trestle.core.commands.import_:96 ERROR: Error while importing OSCAL file: Cache update failure for https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json.  Please confirm the file is json and not html: GET returned code 403: https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json.
Traceback (most recent call last):
  File "/workplace/dfk/trestle-tutorials/.env.trestle/lib/python3.11/site-packages/trestle/core/remote/cache.py", line 100, in _update_cache
    self._do_fetch()
  File "/workplace/dfk/trestle-tutorials/.env.trestle/lib/python3.11/site-packages/trestle/core/remote/cache.py", line 301, in _do_fetch
    raise TrestleError(f'GET returned code {response.status_code}: {self._uri}')
trestle.common.err.TrestleError: GET returned code 403: https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workplace/dfk/trestle-tutorials/.env.trestle/lib/python3.11/site-packages/trestle/core/commands/import_.py", line 68, in _run
    model_read, parent_alias = fetcher.get_oscal(True)
                               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/workplace/dfk/trestle-tutorials/.env.trestle/lib/python3.11/site-packages/trestle/core/remote/cache.py", line 142, in get_oscal
    model_dict = self.get_raw(force_update)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workplace/dfk/trestle-tutorials/.env.trestle/lib/python3.11/site-packages/trestle/core/remote/cache.py", line 110, in get_raw
    self._update_cache(force_update)
  File "/workplace/dfk/trestle-tutorials/.env.trestle/lib/python3.11/site-packages/trestle/core/remote/cache.py", line 103, in _update_cache
    raise TrestleError(
trestle.common.err.TrestleError: Cache update failure for https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json.  Please confirm the file is json and not html: GET returned code 403: https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json.

Environment

• OS: Amazon Linux 2
• Python version: 3.11.10 and 3.12.7
• Installed packages:

% pip list
Package                  Version
------------------------ -----------
annotated-types          0.7.0
anyio                    4.6.2.post1
argcomplete              3.5.1
attrs                    24.2.0
bcrypt                   4.2.1
black                    24.10.0
certifi                  2024.8.30
cffi                     1.17.1
charset-normalizer       3.4.0
click                    8.1.7
cmarkgfm                 2024.1.14
compliance-trestle       3.5.0
cryptography             42.0.4
datamodel-code-generator 0.25.3
defusedxml               0.7.1
dnspython                2.7.0
email_validator          2.2.0
et_xmlfile               2.0.0
furl                     2.1.3
genson                   1.3.0
h11                      0.14.0
httpcore                 1.0.7
httpx                    0.28.0
idna                     3.10
ilcli                    0.3.2
importlib_resources      6.4.5
inflect                  5.6.2
isort                    5.13.2
Jinja2                   3.1.4
MarkupSafe               3.0.2
mypy-extensions          1.0.0
openpyxl                 3.1.5
orderedmultidict         1.0.1
orjson                   3.10.12
packaging                24.2
paramiko                 3.4.0
pathspec                 0.12.1
pip                      24.0
platformdirs             4.3.6
pycparser                2.22
pydantic                 2.10.2
pydantic_core            2.27.1
PyNaCl                   1.5.0
python-dotenv            1.0.1
python-frontmatter       1.1.0
PyYAML                   6.0.2
requests                 2.32.3
ruamel.yaml              0.18.6
ruamel.yaml.clib         0.2.12
setuptools               65.5.0
six                      1.16.0
sniffio                  1.3.1
typing_extensions        4.12.2
urllib3                  2.2.3

[butler54]

@dfkunstler would you mind running either curl or wget on verbose from this environment and send the logs?

[dfkunstler]

Here you go:

% curl -vvvvvv https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json > /dev/null 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 185.199.108.133:443...
* Connected to raw.githubusercontent.com (185.199.108.133) port 443
* ALPN: curl offers h2,http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
{ [5 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3094 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.io
*  start date: Mar 15 00:00:00 2024 GMT
*  expire date: Mar 14 23:59:59 2025 GMT
*  subjectAltName: host "raw.githubusercontent.com" matched cert's "*.githubusercontent.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://raw.githubusercontent.com/usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: raw.githubusercontent.com]
* [HTTP/2] [1] [:path: /usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json]
* [HTTP/2] [1] [user-agent: curl/8.3.0]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /usnistgov/oscal-content/master/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json HTTP/2
> Host: raw.githubusercontent.com
> User-Agent: curl/8.3.0
> Accept: */*
> 
{ [5 bytes data]
< HTTP/2 200 
< cache-control: max-age=300
< content-security-policy: default-src 'none'; style-src 'unsafe-inline'; sandbox
< content-type: text/plain; charset=utf-8
< etag: "9f81eb303d68d8a79d01c838d6116d927fcb5e64255dd14b9633d6020981b715"
< strict-transport-security: max-age=31536000
< x-content-type-options: nosniff
< x-frame-options: deny
< x-xss-protection: 1; mode=block
< x-github-request-id: B659:109E7E:A3131E:B32327:6751D0BB
< accept-ranges: bytes
< date: Thu, 05 Dec 2024 16:15:52 GMT
< via: 1.1 varnish
< x-served-by: cache-iad-kjyo7100145-IAD
< x-cache: HIT
< x-cache-hits: 0
< x-timer: S1733415352.458263,VS0,VE1
< vary: Authorization,Accept-Encoding,Origin
< access-control-allow-origin: *
< cross-origin-resource-policy: cross-origin
< x-fastly-request-id: 794854ec533c045a7eabe7a0e58bc3cf037d74b0
< expires: Thu, 05 Dec 2024 16:20:52 GMT
< source-age: 252
< content-length: 10381604
< 
{ [5 bytes data]
100  9.9M  100  9.9M    0     0   156M      0 --:--:-- --:--:-- --:--:--  157M
* Connection #0 to host raw.githubusercontent.com left intact