Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catch param case sensitivity on PA vote parsing #5105

Merged
merged 2 commits into from
Dec 2, 2024

Conversation

Rkuro
Copy link
Contributor

@Rkuro Rkuro commented Nov 30, 2024

Hey there this is my first PR on this project so lmk if I'm doing something wrong here!

I noticed during running scraping for PA that it seems there can be times where the roll call votes include "sessYr" but it looks like we are looking for "sessyr", and thus we don't see it and attempt to add it, leading to a 404.

Example of log output from the scraper:

00:04:35 INFO scrapelib: GET - 'https://www.palegis.us/senate/committees/roll-call-votes/vote-list/vote-summary?committeecode=17&rollcallid=1378&sessYr=2023&sessInd=0&sessyr=2023'
00:04:37 WARNING openstates: There was an error in scraping https://www.palegis.us/legislation/bills/2023/sr359: Retry 1: Error: 404 while retrieving https://www.palegis.us/404

Seems like we attempted to work around this with a recent PR here: #5086

I'm wondering if they added this onto links on the server side but they added it in a different format.

According to the RFC rfc7230 query params are actually case sensitive so I'm proposing we check for things in lowercase first, and if it does not exist then we add it in the case that the PA site seems to want there.

 

@showerst
Copy link
Contributor

Thanks @Rkuro! We'll review this and either merge for provide feedback on Monday.

Copy link
Contributor

@jessemortenson jessemortenson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Thanks for the catch & fix!

@jessemortenson jessemortenson merged commit d509cea into openstates:main Dec 2, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants