SSL EOF occurred in violation of protocol when syncing dist folder to R2 bucket #3559
Comments
|
Il try too manually tackle this in a similar flow as the releasers would use |
|
I just tried to re-enable the CF upload and ran the nightly promoter manually. It uploaded a few files without any errors. |
|
I ran the |
|
Then I ran it again for the entire downloads directory (as root this time) and after some time it failed: |
|
I don't know how to verify it, but I think it's probably caused by the old version of OpenSSL that's available on the server. |
|
Did the entire sync started with the dist user succeed? |
|
Yes, it did. |
|
Would it be possible to update the openssl version on the server without risking breaking anything? |
|
I suspect not -- usually the openssl version is tightly coupled with the Linux distribution version. |
Ahh I see,, just in case can we try syncing the entire downloads directory as the |
|
We can try, but the initial issue (#3508 (comment)) happened with the |
|
I just started a new sync with |
|
Happens with |
|
Note that there technically is an openssl update available, but we don't have access to it as it requires Ubuntu Pro |
|
If it is due to the openssl version on the server, I wonder if we should explore adding a step in CI for the build servers to sync the new builds now rather than later. Especially since these errors are happening at random it seems instead of under specific circumstances. I believe @MoLow was originally going to do that, but I don't remember if there were any problems he encountered or if he decided it was better for now to just do the sync from the server. I'm not talking about removing the upload to the DO server (we should definitely keep uploading there even after we switch to using the worker fully) but just as another step the build machines do during a release CI run. |
That was most probably me raising an objection as it required installing the s3 client on all of the release machines, but maybe that is the way to go. One of my concerns for releases from #3508 (comment) is that, even if the errors didn't occur, the releaser is basically sitting there waiting for all the release assets to be uploaded from the DO server to R2 (with very little feedback) -- ideally the proposed release assets would already be staged in R2 and the manually triggered promotion step would mark/move them (as well as upload the signed shasums) to dist-prod.
+1. For now we need to continue to upload to the DO server as the backup scripts pull from the DO server. |
|
Server was upgraded to Ubuntu 22.04 (#3564). |
|
We're running a few more runs, just to rule out the fluke scenario. |
|
In case it matters, everytime we do a full sync, the |
Did two more runs. It copied a few V8 canary files. All good. |
|
Thanks so much for this. closing |
Hey all! As a part of #3461, the
/home/dist/folder started to be synced into a R2 bucket. The behavior was added in #3505.It was working fine for nightly builds up until it came time to promote v18.18.1, where it threw a bunch of SSL errors reported here #3508 (comment).
The error is due to
EOF occurred in violation of protocol. I am not sure what exactly the issue is, however I do have some theories:Unfortunately since the sync script was failing it had to be disabled, so the R2 bucket is becoming more and more stale. This issue also blocks #3461 entirely since we're no longer getting new builds added to the bucket.
Ideally we would be able to figure out what is causing the issue, fix it, and then re-enable the sync script for new releases in addition to syncing the downloads that were added over the course of syncing being disabled
cc @ovflowd @MoLow
The text was updated successfully, but these errors were encountered: