Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Twitter branding to X #15

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/postmortems/2022-02-02-inflation-vuln.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ It also details our response, lessons learned, and subsequent changes to our pro
## Incident Summary

A vulnerability in Optimism’s fork of [Geth](https://github.com/ethereum/go-ethereum) (which we refer to as [L2Geth](https://github.com/ethereum-optimism/optimism-legacy/blob/8205f678b7b4ac4625c2afe351b9c82ffaa2e795/l2geth/README.md)) was reported
to us by [Jay Freeman](https://twitter.com/saurik) (AKA saurik) on February 2nd, 2022. If exploited,
to us by [Jay Freeman](https://x.com/saurik) (AKA saurik) on February 2nd, 2022. If exploited,
this vulnerability would allow anyone to mint an unbounded amount of ETH on Optimism.

We confirmed the issue, implemented a fix, and upgraded our infrastructure within 3 hours. We then
Expand Down
6 changes: 3 additions & 3 deletions docs/postmortems/2022-05-31-drop-1.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Summary

On May 31, 2022, prior to the official launch announcement from Optimism, some users noticed that Airdrop #1 claims were live and commented in Discord and on Twitter. Soon after, we saw a massive influx of traffic that caused several of our internal services to fail under the load. We provisioned additional capacity both internally and with our infrastructure providers in response to the failures. During this process some users experienced delays or were prevented from claiming their airdrop. Additionally, our public endpoint experienced high error rates and degraded performance, and the combination of issues caused frustration within our community.
On May 31, 2022, prior to the official launch announcement from Optimism, some users noticed that Airdrop #1 claims were live and commented in Discord and on X. Soon after, we saw a massive influx of traffic that caused several of our internal services to fail under the load. We provisioned additional capacity both internally and with our infrastructure providers in response to the failures. During this process some users experienced delays or were prevented from claiming their airdrop. Additionally, our public endpoint experienced high error rates and degraded performance, and the combination of issues caused frustration within our community.
Our internal teams worked closely with partners and infrastructure providers to mitigate issues and make claims available to all users attempting to access their tokens. The issues were resolved at 2:57 pm PST that day, with all systems operating normally.

We dramatically underestimated the amount of traffic that the airdrop would create. We’re humbled by the excitement from the community but because we didn’t expect this response, we hadn’t given Alchemy a heads up. Since nodes can take up to 26 hours to start up, we had agreed to give Alchemy a 48 hour heads up on large capacity increases. However, only until after the airdrop began did we find out we needed to 7x the capacity of our public endpoint - which meant doubling the global capacity of Optimism.
Expand Down Expand Up @@ -55,7 +55,7 @@ All times listed in PST.
- `14:10` Update is deployed to Warp Speed to clear the backlog
- `14:15` After determining it was safe to do so, batch submitter confirmation depth is further reduced to 2
- `14:17` Status page is set to monitoring
- `14:45` Airdrop is officially announced from Optimism’s Twitter account
- `14:45` Airdrop is officially announced from Optimism’s X account
- `14:57` Status page is set to resolved
- `15:44` A fix is pushed for a bug on the claims backend that prevented claims
- `16:55` Warp Speed backlog clears
Expand Down Expand Up @@ -136,7 +136,7 @@ We’re currently subsidizing the ecosystem’s addiction to the public endpoint

#### Button Up Our Practices
- **Update the status page quickly when issues occur.** It took us several hours to update the status page after the public endpoint went down. This left users in the dark, and contributed to their frustration. It also eroded trust in the status page itself. Whenever something happens that impacts a production service, the status page should be updated within minutes of confirming the issue.
- **Inform our community with timely and transparent communications.** With uncertainty about how long the launch would be delayed - and in an effort to avoid preempting planned announcement communications - we remained quiet in the midst of increased questions and speculation from the community in Discord and on Twitter. Moving forward, we’ll structure our communications to prioritize transparency and community experience.
- **Inform our community with timely and transparent communications.** With uncertainty about how long the launch would be delayed - and in an effort to avoid preempting planned announcement communications - we remained quiet in the midst of increased questions and speculation from the community in Discord and on X. Moving forward, we’ll structure our communications to prioritize transparency and community experience.
- **Monitoring and alerting is a requirement to go to prod.** The claims backend crashed, and it was our users who let us know. We need monitoring and alerting in place for every service before it goes live to production traffic. There’s too much going on on launch day to manually watch every dashboard. We need to get pushed alerts when things go down.
- **Don’t use free-tier infra internally.** We shouldn’t use the public endpoint for internal services for the same reasons the community shouldn’t. We should use a dedicated Alchemy key per service.
- **Lock down our launches.** On-chain activity is public, so whenever smart contracts are involved in a launch those contracts should be pause-able.
Expand Down