Updates for W3S at scale and adding architecture (that page needs work and diagram).

docs/concepts/architecture.md

@@ -0,0 +1,24 @@
+---
+description: Learn more about Web3Signer's architecture.
+sidebar_position: 3
+---
+
+# Architecture 
+
+Web3Signer is a remote signing client comprised of three main components:
+
+- Remote signer
+- Slashing database
+- APIs 
+
+## The Remote Signer
+
+The remote signer [loads private keys](../how-to/load-keys.md) into memory and responds to signature requests. If you are using an [HSM](../how-to/store-keys/hsm/_category_.json) or a [vault](../how-to/store-keys/vaults/_category_.json) for Execution Layer signing, the keys remain at rest. This component communicates with the slashing database, the APIs, and with the keystore, if used, to coordinate remote signing. 
+
+## The Slashing Database
+
+The [slashing database](./slashing-protection.md) Postgres database keeps track of what messages have been signed and by what key. Database locking ensures that if Web3Signer instances load the same keys, only one Web3Signer instance actually signs. 
+
+## The APIs
+
+Web3Signer supports REST and [JSON-RPC APIs](../reference/api/_category_.json) to sign consensus layer and execution layer payloads respectively. These connections should be carefully secured. Web3Signer offers [TLS communication](../how-to/configure-tls.md).

docs/get-started/key-best-practices.md

@@ -0,0 +1,19 @@
+---
+title: Private Key Management Best Practices
+description: Apply best practices to keep keys and Web3Signer secure.
+sidebar_position: 5
+---
+
+When running Web3Signer, there are several best practices to be aware of for proper key management. 
+
+- Generate secure BLS keys
+- Store keys in a vault or HSM - Ideally this should be encrypted which can’t be done right now
+- Use environment authentication rather than password and token authentication where possible with vaults or HSMs
+- Only expose the validator signing API on the necessary network interface
+- Enable TLS authentication between the validator client and web3signer
+- Restrict host access to Web3Signer with `--http-host-allowlist`
+- Disable the key manager API
+    - Alternatively, restrict access to the API entirely
+- Configure the Postgres database with TLS authentication
+- Restrict access to the key config, limiting read access to Web3Signer
+- Run web3signer in a secure enclave e.g. https://aws.amazon.com/blogs/database/aws-nitro-enclaves-for-running-ethereum-validators-part-2

docs/how-to/run-at-scale.md

@@ -0,0 +1,50 @@
+---
+sidebar_label: Run Web3Signer at scale
+description: Configure your instance for better performance at scale.
+sidebar_position: 8
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Running Web3Signer at Scale
+
+There are a few key things to be aware of when running Web3Signer at scale. If you are managing hundreds or thousands of keys, these can help with attestation performance on your validators. 
+
+![architecture-diagram](../../static/img/transparent_background_diagram.png)*Credit: Kiln. Architecture diagram of Web3Signer at scale*
+
+Horizontal scale helps with request latency on Web3Signer. Connecting these instances to the same slashing database will ensure low signing latency with high safety, but there are some things to consider when configuring your environment. The majority of cost is bared at startup, with more keys causing Web3Signer to take longer to start up. This is a one time cost (per restart).
+
+## Database Proximity
+
+The [slashing database](./configure-slashing-protection.md) is crucial to managing many validators safely. Reducing latency and overhead on this database will greatly improve performance. 
+
+* **Reduced Geographic Latency**: Strategically place Web3Signer instances to ensure minimal distance to the slashing protection database.
+* **Performance Tuning**: Optimize database configurations for rapid access, considering factors like indexing and connection pooling.
+
+## Threading Model Optimization
+
+Web3Signer's threading framework, [Vertx](https://vertx.io/docs/vertx-core/java/), while powerful, is not suitable for all environments without configuration. If you are experiencing request latency or blocked threads, consider adjusting the [worker pool size](../reference/cli/options.md#vertx-worker-pool-size).
+
+* **Concurrency Management**: Tailor the size of Web3Signer's thread pool to the expected load, preventing bottlenecks. If you see decreased attestation performance while the signing load is at its peak, increase the pool size.
+* **Dynamic Adjustments**: Implement monitoring tools to adjust threads in real-time based on current demand and workload. You should measure spikes and adjust the pool accordingly. We provide two [metrics](./monitor/metrics.md) that may help here: 
+    * http_vertx_worker_queue_delay: time spent by requests in queues before being processed
+    * http_vertx_worker_pool_completed_total : number of queries processed by Web3Signer
+
+## Load Balancing
+
+At scale, you will likely need multiple instances of Web3Signer connected to a load balancer. You want to ensure a balanced distribution of requests.
+
+* **Request Distribution**: Apply an ingress load balancer to ensure requests are evenly spread, preventing overloading of single instances. Using the same slashing database allows for replicas of Web3Signer to sign in parallel without slashing risk. 
+
+Credit to our friends at [Kiln](https://www.kiln.fi/) for this [helpful article](https://www.kiln.fi/post/learnings-from-running-web3signer-at-scale-on-holesky).
+
+## Hardware Recommendations
+
+We test our nodes with 10k keys on various testnets like Goerli. In an example, we use a single cloud VM instance running Besu, Teku, and Web3Signer on an Azure Standard D8as v5 (8 vCPUs, 32 GiB memory). This may be more than needed, depending on your usage (see the example dashboard below).
+
+![Dashboard for Web3Signer](../../static/img/dashboard_hw.png)
+
+The Web3Signer process alone, with 10K keys uses < 2GB of JVM heap.
+
+Here we have assumed only one validator client connecting to Web3Signer. Multiple VCs may change requirements, although the number of requests should be the same if it's the same 10K keys partitioned across multiple clients.

docs/how-to/store-keys/index.md

@@ -8,7 +8,7 @@ import TabItem from '@theme/TabItem';
 
 # Store signing keys
 
-Web3Signer supports BLS12-381 or secp256k1 signing keys stored in the following ways:
+Web3Signer supports BLS12-381 (`Eth2`) or secp256k1 (`Eth1`) signing keys stored in the following ways:
 
 | Key storage                          | SECP256K1 | BLS |
 |--------------------------------------|:---------:|:---:|
@@ -24,6 +24,8 @@ Web3Signer supports BLS12-381 or secp256k1 signing keys stored in the following
 | [YubiHSM 2]                          |     x     |  x  |
 | [USB Armory Mk II]                   |     x     |  x  |
 
+Web3Signer supports `Eth1` signing from HSMs and Vaults, but needs to load private keys into memory for `Eth2` signing.  Make sure to follow [best practices](../../get-started/key-best-practices.md) when storing private keys.
+
 After storing keys, [load keys into Web3Signer](../load-keys.md).
 
 <!-- links -->