Support traffic replay

[djshow832]

Feature Request

Describe your feature request related problem

There are some cases when users want to capture the traffic on the production cluster and replay the traffic on a testing cluster:

• A new TiDB version may have compatibility breakers, such as the statements failing, running slower, or resulting in different query results.
• When the cluster runs unexpectedly, users want to capture the traffic so that they can investigate it later by replaying the traffic.
• Test the maximum throughput of a scaled-up or scaled-down cluster using the real workload instead of Sysbench or TPCC.

Some traffic replay tools are widely used, including tcpcopy, mysql-replay, and query-playback. Tcpcopy and mysql-replay capture data like tcpdump, while query-playback is based on slow logs. Although some of them are built for MySQL, deploying them on the proxy instance also works.

However, they have some limitations:

• Users need to learn the deployment and usage of these tools.
• Tcpcopy only captures new connections, which is unfriendly for persistent connections. Mysql-replay can capture existing connections but it loses session states such as prepared statements and session variables, which may make replay fail.
• The tools replay the traffic with one username and one current schema, which requires modification to the testing cluster.
• Tcpcopy and mysql-replay don't support TLS because they can't decode the encrypted data.
• Users need to verify the results and performance manually.

Describe the feature you'd like

Capturing traffic on one TiDB cluster and replaying the traffic on a new TiDB cluster to verify the SQL compatibility and performance of the new TiDB cluster.

Tasks

Design

• RFC: Traffic Capture and Replay #643

Traffic Capture

P1:

• Support capturing commands, connection ID, and start time #645
• Store traffic to files with rotation and compression #647
• Output the session states to traffic files #674
• Output disconnection to traffic files #676
• Output metadata to the output directory #672
• Filter sensitive commands #681

P2:

• Support writing huge commands
• Support more command types
• Support encryption options #717
• Support compression options #719
• Output traffic files to a remote address

Traffic Replay

P1:

• Connect to TiDB with a username and password #652
• Manage connections in traffic replay #654
• Load and parse traffic files #657
• Replay traffic at a customized speed #659
• Maintain the mapping of prepared statement ID and text #678
• Add a connection ID allocator #687
• Slow down decoding when there are too many pending commands #700

P2:

• Streaming reading and sending large commands
• Support replaying read-only statements #711
• Track and limit the memory of pending commands

P3:

• Support more auth plugins
• Support replaying as fast as possible
• Support reading from MySQL slow logs
• Support reading from tcpdump files
• Support reading from TiDB statements summary tables
• Support reading from MySQL statements summary tables

Comparison and Report

P1:

• Output failed statements and error messages into TiDB tables #661
• Output the capture and replay time of failed statements #683

P2:

• Add a timestamp column in replay tables to identify the replay job

User Interface

P1:

• Manage capture and replay jobs #665
• Support HTTP API to capture and replay traffic #667
• Support TiProxyCtl to capture and replay traffic #669
• Show the history of traffic jobs #685

P2:

• Add job start time to the report and API parameters #723
• Support SQL to capture and replay traffic
• Add a --force arg to remove the traffic file or report forcibly if it exists
• Show the estimated disk usage of the traffic files