Merge pull request #9 from stefafafan/readme

add to the README
stefafafan · Nov 23, 2024 · a50ddd9 · a50ddd9
2 parents 936632b + 9ae594a
commit a50ddd9
Showing 1 changed file with 150 additions and 57 deletions.
diff --git a/README.md b/README.md
@@ -4,11 +4,70 @@ seki is a CLI tool to help parse and aggregate access log data for further analy
 
 The name seki comes from the Japanese terms 積 (from 蓄積 which means accumulate) and 析 (from 分析 which means analysis). These kanji are both read "seki".
 
+This tool is heavily influenced by [tkuchiki/alp](https://github.com/tkuchiki/alp) and [matsuu/kataribe](https://github.com/matsuu/kataribe).
+
+## Why seki?
+
+Other tools do everything for you from aggregation to pretty-print display. For most users these tools are sufficient.
+
+seki focuses on parsing and outputting metrics as JSON, for easy integration and processing.
+
 ## Installation
 
 TODO
 
-## Example
+## Usage
+
+seki reads access log in JSON format from stdin, and outputs to stdout as JSON.
+
+```sh
+$ cat access.log | seki
+```
+
+seki parses the following keys, make sure the access log has these:
+- `method`
+- `uri`
+- `status`
+- `response_time`
+
+For nginx, set the log_format to something like this:
+
+```nginx.conf
+log_format json escape=json '{'
+                            '"time":"$time_iso8601",'
+                            '"host":"$remote_addr",'
+                            '"method":"$request_method",'
+                            '"uri":"$request_uri",'
+                            '"status":"$status",'
+                            '"body_bytes":"$body_bytes_sent",'
+                            '"referer":"$http_referer",'
+                            '"ua":"$http_user_agent",'
+                            '"request_time":"$request_time",'
+                            '"response_time":"$upstream_response_time"'
+                            '}';
+```
+
+### Grouping endpoints with the config file
+
+Sometimes you will want to group endpoints together (for example, `/post/123` and `/post/456`).
+
+Create a `config.toml` file to define these groupings.
+
+```sh
+$ cp config.toml.example config.toml
+$ # edit config.toml with your favorite editor.
+```
+
+You can either use the default path to the config file, or specify with the command line option.
+
+```sh
+# uses config.toml if it exists
+$ cat access.log | seki
+# setting the path to the config file.
+$ cat access.log | seki --config ~/foo/config.toml
+```
+
+## Examples
 
 Let's say you have an access log file like this:
 
@@ -25,70 +84,85 @@ seki aggregates the logs into one simple JSON.
 ```sh
 $ cat access.log | seki
 [
-    {
-        "uri": "/foobar",
-        "method": "GET",
-        "count": 2,
-        "status_code": {
-            "1xx": 0,
-            "2xx": 2,
-            "3xx": 0,
-            "4xx": 0,
-            "5xx": 0
-        },
-        "response_time": {
-            "min": 0.476,
-            "max": 0.732,
-            "sum": 1.208,
-            "avg": 0.604,
-            "p99": 0.732
-        }
+  {
+    "method": "GET",
+    "uri": "/hello",
+    "count": 1,
+    "status_code": {
+      "status_1xx": 0,
+      "status_2xx": 0,
+      "status_3xx": 1,
+      "status_4xx": 0,
+      "status_5xx": 0
     },
-    {
-        "uri": "/",
-        "method": "GET",
-        "count": 1,
-        "status_code": {
-            "1xx": 0,
-            "2xx": 0,
-            "3xx": 0,
-            "4xx": 1,
-            "5xx": 0
-        },
-        "response_time": {
-            "min": 0.239,
-            "max": 0.239,
-            "sum": 0.239,
-            "avg": 0.239,
-            "p99": 0.239
-        }
+    "response_time": {
+      "min": 0.0,
+      "max": 0.113,
+      "avg": 0.113,
+      "sum": 0.113,
+      "p50": 0.113,
+      "p75": 0.113,
+      "p90": 0.113,
+      "p95": 0.113,
+      "p99": 0.113
+    }
+  },
+  {
+    "method": "GET",
+    "uri": "/foobar",
+    "count": 2,
+    "status_code": {
+      "status_1xx": 0,
+      "status_2xx": 2,
+      "status_3xx": 0,
+      "status_4xx": 0,
+      "status_5xx": 0
+    },
+    "response_time": {
+      "min": 0.0,
+      "max": 0.732,
+      "avg": 0.604,
+      "sum": 1.208,
+      "p50": 0.732,
+      "p75": 0.732,
+      "p90": 0.732,
+      "p95": 0.732,
+      "p99": 0.732
+    }
+  },
+  {
+    "method": "GET",
+    "uri": "/",
+    "count": 1,
+    "status_code": {
+      "status_1xx": 0,
+      "status_2xx": 0,
+      "status_3xx": 0,
+      "status_4xx": 1,
+      "status_5xx": 0
     },
-    {
-        "uri": "/hello",
-        "method": "GET",
-        "count": 1,
-        "status_code": {
-            "1xx": 0,
-            "2xx": 0,
-            "3xx": 1,
-            "4xx": 0,
-            "5xx": 0
-        },
-        "response_time": {
-            "min": 0.113,
-            "max": 0.113,
-            "sum": 0.113,
-            "avg": 0.113,
-            "p99": 0.113
-        }
+    "response_time": {
+      "min": 0.0,
+      "max": 0.239,
+      "avg": 0.239,
+      "sum": 0.239,
+      "p50": 0.239,
+      "p75": 0.239,
+      "p90": 0.239,
+      "p95": 0.239,
+      "p99": 0.239
     }
+  }
 ]
 ```
 
-This can now be used in conjunction with jq. The following sample demonstrates sorting the `uri`s by `response_time.sum`.
+### Using seki with jq
+
+The previous example can be used in conjunction with [jq](https://github.com/jqlang/jq). The following sample demonstrates sorting the `uri`s by `response_time.sum`.
 
 ```sh
-$ cat access.log | seki | jq 'sort_by(-.response_time.sum) | .[] | {uri: .uri, response_time_sum: .response_time.sum}'
+$ cat access.log | seki | \
+    jq 'sort_by(-.response_time.sum) | .[] | {uri: .uri, response_time_sum: .response_time.sum}'
 {
   "uri": "/foobar",
   "response_time_sum": 1.208
@@ -102,3 +176,22 @@ $ cat access.log | seki | jq 'sort_by(-.response_time.sum) | .[] | {uri: .uri, r
   "response_time_sum": 0.113
 }
 ```
+
+If you prefer a tabular format, maybe you can use jq in conjunction with the column command to do this:
+
+```sh
+$ cat access.log | seki | \
+    jq -r "sort_by(-.response_time.sum) | \
+        .[] | \
+        [.method + \" \" + .uri, \
+        .status_code.status_2xx, \
+        .status_code.status_3xx, \
+        .status_code.status_4xx, \
+        .status_code.status_5xx, \
+        .response_time.sum] | \
+        @tsv" | \
+    column -t
+GET  /foobar  2  0  0  0  1.208
+GET  /        0  0  1  0  0.239
+GET  /hello   0  1  0  0  0.113
+```