feat: POC: brrr lib & demo

Functional POC.

Co-authored-by: Jesse Zwaan <[email protected]>

.editorconfig

@@ -0,0 +1,11 @@
+root = true
+
+[*]
+end_of_line = lf
+insert_final_newline = true
+trim_trailing_whitespace = true
+charset = utf-8
+
+[*.py]
+indent_style = space
+indent_size = 4

.github/workflows/ci.yaml

@@ -0,0 +1,12 @@
+on:
+  pull_request:
+  push:
+
+name: Syntax
+jobs:
+  nocommit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: "nocommit checker"
+        uses: nobssoftware/nocommit@v2

.gitignore

@@ -1 +1,7 @@
-.venv
+.venv
+.envrc
+.terraform*
+data
+__pycache__
+result
+.direnv

README.md

@@ -0,0 +1,91 @@
+# Brrr: high performance workflow scheduling
+
+Brrr is a POC for ultra high performance workflow scheduling.
+
+Differences between Brrr and other workflow schedulers:
+
+- Brrr is **queue & database agnostic**. Others lock you in to e.g. PostgreSQL, which inevitably becomes an unscalable point of failure.
+- Brrr **lets the queue & database provide stability & concurrency guarantees**.  Others tend to reinvent the wheel and reimplement a half-hearted queue on top of a database.  Brrr lets your queue and DB do what they do best.
+- Brrr **requires idempotency** of the call graph.  It’s ok if your tasks return a different result per call, but the sub tasks they spin up must always be exactly the same, with the same inputs.
+- Brrr tasks **look sequential & blocking**.  Your Python code looks like a simple linear function.
+- Brrr tasks **aren’t actually blocking** which means you don’t need to lock up RAM in your fleet equivalent to the entire call graph’s execution stack.  In other words: A Brrr fleet’s memory usage is *O(fleet)*, not *O(call graph)*.
+- Brrr offers **no logging, monitoring, error handling, or tracing**.  Brrr does one thing and one thing only: workflow scheduling.  Bring your own logging.
+- Brrr has **no agent**.  Every worker connects directly to the underlying queue, jobs are scheduled by directly sending them to the queue.  This allows *massive parallelism*: your only limit is your queue & DB capacity.
+
+N.B.: That last point means that you can use Brrr with SQS & DynamoDB to scale basically as far as your wallet can stretch without any further config.
+
+There are currently implementations for Redis and SQS as queues, and for DynamoDB as a database.
+
+## Python Library
+
+Brrr is a dependency-free Python uv bundle which you can import and use directly.
+
+Look at the [`brrr_demo.py`](brrr_demo.py) file for a full demo.
+
+Highlights:
+
+```py
+from brrr import task
+
+@task
+def fib(n: int, salt=None):
+    match n:
+        case 0: return 0
+        case 1: return 1
+        case _: return sum(fib.map([[n - 2, salt], [n - 1, salt]]))
+
+@task
+def fib_and_print(n: str):
+    f = fib(int(n))
+    print(f"fib({n}) = {f}", flush=True)
+    return f
+
+@task
+def hello(greetee: str):
+    greeting = f"Hello, {greetee}!"
+    print(greeting, flush=True)
+    return greeting
+```
+
+Note: the `fib()` function looks like it blocks for two sub-calls to `fib(n-1)` and `fib(n-2)`, but in reality it is aborted and re-executed multiple times until all its inputs are available.
+
+Benefit: your code looks intuitive.
+
+Drawback: the call graph must be idempotent, meaning: for the same inputs, a task must always call the same sub-tasks with the same arguments.  It is allowed to return a different result each time.
+
+
+## Demo
+
+### Requirements
+
+- Docker
+- [Nix](https://nixos.org)
+
+### Instructions
+
+Make sure you have Docker running, and start the full demo:
+
+```
+$ nix run github:nobssoftware/brrr#demo
+```
+
+In the process list, select the worker process so you can see its output.   Now in another terminal:
+
+```
+$ curl 'http://localhost:8333/hello?greetee=John'
+```
+
+You should see the worker print a greeting.
+
+You can also run a Fibonacci job:
+
+```
+$ curl 'http://localhost:8333/fib_and_print?n=11'
+```
+
+
+## Copyright & License
+
+Brrr was written by Hraban Luyat and Jesse Zwaan.  It is available under the AGPLv3 license (not later).
+
+See the [LICENSE](LICENSE) file.

brrr-demo.nix

@@ -0,0 +1,42 @@
+# Copyright © 2024  Brrr Authors
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published
+# by the Free Software Foundation, version 3 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+# Brrr-demo module for process-compose-flake
+
+{ config, pkgs, name, lib, ... }: {
+  options = with lib.types; {
+    # You’ll want to override this unless you use an overlay
+    package = lib.mkPackageOption pkgs "brrr-demo" { };
+    args = lib.mkOption {
+      default = [];
+      type = listOf str;
+    };
+    environment = lib.mkOption {
+      type = types.attrsOf types.str;
+      default = { };
+      example = {
+        AWS_ENDPOINT_URL = "http://localhost:12345";
+      };
+      description = ''
+        Extra environment variables passed to the `brrr-demo` process.
+      '';
+    };
+  };
+  config = {
+    outputs.settings.processes.${name} = {
+      environment = config.environment;
+      command = "${lib.getExe config.package} ${lib.escapeShellArgs config.args}";
+    };
+  };
+}