support multiple shas (#88)

* Adds an infinite number of bundles

Also the /graphql sticks in the first commit, which is undesired. There
are also some debugging messages that need to be cleaned up.

* special middleware to serve /graphql

* add new bundles to cache, with ttl

The ttl is not currently enforced (no cache invalidation yet).

Additionally some methods have been fixed in order to obtain data from
the latest bundles.

* invalidate cache and print cache info

* remove unnecessary lines

* Fix reloading problems. Requires version updates.

* rename existing tests

* remove unnecessary lines

* add multisha tests

* refactor bundle removal logic

* lint

* bump qontract-server version

* add new cache and router metrics

* refactor metrics

* typo

* support redirecting GET /graphql to POST /graphql

* renamed metrics

* typo

* typo in default BUNDLE_SHA_TTL

* implements bad bundle reload test

* rename metrics

* update README.md

* enable yarn lint test in travis

* fix linting

.travis.yml

@@ -3,3 +3,6 @@ node_js:
   - "10"
 notifications:
   email: false
+script:
+  - "yarn run lint"
+  - "yarn test"

README.md

@@ -21,6 +21,56 @@ The Reconcile loop is implementation specific. Any tool that conforms with the f
 
 An example of an implementation reconcile tools can be obtained from here: [qontract-reconcile](https://github.com/app-sre/qontract-reconcile).
 
+## Configuration parameters
+
+This server is configured via environment variables.
+
+- `BUNDLE_SHA_TTL`: (not required) Expiration time for bundles. Defaults to `20m`.
+- `LOAD_METHOD`: (required) `fs` | `s3`. Source of the data.
+- `AWS_ACCESS_KEY_ID`: (required if `LOAD_METHOD=fs`) AWS access key ID.
+- `AWS_SECRET_ACCESS_KEY`: (required if `LOAD_METHOD=fs`) AWS secret access key.
+- `AWS_REGION`: (required if `LOAD_METHOD=fs`) AWS region.
+- `AWS_S3_BUCKET`: (required if `LOAD_METHOD=fs`) AWS s3 bucket name.
+- `AWS_S3_KEY`: (required if `LOAD_METHOD=fs`)  AWS s3 key name.
+
+## Bundle Caching
+
+This server is able to store multiple bundle versions. Each time the data is reloaded, the new bundle is exposed on its new `<sha>` on `POST /graphqlsha/<sha>`. Previous shas will continue to work until they have expired.
+
+The shas will expire after a certain amount of time:
+
+- When the data is loaded for the first time, the expiration time is set to 20 minutes in the future (can be overriden by the `BUNDLE_SHA_TTL` environment variable).
+- Each time a sha is queried specifically the expiration is refreshed to the `BUNDLE_SHA_TTL` in the future again. This means that shas can be kept available forever by querying them before the `BUNDLE_SHA_TTL` has passed.
+- The latest sha, which is the one pointed at by `POST /graphql` will never expire.
+- Shas are only expired when `GET /reload` is queried.
+- If `GET /reload` is called and there is no new data available, then no shas will be expired.
+
+## API
+
+- `POST /graphqlsha/:sha`: the request body should contain the GraphQL query. The query will be directed at the specified bundle.
+- `POST /graphql`: the request body should contain the GraphQL query. The query will be directed at the latest bundle.
+- `GET /graphql`: redirects to `POST /graphql`.
+- `GET /sha256`: returns the sha of the latest bundle.
+- `GET /git-commit`: returns the git commit for the latest bundle.
+- `GET /git-commit/:sha`: returns the git commit for the specified bundle.
+- `GET /cache`: returns a json with the cache information.
+- `GET /reload`: reloads data from the configured data source.
+- `GET /metrics`: prometheus metrics.
+
+## Metrics
+
+This server exposes prometheus metrics under `/metrics`.
+
+It includes some custom metrics:
+
+- `qontract_server_reloads_total`: Number of reloads for qontract server.
+- `qontract_server_datafiles`: Number of datafiles for a specific schema.
+- `qontract_server_router_stack_layers`: Number of layers in the router stack.
+- `qontract_server_bundle_object_shas`: Number of shas cached by the application in the bundle object.
+- `qontract_server_bundle_cache_object_shas`: Number of shas cached by the application in the bundleCache object.
+
+In addition, it also contains the metrics exposed by the [express prometheus bundle](https://github.com/jochen-schweizer/express-prom-bundle). Note that the `/graphqlsha/<sha>` path has been normalized to avoid cardinality explosion.
+
 ## Development Environment
 
 ### Setting up yarn
@@ -54,7 +104,7 @@ The data files bundle is required to start the server. Once you're in the `qontr
 ```sh
 make bundle
 ```
-Note that this requires Docker to be running on the host. 
+Note that this requires Docker to be running on the host.
 
 Optionally, if you want to specify the path for the app-interface repo on your local filesystem, you can use the parameter:
 * `APP_INTERFACE_PATH` - (optional) path to a local app-interface repo (Default: `$PWD/../../service/app-interface`).

package.json

@@ -1,6 +1,6 @@
 {
   "name": "qontract-server",
-  "version": "1.0.0",
+  "version": "2.0.0",
   "description": "qontract graphql server",
   "author": "Red Hat App SREs <[email protected]>",
   "license": "Apache-2.0",
@@ -16,7 +16,7 @@
     "update-graphql-schema": "scripts/update-graphql-schema.js test/graphql_schema.json $(find test -name '*.data.json')"
   },
   "dependencies": {
-    "apollo-server-express": "^2.2.0",
+    "apollo-server-express": "^2.18.0",
     "aws-sdk": "^2.364.0",
     "bufferutil": "^4.0.1",
     "dotenv": "^8.2.0",

src/metrics.ts

@@ -0,0 +1,75 @@
+import * as express from 'express';
+import * as db from './db';
+
+import promClient = require('prom-client');
+const promBundle = require('express-prom-bundle');
+
+interface IAcct {
+  [key: string]: number;
+}
+
+// metrics middleware for express-prom-bundle
+export const metricsMiddleware = promBundle({
+  includeMethod: true,
+  includePath: true,
+  normalizePath: [
+    ['^/graphqlsha/.*', '/graphqlsha/#sha'],
+  ],
+  buckets: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10],
+  formatStatusCode: (res: express.Response) => `${Math.floor(res.statusCode / 100)}xx`,
+});
+
+// enable prom-client to expose default application metrics
+promClient.collectDefaultMetrics({ prefix: 'qontract_server_' });
+
+// Create metric stores
+const reloadCounter = new promClient.Counter({
+  name: 'qontract_server_reloads_total',
+  help: 'Number of reloads for qontract server',
+});
+
+const datafilesGauge = new promClient.Gauge({
+  name: 'qontract_server_datafiles',
+  help: 'Number of datafiles for a specific schema',
+  labelNames: ['schema'],
+});
+
+const routerStackGauge = new promClient.Gauge({
+  name: 'qontract_server_router_stack_layers',
+  help: 'Number of layers in the router stack',
+});
+
+const bundleGauge = new promClient.Gauge({
+  name: 'qontract_server_bundle_object_shas',
+  help: 'Number of shas cached by the application in the bundle object',
+});
+
+const bundleCacheGauge = new promClient.Gauge({
+  name: 'qontract_server_bundle_cache_object_shas',
+  help: 'Number of shas cached by the application in the bundleCache object',
+});
+
+export const updateCacheMetrics = (app: express.Express) => {
+  routerStackGauge.set(app._router.stack.length);
+  bundleGauge.set(Object.keys(app.get('bundles')).length);
+  bundleCacheGauge.set(Object.keys(app.get('bundleCache')).length);
+};
+
+export const updateResourceMetrics = (bundle: db.Bundle) => {
+  // Count number of files for each schema type
+  const reducer = (acc: IAcct, d: any) => {
+    if (!(d.$schema in acc)) {
+      acc[d.$schema] = 0;
+    }
+    acc[d.$schema] += 1;
+    return acc;
+  };
+  const schemaCount: IAcct = bundle.datafiles.reduce(reducer, {});
+
+  // Set the Gauge based on counted metrics
+  Object.keys(schemaCount).map(schemaName =>
+    datafilesGauge.set({ schema: schemaName }, schemaCount[schemaName]),
+  );
+
+  reloadCounter.inc(1);
+};