The Hatchery

Hatchery provides a Cookiecutter template for quickly generating base projects.

Note: This is relatively new and untested. Bolivian Tree Lizards may reside here.

Usage

• Install Cookiecutter
• Run cookiecutter git+ssh://[email protected]/FundingOptions/hatchery.git
• Fill in the project details

What is included here

This currently provides a thin template for Python projects. At its core, it provides tooling configuration, and a task runner.

By extension, it provides configuration for hosting on AWS Lambda, including:

• deployment routines
• app structure (basic, web)
• bootstraping of Sentry + AWS X-Ray

Frontend

Sorry, nothing here yet. It might be we want to look into something node specific, so we've not imposed anything for us here.

Testing

Pytest

Our testing tool of choice, it's quicker to craft, and the fixture support is incredibly powerful. Also looks much cleaner than unittest.

pytest-cov

Coverage support for pytest. Best used as an indicator that we have untested code. Always remember that covered code doesn't mean it's tested

pytest-randomly

Generates seeds for Python's random module, faker, and also forces tests to run in different orders each time.

• This helps ensure that each test run is unique, but repeatable (--randomly-seed).
• By having tests run in different orders, we expose issues such as one test setting data in a database, and another depending on it being there. This helps avoid flakey tests earlier on in development.

Faker

Instead of using static data repeated everywhere, try to use generated data as much as possible.

This will:

• prevent "slimey" implementations, where the implementation isn't flexible enough in the real world
• force more code into fixtures, which thins out actual test code, and makes data declarative

A pytest fixture has been configured, named faker.

Linting

black

The uncompromising formatter. This takes PEP8, applies some opinions onto it, then just formats the code. Safely.

This was selected as "It Just Works", and is relatively stable. It's also been adopted by the PSF

autoflake

Remove all our unused variables + imports, to keep things a bit cleaner and less dead.

isort

Sorts our imports in accordance with the black standards. Who doesn't want alphabetised imports.

Task runner

GNU Make

make has been selected as it's ubiquitously installed by default amongst most Operating Systems (Sorry Windows).

Whilest it's not the easiest system to work with, it does allow us to quickly write repeated commands, though without the option for having parameters.

a make help has also been written to allow us to write CLI docs for our make targets, even if it is a little hacky.

Configs

.gitignore

a .gitignore file is generated by the gitignore.io service.

This service is continually updated with the latest tools + languages, and provides an excellent base to work from.

.editorconfig

a .editorconfig is added, with some useful defaults. What this amounts to is:

• forces utf-8 for all files
• forces all files to use lf as the line ending (great as we're a multi-OS team)
• ensures we always have a new line at end of files
• trims any trailing whitespace
• sets indenting to 4 spaces
  • Makefiles use tabs with a width of 4 spaces
  • Terraform files use 2 space indentation to align with terraform fmt

These enforcements are subject to using an editor with .editorconfig support. See here to see if your editor is supported.

.circleci/config.yml

A generic workflow has been created, that Tests and Lints.

CircleCI was chosen as it's easy to setup, and doesn't require us to self-host.

Diagnosis

Hatchery configures a few useful tools out of the box for us to help diagnose bugs we have, whether silent or loud.

Sentry

Sentry is initialised out of the box, and loads the Lambda Integration to provide additional context to error reporting.

Useful for knowing when we get critical errors within the application, or logging warnings about potential future problems.

AWS X-Ray

X-Ray provides tracing to us, including:

• All outgoing requests, including destination + timings
• boto3 usage (AWS SDK)
• Database connections + queries (sqlalchemy integration)

The traces join together with traces from other services, to give us a full service map of everything going on.

Useful for verifying services interact correctly, and for gathering timing stats on arbitrary branches.

Web

We recommend using FastAPI in Hatchery Projects going forward. There are a few reasons as to why it's been used for recent projects:

Doc Generation

The design of FastAPI results in the need to create models to represent and validate our request and response structures. That is to say, we need to declare what's actually going to be returned to the user.

This means that documentation can be generated from what amounts to be our domain objects. An example of this is in the ibis project.

Simple Controllers

FastAPI takes inspiration from Flask, to allow us to define plain controllers.

It does this by use of type hints. a Query parameter is defined as part of a normal signature.

The bellow allows for:

• a GET requests to be sent to /hello/
• an optional param name, which defaults to the string "World"
• a required param age, which must be an integer
• returns a JSON object of Hello="{name}", age={age}

@api.get("/hello/")
def hello(age: int, name: str = "World"):
  return {"Hello": name, "age": age}

Extensive Validation

The key to clean, stable code is to ensure we always have what we expect.

FastAPI, by use of pydantic, has first line support for validating incoming requests and responses.

This is very good, but can take a bit of getting used to:

• Provided we define our expectations, we will never receive an invalid request
• Provided we have defined our Domain objects well, we will never give an invalid response

The result is the burden of ensuring we're in the correct state is up front. Provided the application is layered well, and the only tests are not just following the "happy path", we'll catch most of the issues before go live.

Future: Support for GraphQL

There's been a lot of talk about GraphQL, especially now we're using Gatsby on the frontend. As FastAPI provides support for GraphQL, and Pydantic has a plugin to also support schema generation, it could be a good starting point in the future.

Future

Tools

Some tools that I'd like to look into going forward:

• mypy, for static type checking in Python
• hypothesis, For a bit more structure on fuzzy testing
• ptyest-bdd, for writing User Stories as Tests
• pre-commit, for integrating Git Hooks
• Github Actions, For setting up Repo actions when hosted