add new serverless-framework-sqs-dynamodb kit #709

dustinsgoodman · 2022-12-23T17:56:59Z

Type of change

Documentation change
Bug fix
New kit

Summary of change

Adds a new starter kit to the project that utilizes a fully serverless AWS stack. This kit includes:

Checklist

These changes follow the contributing guidelines
This starter kit has been approved by the maintainers
Changes for this new starter kit are being pushed to an integration branch instead of main
All dependencies are version locked
~~[ ] This fix resolves #~~
~~[ ] I have verified the fix works and introduces no further errors~~

- initialize package.json - add .gitignore with v1 values - add .nvmrc and set to node16 - add init README with some scripts - add start, build, and infrastructure scripts - install deps for establishing and running the project initially

netlify · 2022-12-23T17:57:04Z

✅ Deploy Preview for starter-dev canceled.

Name	Link
🔨 Latest commit	`fea8909`
🔍 Latest deploy log	https://app.netlify.com/sites/starter-dev/deploys/63c80072afd6a2000914aeb6

aws-amplify-us-east-1 · 2022-12-23T17:59:28Z

This pull request is automatically being deployed by Amplify Hosting (learn more).

Access this pull request here: https://pr-709.d2zmkukv1jdnvq.amplifyapp.com

- add job generator - add job processor - add dynamodb stream config and handlers - add sqs utils

See completecoding/serverless-auto-swagger#16

mbicknese

Awesome to have this as a starter! Left some food for thought, but no show-stoppers.

mbicknese · 2023-01-02T10:19:12Z

starters/serverless-framework-sqs-dynamodb/docker-compose.yml

+  # configures DynamoDB resource for persistent storage
+  dynamodb:
+    image: amazon/dynamodb-local
+    hostname: dynamodb


Small remark, this attribute seems redundant as the service name is used as hostname by default. It stood out to me as the other services don't have it specified.

It is, but isn't. If someone were to change the docker name, which they totally can, but this attribute allows me to reference it later in the doc for the db-admin feature so it's necessary.

mbicknese · 2023-01-02T10:21:45Z

starters/serverless-framework-sqs-dynamodb/package.json

+	"name": "serverless-framework-sqs-dynamodb",
+	"version": "1.0.0",
+	"description": "A starter kit utilizing the Serverless Framework in conjunction with AWS DynamoDB & SQS.",
+	"author": "Dustin Goodman",


Maybe you don't want to put yourself up as "THE" author.

Well I am THE author of this kit 😉 - I do want to start introducing authors to kits though for better attribution to make this project more open source friendly and crediting.

@TapaiBalazs added some good context. I was under the impression that we didn't have it on the other kits, due to it being kits of us all. But maybe I'm just a bit too much of a communist 😉

I just read more on the official docs on this: https://docs.npmjs.com/cli/v9/configuring-npm/package-json#people-fields-author-contributors

Looks like we need author & contributor fields. IMO the person who defined the kit elements is the author and everyone who helped make it what it is are the contributors.

I understand your point, but the generator script currently does the following:

User runs the generator

User selects which kit they want, and then provide a name

the script downloads the folder of the starter kit from github

the package.json "name" and "version" properties get overwritten

Therefore if Jane Doe generates this starter kit, their package.json will contain you as the author in their project. This is the reason why other kits does not have the author field in the package.json file.

Of course, the generator script can be updated to remove the author from the package.json, we could even add a third prompt, so the user could set their own author field during script run.

Update: we could update the script to get the user name and e-mail from their git configuration and set them as the author field.

Hmmm... this raises a good discussion point. Something I've been pondering is if we put a "starter-dev.json" or equivalent file in each kit that we use instead of the package.json for the CLI. In general, here are the problems I'm thinking about and wanting to solve:

Are contributors properly given credit for their submissions and work regardless of the initiation point?

Can we support more non-Node based starter kits? Right now the Deno kit has a bit of a hack to be included but what if we started to include non-JavaScript tooling as well? We need to maybe rethink our CLI strategy in this regard.

I'll be honest that I'm less concerned about the generated kit containing the original authors details. I think that's actually appropriate in this case. Renaming was to allow others to have a better project name custom to what they were doing out of the box. I think this other metadata file I'm thinking about is maybe the solve though so maybe this is a moot point.

mbicknese · 2023-01-02T10:25:30Z

starters/serverless-framework-sqs-dynamodb/serverless.ts

+			},
+		},
+		'esbuild': {
+			packager: 'yarn',


I suppose there's no simple way to set this to the package manager used to install the deps?

My concern is that not all devs have yarn locally, but all node installations come with npm.

I'll note this in the README in the serverless section. That's a really good shoutout. I was developing using yarn so I needed it set to this and my instructions recommend it.

mbicknese · 2023-01-02T10:48:08Z

starters/serverless-framework-sqs-dynamodb/src/handlers/example_stream_processor.ts

+export const handler: DynamoDBStreamHandler = async (event) => {
+	console.log('Example Stream Processor Handler initiated');
+
+	const recordHandler = async (record: DynamoDBRecord) => {


Seeing this method does not rely on the handlers state, we could move it out of the body. It's a micro-optimization, I'm aware, but it might set a good example.

That's a great recommendation. I concur.

mbicknese · 2023-01-02T11:08:46Z

starters/serverless-framework-sqs-dynamodb/src/utils/cache/constants.ts

@@ -0,0 +1 @@
+export const DEFAULT_CACHE_TIME = 300_000; // 5 minutes = 1000 ms/s * 60 s/min  * 5 min


Underscore thousand separators, nice!

TapaiBalazs

Looks alright, but I have some comments and questions

TapaiBalazs · 2023-01-02T08:48:45Z

packages/website/src/icons/index.ts

@@ -38,7 +38,10 @@ export { ShareIcon } from './ShareIcon';
 export { LinkedinIcon } from './LinkedinIcon';
 export { QwikIcon } from './QwikIcon';
 export { SolidJsIcon } from './SolidJsIcon';
-export { DenoIcon } from './DenoIcon.tsx';
+export { DenoIcon } from './DenoIcon';


TapaiBalazs · 2023-01-02T08:54:49Z

starters/serverless-framework-sqs-dynamodb/README.md

+
+### General Commands
+
+- `build` bundles the project using the serverless packaging serverless. The produced artifacts will ship bundles shipped to AWS on deployment. You can optionally pass `--analyze <function name>` to run the bundle analyzer and visualize the results to understand your handler bundles.


using the serverless packaging serverless.

This feels a bit strange wording to me.

will ship bundles shipped to AWS on deployment

This too feels a bit strange

LOL - I was tired when I wrote these. Will fix. Good catch.

TapaiBalazs · 2023-01-02T09:00:25Z

starters/serverless-framework-sqs-dynamodb/README.md

+- `deploy` ships the project to the configured AWS account using the Serverless Framework CLI command.
+- `start` runs the `serverless-offline` provided server for local development and testing. Be sure to have the local docker infrastructure running to emulate the related services.
+- `test` runs `jest` under the hood.
+- `lint` runs `eslint` under the hood. You can use all the eslint available command line arguments. To lint the entire project, run `yarn lint .`, or equivalent. You can affix `--fix` to auto-correct linting issues that eslint can handle.


You can use all the eslint available command line arguments

I think You can use all the available eslint command line arguments would be the word order

TapaiBalazs · 2023-01-02T09:01:17Z

starters/serverless-framework-sqs-dynamodb/README.md

+
+This README uses `yarn` for commands. If you're using `npm` or `pnpm`, utilize the equivalent version of the commands.
+
+1. Create a `.env` file:


Do you need to have a set up, configured and logged in AWS account before these steps? 🤔

Technically speaking - no. This kit works FULLY offline and does not require AWS at all. That being said, for deployment and other things, it might be nice later. Will add notes.

TapaiBalazs · 2023-01-02T09:09:09Z

starters/serverless-framework-sqs-dynamodb/README.md

+
+### Infrastructure Commands
+
+- `infrastructure:up` creates docker container and related images and runs them in the background. This should only be needed once during initial setup.


Based on the docker-compose.yml the infrastructure creates 4 containers (dynamodb, dynamodb-admin, sqs, and redis), so in these instructions I think docker containers should be used in the sentences.

LOL - good catch.

Actually, looking at the Docker Desktop dashboard, it's 1 container with 4 images running so I think what I have is correct

I don't mean to be annoying on this, but that's 4 containers running 4 images. As far as I understand docker: 1 container runs 1 image. On your screenshot you have 4 running containers.

What you call a container with the dropdown that's just UI grouping, if you run docker ps you will see all four of those as separate containers with their separate container ID

I interpret the screenshot as 1 container, 4 images but I won't admit to being a Docker expert. I will ping a DevOps buddy to confirm.

Dustin's friend here. @TapaiBalazs is correct. If you look at the docker-compose file, you'll see that each one of those containers are defined with their own image and configuration.

TIL - thanks @mpstein ❤️

TapaiBalazs · 2023-01-02T11:06:32Z

starters/serverless-framework-sqs-dynamodb/src/models/technology/destroy.ts

+import { getCacheKey } from './getCacheKey';
+
+export const destroy = async (key: string) => {
+	const item = await deleteItem(process.env.TECHNOLOGIES_TABLE, { id: key });


does the deleteItem method return the deleted entry? 🤔

hat's what the unit test validates, but it feels strange for me after postgresql, where delete did not return anything, just information about how many rows were affected and the rest endpoint returns with OK.

My implementation has chosen to return the record. It doesn't necessarily have to do this, but DynamoDB does return it, therefore, I do.

Understood, I just wanted confirmation on the dynamodb implementation.

TapaiBalazs · 2023-01-02T11:07:25Z

starters/serverless-framework-sqs-dynamodb/src/models/technology/create.ts

+		return newTechnology;
+	}
+
+	return null;


Do we suppress an error here? If yes, shouldn't we log it? Should this return null ever be reached? 🤔

No error suppression so this is good as is.

I was now able to put together a mental model of the structure and understand that this is not the handler, but a level down.

TapaiBalazs · 2023-01-02T11:12:26Z

starters/serverless-framework-sqs-dynamodb/src/utils/isOffline/isOffline.ts

@@ -0,0 +1,7 @@
+/**
+ * Utility function for checking where functions are being run locally via serverless offline


Suggested change

* Utility function for checking where functions are being run locally via serverless offline

* Utility function for checking if functions are being run locally via serverless offline

TapaiBalazs · 2023-01-02T11:15:16Z

starters/serverless-framework-sqs-dynamodb/src/handlers/technology_create.test.ts

+
+		afterAll(async () => {
+			ddbMock.restore();
+			await removeFromCache(getCacheKey('87af19b1-aa0d-4178-a30c-2fa8cd1f2cff'));


Where do you set/hardcode this cache key when setting the item into the cache? 🤔

I also think that this uuid string should be exported as a MOCK_TECHNOLOGY_ID constant and use across tests.

These were just bad copy/pastes from other tests I wrote. They aren't actually needed.

TapaiBalazs · 2023-01-02T11:22:09Z

starters/serverless-framework-sqs-dynamodb/src/handlers/technology_update.ts

+			return responseHelper(StatusCodes.NOT_FOUND, null);
+		}
+
+		return responseHelper(StatusCodes.OK, technology);


Should this method invalidate the cache entry for the item's id? 🤔

If the GET technology/:id enpoint is cached for 5 minutes, and then an update occurs, the previous value gets returned back in the GET endpoint. I guess same goes for the DELETE and the CREATE methods as well, namely the caching of the all-technologies endpoint should also be invalidated if that is cached. 🤔

I handle it a layer deeper in the models/technology/update. it is doing the invalidation but the invalidation is handled by the model class that determines whether a value is from cache or database instead of the handler layer. Separation of concerns was my reasoning.

dustinsgoodman added 22 commits December 22, 2022 17:29

init commit

567ff6a

- initialize package.json - add .gitignore with v1 values - add .nvmrc and set to node16 - add init README with some scripts - add start, build, and infrastructure scripts - install deps for establishing and running the project initially

add docker-compose.yml with dynamodb and elasticmq config

f0fed2e

add tsconfig

fd662b2

add .editorconfig

916f815

add serverless configuration

0f2bd12

add initial healthcheck

734e6bf

add project linting

92fea91

add jest and related config

b94767c

add comment to serverless config about profiles

873a7e6

add isOffline util

154e8b1

add dynamodb client

53bfa47

update dynamodb setup

c5cec69

remove serverless-dynamodb-local

0ff732d

update eslint config

055cb96

add path aliases

2858ec5

setup .env

c204411

fix path aliases

a071b26

update dynamodb/getClient to use correct client

74c90c8

add technology type

0fc18c6

add technologies seed file

7c32890

readd serverless-dynamodb-local and fix config

e1b9c89

update README

9dac4e5

dustinsgoodman added WIP Work in Progress DO NOT MERGE labels Dec 23, 2022

dustinsgoodman added 4 commits December 24, 2022 15:55

update jest ignore paths to avoid build

6f617ab

add dynamodb and uuid packages

047280f

fix module reloading for lamdba

7a20a73

enable typescript strict mod and new models alias

ebbe95c

dustinsgoodman added 14 commits December 31, 2022 21:11

fix lint and bundle optimizations

8e2467c

fix install of ioredis-mock

97bddbc

add deploy script

ee9398d

update readme

2a6151a

add swagger and configure endpoint docs

2efcd59

add queue examples

2c7ec1e

- add job generator - add job processor - add dynamodb stream config and handlers - add sqs utils

update linting rules

858ca82

fix caching bug in update

7144919

fix autoswagger config

cfee852

See completecoding/serverless-auto-swagger#16

lock deps

f507499

update healthcheck endpoint

3d3d021

fix permissions

8a090f5

update readme

f6f373d

add new kit to website and CLI

f6ea679

dustinsgoodman removed WIP Work in Progress DO NOT MERGE labels Jan 1, 2023

dustinsgoodman marked this pull request as ready for review January 1, 2023 20:41

dustinsgoodman requested review from TapaiBalazs and morgnism January 1, 2023 20:41

mbicknese previously approved these changes Jan 2, 2023

View reviewed changes

TapaiBalazs reviewed Jan 2, 2023

View reviewed changes

dustinsgoodman added 3 commits January 15, 2023 13:50

update docs to use npm by default

c54d859

update processors to optimize record handler instantiation

b33cd7b

address PR feedback

5012a6a

dustinsgoodman dismissed mbicknese’s stale review via 5012a6a January 15, 2023 21:29

add notes about aws

c960a81

dustinsgoodman requested review from TapaiBalazs and mbicknese January 15, 2023 21:33

dustinsgoodman added 2 commits January 17, 2023 10:09

fix docker refernces in readme

d3c47ba

fix analyzer command

fea8909

		@@ -0,0 +1 @@
		export const DEFAULT_CACHE_TIME = 300_000; // 5 minutes = 1000 ms/s * 60 s/min * 5 min


		### General Commands

		- `build` bundles the project using the serverless packaging serverless. The produced artifacts will ship bundles shipped to AWS on deployment. You can optionally pass `--analyze <function name>` to run the bundle analyzer and visualize the results to understand your handler bundles.


		This README uses `yarn` for commands. If you're using `npm` or `pnpm`, utilize the equivalent version of the commands.

		1. Create a `.env` file:


		### Infrastructure Commands

		- `infrastructure:up` creates docker container and related images and runs them in the background. This should only be needed once during initial setup.

		@@ -0,0 +1,7 @@
		/**
		* Utility function for checking where functions are being run locally via serverless offline

	* Utility function for checking where functions are being run locally via serverless offline
	* Utility function for checking if functions are being run locally via serverless offline

add new serverless-framework-sqs-dynamodb kit #709

Are you sure you want to change the base?

add new serverless-framework-sqs-dynamodb kit #709

Conversation

dustinsgoodman commented Dec 23, 2022 • edited Loading

Type of change

Summary of change

Checklist

netlify bot commented Dec 23, 2022 • edited Loading

✅ Deploy Preview for starter-dev canceled.

aws-amplify-us-east-1 bot commented Dec 23, 2022

mbicknese left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TapaiBalazs Jan 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TapaiBalazs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TapaiBalazs Jan 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dustinsgoodman commented Dec 23, 2022 •

edited

Loading

netlify bot commented Dec 23, 2022 •

edited

Loading

TapaiBalazs Jan 13, 2023 •

edited

Loading

TapaiBalazs Jan 16, 2023 •

edited

Loading