file upload

[gedw99]

Hey

I would like to get file uploads working over http and nats. Here's what i think might work:

The Caddy config describes the Domain and Path.
A NATS upload will place the file under the path folder described in the caddy config.
A HTTP upload will do the same.

The NATS data plane will need to place the file in the NATS Object store, and then materialise the file onto the file system.
The HTTP data plane just places it onto the file system.

Then when I make a download request over NATS or HTTP, the system returns the file.

Large files:
I need to add chunking for that later.

nats example:
nats object add uploadsBucket
nats object put uploadsBucket ~/Uploads/file.txt

http example:
curl -u "demo" -d @file1.txt https://example.com/uploads

[skurfuerst]

Hey @gedw99 ,

I don't fully understand your use cases yet. Could you explain what goal you try to achieve and why you want to do this?

Thanks,
Sebastian

[gedw99]

https://docs.nats.io/using-nats/developer/develop_jetstream/object

This is cool because you can upload via nats and download via http ( caddy ).

It can do 100 gb files over nats Object Store. Highly resilient.

i have it all working without caddy.

All we need is for caddy to subscribe to the NATS object store.

[gedw99]

Hey @skurfuerst

I saw just now the new feature to offload payloads into nats obj store. It looks like a good start.

I wrote some code to chunk the payload in and out of nats Object Store a while ago. If you’re interested let me know and I can dig it up.

The file upload idea i had is to allow users to upload via nats caddy Bridge and then put it into minio /s3. It’s useful for blob stuff like images or binaries.

then later a user can ask for that image or binary via the nats caddy bridge.

Minio on Hetzner is cost effective and it can also do Tiering and offload data not used often to backblaze. So if a user requests a file that has been put into cold storage we can bring it back from backblaze and into minio using the nats caddy Bridge to handle the async aspects.

hope that sone of these ideas are useful.