Building an efficient zipfile explorer on Cloudflare

[CodeFromAnywhere]

I tried this in a cloudflare worker to stream all files of a zip into r2, but can't get it to respond, it keeps hanging at the end

See reproducible repo:

https://github.com/CodeFromAnywhere/cloudflare-zip

I've tried many things to make the hang go away but didn't succeed. Just leaving it here because maybe anyone has encountered this problem before?

[gildas-lormeau]

I tested your code but I was not able to reproduce the issue. Here is what I did.

Run the following commands in the terminal:

1. git clone https://github.com/CodeFromAnywhere/cloudflare-zip
2. cd cloudflare-zip
3. npm install
4. npx wrangler dev

Comment this part https://github.com/CodeFromAnywhere/cloudflare-zip/blob/21ba4e08fece4dbd764fae2fec77a2c8d9db6cf5/main.ts#L42-L58.

Press band go to http://localhost:8787/?url=https://github.com/facebook/react/archive/refs/heads/main.zip.

On my machine, the message Processing complete is always displayed when refreshing the page with the cache disabled. I also added await new Promise(resolve => setTimeout(resolve, 10)); in the main loop but I got the same result. I don't see any particular issue.

[gildas-lormeau]

It's probably due to the Worker limits (cf. CPU time) on Cloudflare.

[CodeFromAnywhere]

Maybe this depends on the size of the zip. Try http://localhost:8787/?url=https://github.com/oven-sh/bun/archive/refs/heads/main.zip that one got stuck for me for sure, the facebook/react sometimes only gave errors.

I think it depending on the worker limits could be strange because it keeps hanging at the end, it doesn't give any error. However maybe putting this into a deployment would shed more light on it.

I don't have time to get back into it now though, it's not my priority. I think it'd be very useful though to use a serverless environment to unpack and store particular files from zips.

[gildas-lormeau]

I confirm I can reproduce the issue with this zip file but it does not really seem related to zip.js though.

Maybe the Worker memory (max. 128MB) is the cause the problem in that case. I tried to debug it in VSCode (see https://miniflare.dev/developing/debugger) but the debugging bugged...

[CodeFromAnywhere]

Could be the memory indeed, and I know 128 mb is quite the restriction, but how would it run out of memory? I'm using zipReader.getEntriesGenerator so I thought this would be pretty memory efficient, as it streams the zip, rather than loading it all in memory. Or is there a way to make it even more memory efficient?

[gildas-lormeau]

I don't know what could cause this issue but I can reproduce it with the code below and the last example of URL you mentioned. As you can see, it does not depend on zip.js at all. Maybe you should contact the support at Cloudflare because it looks like it's a bug.

export default {
  async fetch(request) {
    const url = new URL(request.url);
    const zipUrl = url.searchParams.get("url");
    try {
      const response = await fetch(zipUrl);
      const blob = await response.blob();
      return new Response("Processing complete", { status: 200 });
    } catch (error) {
      return new Response(`Error: ${error.message}`, { status: 500 });
    }
  }
};

[CodeFromAnywhere]

Thanks for that!

I changed the topic of this discussion and would like to share my plan of building a efficient zip file explorer on cloudflare: https://github.com/CodeFromAnywhere/cloudflare-zip

Do you think this seems feasible? I will give this a go soon

Basically what I'm trying to do is use the range header so we can only get the central directory first, then create a filter on that and do a follow up range request so we only get the files we want.

Would this be possible with zip.js? Is it true (what Claude suggests) that zip.js only does a range request when you do entry.getData with a HttpRangeRequest? How would it work if I would use entry.getData multiple times? Would the requests be somehow batched? Is it possible to specify to only do a single range, rather than multiple ranges in a single header?

I am planning on trying to build this out soon

[gildas-lormeau]

Normally, if you want to use requests with the Range header, you only need to use HttpRangeReader. The central directory will be read in its entirety with a range request, but the data of the entries included in the zip file will be read chunk by chunk. Also, zip.js implements a pool of worker so reading entries in parallel should not be a problem regarding the RAM usage.

[gildas-lormeau]

BTW, this test is very similar to what you want to achieve: https://github.com/gildas-lormeau/zip.js/blob/master/tests/all/test-sw-worker.js