Reading Partial Zip content

[TJ-Edwards]

Hello,

I am having an issue with parsing a partial zip request. My scenario is that I need to be able to read images out of the archive, but due to the size I want to retrieve the content in chunks. However, I believe I am missing something. Whenever I try to process my blob, I get the following error.

//blob
size: 1124376968
type: "application/zip"

//Error
ZipExtractor.ts:23 ERROR Error: Uncaught (in promise): File format is not recognized.

Currently I am just retrieving a random byte range based on the archive size. This is a very temporary approach to see if partial reads could be accomplished. I realize I might be cutting files off because I dont have the central directory. I plan on getting that from an external source and not the library itself. Does anyone see this as being a potential problem?

For a bit more context I am using the ZipExtractor class provided in the repo for parsing the blob. I did some messing around to force the blob type to see if that would help, but that did not work out.

//ZipExtractor.ts
export interface ZipContent {
[filePath: string]: {
name: string
content: string
}
}

export class ZipExtractor {
public extract(zipData: Blob): Promise {
zipData = zipData.slice(0, zipData.size, "application/zip");
let zipContent: ZipContent = {};
return new Promise((resolve, reject) => {
zip.createReader(new zip.BlobReader(zipData), (zipReader) => {
let beginDate: Date = new Date();
console.info(zip reader opened at ${beginDate}, zipData);

            zipReader.getEntries(async (zipEntries: zip.Entry[]) => {
                console.info(`${zipEntries.length} entries read`, zipData);

                try {
                    let i = 0;
                    for (let zipEntry of zipEntries) {
                        if (zipEntry.filename.includes(".jpg")) {
                            let entryContent: string = await this.readEntryContent(zipEntry);
                            zipContent[zipEntry.filename] = {
                                name: zipEntry.filename,
                                content: entryContent
                            };
                            console.info(`#${++i} => ${zipEntry.filename}`);
                        }
                    }

                    zipReader.close(() => {
                        let endDate: Date = new Date();
                        console.info(`zip read ended at ${endDate} - ${endDate.getTime() - beginDate.getTime()}ms`);
                        resolve(zipContent);
                    });

                } catch (e) {
                    zipReader.close(() => {
                        reject(e);
                    });
                }
            });
        }, reject);
    });
}

private readEntryContent(entry: zip.Entry): Promise<string> {
    return new Promise<string>((resolve, reject) => {
        entry.getData(
            new zip.BlobWriter('text/plain'),
            (resultText) => {
                resolve(URL.createObjectURL(resultText));
            },
            (current: number, total: number) => {
                console.debug(`${entry.filename}: ${current}/${total}B`);
            });
    });
}

}

Any guidance here would be greatly appreciated! Thank you for your time.

[gildas-lormeau]

You're using the legacy version of zip.js (i.e. < 2.x) which has been abandoned more than 4 years ago. You have to update zip.js to the last version.

[TJ-Edwards]

I figured out what my issue was regarding the version. I had old types installed. I am facing a new issue though. Whenever I try to get entries on a partial zip download I get the following error: "Uncaught (in promise): Error: End of central directory not found" (line below).

const entries = await reader.getEntries();

Basically I am creating a image viewer for large archives so the user doesnt have to download the entire zip to view its content. So I figured I would ask for a gig of content at a time and process the images out of that batch. Does your library support that and if so do you have any samples I could reference?

Thanks for your time.

[gildas-lormeau]

zip.js does not support reading entries without reading first the central directory located at the end of the zip file. This is deliberate, because it's the only way to reliably read a zip file. However, you could maybe use a instance of HttpRangeReader in order to read a zip file from the network without downloading it entirely.

[TJ-Edwards]

Is it possible to set up a HTTP Request with a range header using that class? I have a test that seems to be making a ton of HTTP Request for the file. Below is my sample. Ideally I would like to pass a Byte Range and ask for 500mb at a time, so I can download and process a file in chunks. If it is possible can you reply with a sample? What is the purpose of chunkSize in the configuration below?

loadZip(source){
        zip.configure({ chunkSize: 256, useWebWorkers: true });
        const zipReader = new zip.ZipReader(new zip.HttpRangeReader(this.source, { useXHR: false }));
        const entries = await zipReader.getEntries();
        await this.parseZip(entries);
}

public async parseZip(entries) {
        let zipContent: ZipContent = {};
        if (entries.length) {
            let i = 0;
            for (let zipEntry of entries) {
                if (zipEntry.filename.includes(".jpg") && i <= 5) {
                    const dataBlobWriter = new zip.BlobWriter();
                    zipEntry.getData(dataBlobWriter, (resultText) => {
                        console.log(URL.createObjectURL(resultText));
                    });
                    console.info(`#${++i} => ${zipEntry.filename}`);
                }
            }
        }
    }

[gildas-lormeau]

Remove zip.configure({ chunkSize: 256, useWebWorkers: true }); or set chunkSize to a higher value like 512 * 1024 * 1024. FYI, it's documented here: https://gildas-lormeau.github.io/zip.js/api/interfaces/Configuration.html#chunkSize. The API documentation is generated from this definition file that you should use if you're writing TS code.

[gildas-lormeau]

BTW, the call to zipEntry.getData is incorrect, it does not accept a callback function as 2nd parameter anymore since the version 2.x but returns a Promise, see https://gildas-lormeau.github.io/zip.js/api/interfaces/Entry.html#getData.getData-1.

[TJ-Edwards]

@gildas-lormeau Do you know of a way to get a callback or promise on painting an image to the UI? We have some timers involved with the slideshow and at some points things get out of sync. The idea would be to initiate our interval after the image is painted to ensure nothing is missed.

[gildas-lormeau]

zip.js is in no way linked to graphical events. Could you tell me how I can help you?

[TJ-Edwards]

@gildas-lormeau Do you have a way to easily cancel HTTP requests from the HttpRangeReader?

[gildas-lormeau]

@TJ-Edwards You should be able to set the signaloption to an AbortSignal instance obtained with an AbortController instance. It allows you to cancel the request via the abort() method.

You can also set the signaloption when calling Entry#getData(). This will cancel the decompression process.

[TJ-Edwards]

@gildas-lormeau Yea I am trying to with the following but not having any luck yet. Sample below.

public abortPromise = new AbortController();

cancel(){
this.abortPromise.abort('Navigation');
}

    await Promise.all(batch.map(async (entry, i) => {
        const abortListener = ({ target }) => {
            if (this.zipReader) {
                this.zipReader.close();
            }
            console.log(target);
            this.abortPromise.signal.removeEventListener('abort', abortListener);
            Promise.reject(target.reason);
        }
        this.abortPromise.signal.addEventListener('abort', abortListener);

...

[gildas-lormeau]

I would need to see how you call the HTTPRangeReader() constructor, and the getData() method (or the ZipWriter() constructor).

FYI, you can find an example here: https://github.com/gildas-lormeau/zip.js/blob/70fca50a361a9c5473b036280ddac8d44f07f4e2/tests/all/test-abort-signal-read.js. In this example controller.abort() is called as soon as the decompression/download is started. However,it could be called at any moment later (or sooner).

See also the code of this demo here https://github.com/gildas-lormeau/zip.js/blob/gh-pages/demos/demo-read-file.js for a complete example.