gzipped CSV files

[psaffrey-biomodal]

Hi there,

Does anybody have a recipe or pattern for working efficiently with gzipped CSV files? I'm working in bioinformatics, where these are extremely common! I could unzip with a JS library like pako but then I would probably have to do unzip and then parse (two steps), and I'd rather do this in one go, perhaps in a stream, for speed.

Cheers,

Peter

[jheer]

Hi! Arquero does not currently include support for gzipped CSV files. So unfortunately you will need to decompress using a separate library, then pass the resulting CSV text to Arquero. I don't have much experience with JS gzip decoders, so I don't have much more to add.

[jheer]

Potentially of interest -- it looks like gzip decompression is now available as a native feature both in Node and in the browser. For example, see https://developer.mozilla.org/en-US/docs/Web/API/DecompressionStream/DecompressionStream

[psaffrey-biomodal]

Yes, I have tried using these DecompressionStreams with other CSV libraries, but to make proper use of it this would need Arquero to accept a stream as an input to the CSV parsing routines, like fromCSV. Is this possible in Arquero? I've found the documentation and example for streams are a bit patchy, but it is a powerful feature if you're working with large files.