Using larger-than-default window sizes has the drawback of requiring that the same --long=xx argument be passed during decompression reducing compatibility somewhat.
Removing newlines in FASTA file increases ZSTD compression ratio by 10x
11–20 of 118 posts
Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x
#12Now I'm wondering why this works. DNA clearly has some interesting redundancy strategies. (it might also depend on genome?)
This is a dataset of bacterial DNA. Any two related bacteria will have long strings of the same letters. But it won't be neatly aligned, so the line breaks will mess up pattern matching.
Exactly. The line breaks break the runs of otherwise identical bits in identical sequences. Unless two identical subsequences are exactly in phase with respect to their line breaks, the hashes used for long range matching are different for otherwise identical subsequences.
Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x
#13Using larger-than-default window sizes has the drawback of requiring that the same --long=xx argument be passed during decompression reducing compatibility somewhat. Interesting. Any idea why this can't be stored in the metadata of the compressed file?
It uses more memory (up to +2gb) during decompression as well -> potential DoS.
Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x
#14Using larger-than-default window sizes has the drawback of requiring that the same --long=xx argument be passed during decompression reducing compatibility somewhat. Interesting. Any idea why this can't be stored in the metadata of the compressed file?
It uses more memory (up to +2gb) during decompression as well -> potential DoS.
Sending a .zip filled with all zeroes, so it compresses extremely well, is a well-known DoS historically (zip bomb, making the server run out of space in trying to read the archive)
You always need resource limits when dealing with untrusted data. RAM is one of the obvious ones. They could introduce a memory limit parameter; require passing --long with a value equal to or greater than what the stream requires to successfully decompress; require seeking support for the input stream so they can look back that way (TMTO); fall back to using temp files; or interactively prompt the user if there's a terminal attached. Lots of options, each with pros and cons of course, that would all allow a scenario where the required information for the decoder is stored in the compressed data file
Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x
#15Using larger-than-default window sizes has the drawback of requiring that the same --long=xx argument be passed during decompression reducing compatibility somewhat. Interesting. Any idea why this can't be stored in the metadata of the compressed file?
It is stored in the metadata [1], but anything larger than 8 MiB is not guaranteed to be supported. So there has to be an out-of-band agreement between compressor and decompressor.
[1] https://datatracker.ietf.org/doc/html/rfc8878#name-window-de...
Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x
#16Nice observation!
Took me a while to realize that Grace Blackwell refers to a person and not an Nvidia chip :)
I’ve worked with large genomic datasets on my own dime, and the default formats show their limits quickly. With FASTA, the first step for me is usually conversion: unzip headers from sequences, store them in Arrow-like tapes for CPU/GPU processing, and persist as Parquet when needed. It’s straightforward, but surprisingly underused in bioinformatics — most pipelines stick to plain text even when modern data tooling would make things much easier :(
Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x
#17I've also noticed this. Zstandard doesn't see very common patterns
For me it was an increasing number (think of unix timestamps in a data logger that stores one entry per second, so you are just counting up until there's a gap in your data), in the article it's a fixed value every 60 bytes
Of course, our brains are exceedingly good at finding patterns (to the point where we often find phantom ones). I was just expecting some basic checks like "does it make sense to store the difference instead of the absolute value for some of these bytes here". Seeing as the difference is 0 between every 60th byte in the submitted article, that should fix both our issues
Bzip2 performed much better for me but it's also incredibly slow. If it were only the compressor, that might be fine for many applications, but also decompressing is an exercise in patience so I've moved to Zstandard at the standard thing to use
Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x
#18FASTA is a candidate for the stupidest file format ever invented and a testament to the massive gap in perceived vs actual programming ability of the average bioinformatician.
Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x
#19The FASTA format looks like:
> title
bases with optional newlines
> title
bases with optional newlines
...
It makes a lot of sense that this would work: bacteria have many subsequences in common, but if you insert non-semantic newlines at effectively random offsets then compression tools will not be able to use the repetition effectively.
Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x
#20Damn surely you stop using ASCII formats before your dataset gets to 2 TB??