Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Removing newlines in FASTA file increases ZSTD compression ratio by 10x

log.bede.im

Post reply on HN
Print view View original

21–30 of 118 posts

Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x

#21

Post by semiinfinitely »

semiinfinitely wrote: post #18

FASTA is a candidate for the stupidest file format ever invented and a testament to the massive gap in perceived vs actual programming ability of the average bioinformatician.

other file formats that rival fasta in stupidity include fastq pdb bed sam cram vcf. further reading [1]

> "intentionally or not, bioinformatics found a way to survive: obfuscation. By making the tools unusable, by inventing file format after file format, by seeking out the most brittle techniques"

1. https://madhadron.com/science/farewell_to_bioinformatics.htm...

Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x

#22

Post by Fraterkes »

semiinfinitely wrote: post #18

FASTA is a candidate for the stupidest file format ever invented and a testament to the massive gap in perceived vs actual programming ability of the average bioinformatician.

I’ll do you the immense favor of taking the bait. What’s so bad about it?

Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x

#23

Post by pajko »

Aachen wrote: post #17

I've also noticed this. Zstandard doesn't see very common patterns For me it was an increasing number (think of unix timestamps in a data logger that stores one entry per second, so you are just counting up until there's a gap in your data), in the article it's a fixed value every 60 bytes Of course, our brains are exceedingly good at finding patterns (to the point where we often find phantom ones). I was just expect…

Bzip2 performs exactly better because it rearranges the input to achieve better pattern matches: https://en.m.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_tran...

Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x

#24

Post by pbronez »

lifthrasiir wrote: post #15
mfld wrote: post #11

Using larger-than-default window sizes has the drawback of requiring that the same --long=xx argument be passed during decompression reducing compatibility somewhat. Interesting. Any idea why this can't be stored in the metadata of the compressed file?

It is stored in the metadata [1], but anything larger than 8 MiB is not guaranteed to be supported. So there has to be an out-of-band agreement between compressor and decompressor. [1] https://datatracker.ietf.org/doc/html/rfc8878#name-window-de...

Seems useful for games marketplaces like Steam and Xbox. You control the CDN and client, so you can use tricky but effective compression settings all day long.

Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x

#26

Post by AndrewOMartin »

jefftk wrote: post #19

The FASTA format looks like: > title bases with optional newlines > title bases with optional newlines ... The author is talking about removing the non-semantic optional newlines (hard wrapping), not all the newlines in the file. It makes a lot of sense that this would work: bacteria have many subsequences in common, but if you insert non-semantic newlines at effectively random offsets then compression tools will not…

The compression ratio will likely skyrocket if you sorted the list of bases.

Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x

#27

Post by bede »

jefftk wrote: post #19

The FASTA format looks like: > title bases with optional newlines > title bases with optional newlines ... The author is talking about removing the non-semantic optional newlines (hard wrapping), not all the newlines in the file. It makes a lot of sense that this would work: bacteria have many subsequences in common, but if you insert non-semantic newlines at effectively random offsets then compression tools will not…

Thank you for clarifying this – yes the non-semantic nature of these particular line breaks is a key detail I omitted.

Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x

#28

Post by jltsiren »

ashvardanian wrote: post #16

Nice observation! Took me a while to realize that Grace Blackwell refers to a person and not an Nvidia chip :) I’ve worked with large genomic datasets on my own dime, and the default formats show their limits quickly. With FASTA, the first step for me is usually conversion: unzip headers from sequences, store them in Arrow-like tapes for CPU/GPU processing, and persist as Parquet when needed. It’s straightforward, bu…

Basic text formats persist, because everyone supports them. Many tools have better file formats for internal purposes, but they are rarely flexible enough and robust enough for wider use. There are occasional proposals for better general purpose formats, but the people proposing them rarely agree which of the competing proposals should be adopted. And even if they manage to agree, they probably don't have the time and the money to make it actually happen.

Re: Removing newlines in FASTA file increases ZSTD compression ratio by 10x

#30

Post by shellfishgene »

AndrewOMartin wrote: post #26
jefftk wrote: post #19

The FASTA format looks like: > title bases with optional newlines > title bases with optional newlines ... The author is talking about removing the non-semantic optional newlines (hard wrapping), not all the newlines in the file. It makes a lot of sense that this would work: bacteria have many subsequences in common, but if you insert non-semantic newlines at effectively random offsets then compression tools will not…

The compression ratio will likely skyrocket if you sorted the list of bases.

You're joking, but a few bioinformatics tools use the Burrows-Wheeler transform to save memory, which is a bit like sorting the bases.
Post reply on HN