Live data from Hacker News

A Desperate Plea for a Free Software Alternative to Aspera (2018)

ccdatalab.org

121–130 of 172 posts

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#121
post #102

Earlier quoted context omitted.

I solved it by cryptographically signing each packet. The signature was checked before decrypting the packet. Some customers (e.g. DOD) sent in clear, unsigned. They trusted their networks.

not true, although i'm sure there are one-off instances where there's no encryption. but in general there's lots of encryption on each of the different levels of secure government networks.

Right, they didn't want us encrypting what they were going to encrypt in their routers anyway. "Trusted" is a technical term.

So they started using Aspera to send video from the UAVs over Iraq etc. to the Pentagon. Before that, they were literally flying boxes of videotapes (or CDs? Tapes seem hard to believe, except this was govt.) to a warehouse, where analysts could check them out, like from your neighborhood library.

Probably now it's all realtime. But they have to stream it, somehow, so it is likely still going via Aspera to Nevada or wherever they work the remote pilots.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#122
post #112
post #58

Earlier quoted context omitted.

Zip is now effectively obsolete. Good ones lately are Zstd and Lz4, depending on your needs. Lz4 doesn't compress as much, but it is very, very fast. Zstd compresses better than Zip, and unpacks many times faster. It can re-use a custom dictionary saved off from a previous compression, so you can get quick startup time for later fragments.

Nitpicking a bit, but a format that is still used ~90% of the time is not obsolete. Not optimal (anymore) is what you meant, I think.

I mean there is no objective reason to use gzip (deflate) compression anymore, unless whoever has to use the data cannot themselves upgrade and is stuck with a gzip deflator.

It is little different from building with ARMs instead of Z80s. Where Z80s were once a good choice, now they are almost always a bad choice. Obsolete. USB sticks vs floppies, OLED vs fluorescent-backed LCD, valacyclovir vs acyclovir, Google vs Altavista, Git vs Subversion vs RCS. Need I go on?

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#123
post #102

Earlier quoted context omitted.

I solved it by cryptographically signing each packet. The signature was checked before decrypting the packet. Some customers (e.g. DOD) sent in clear, unsigned. They trusted their networks.

> Some customers (e.g. DOD) sent in clear, unsigned. They trusted their networks. Wow. I had no idea. This may be one of the dumbest things I've ever read. Encryption is free.

Bandwidth for signature values in each packet is not. Additionally, as is pointed out down-thread, crypto wasn’t always free either.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#124

Question at the meta-level: what software/platforms are producing, analyzing, modifying & consuming this data? Is it possible some legacy systems that produce and consume these massive files on-site would more sensibly run in the cloud, directly & selectively accessing the data chunks they need over fast backbone connections? Also, is there any room for an rsync type approach, sending compressed deltas rather than na…

Cloud vendors make it cheap to move data in, expensive to move it out, but also expensive (and curiously inconvenient) to keep there.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#125

Question at the meta-level: what software/platforms are producing, analyzing, modifying & consuming this data? Is it possible some legacy systems that produce and consume these massive files on-site would more sensibly run in the cloud, directly & selectively accessing the data chunks they need over fast backbone connections? Also, is there any room for an rsync type approach, sending compressed deltas rather than na…

The systems that produce massive DNA sequence files are machines reading the sequences of actual physical DNA molecules. The cloud can't sequence DNA, as far as I know.

Most DNA sequences are only slightly different from sequences you already have. There ought to be mature differs by now that would enable you to send just differences. If not, there is a huge opportunity for somebody.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#127
post #58

Earlier quoted context omitted.

You can compress your data beforehand using zip or another tool. Eg. Tools like https://en.wikipedia.org/wiki/Compression_of_Genomic_Sequenc...

Zip is now effectively obsolete. Good ones lately are Zstd and Lz4, depending on your needs. Lz4 doesn't compress as much, but it is very, very fast. Zstd compresses better than Zip, and unpacks many times faster. It can re-use a custom dictionary saved off from a previous compression, so you can get quick startup time for later fragments.

You're not wrong, but the opening statement "Zip is now effectively obsolete." is sort of true but not really in a lot of contexts. It isn't something I'd say with conviction.

From your point of view was the implied statement something akin to: "In the context of a high-throughput network transfer program choosing DEFLATE over an algorithm more well optimized for computational weight wouldn't be a good idea." ?

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#128
post #123

Earlier quoted context omitted.

> Some customers (e.g. DOD) sent in clear, unsigned. They trusted their networks. Wow. I had no idea. This may be one of the dumbest things I've ever read. Encryption is free.

Bandwidth for signature values in each packet is not. Additionally, as is pointed out down-thread, crypto wasn’t always free either.

Actually the signature was almost free, because it was also the nonce for the header, which was the nonce for the rest of the packet. IIRC.

Re: A Desperate Plea for a Free Software Alternative to Aspera (2018)

#130
post #44

Ex Aspera Dev here. I did the encryption and early parallel work. There is a lot of good science behind fasp. An advantage it has over IETF protocols is that both ends trust one another. Another advantage, until recently, was out-of-order delivery. The protocol totally ignores drops, for flow control. Instead, it measures change in transit time. The receiver knows, the sender needs to know, but the useful lifetime of…

Do you have any anecdotal experience with how this sort of approach affects very short, bursty stop-and-start transmissions (notwithstanding the fact that you obviously can't go faster than link latency)?
Post reply on HN