111–120 of 196 posts
Earlier quoted context omitted.
I'd love to have something like this for archiving DVD ISOs though, where the VOBs are compressed with old-school MPEG.
Shouldn't you just use Handbrake and H.264/AAC? (Assuming your computer is fast/new enough to play it back.)
It's interesting to do a cost analysis here: It's saved "multiple petabytes" of space. Backblaze storage is $0.005/GB/Month = $5k/PB/Month. The GitHub repo has 7 authors, perhaps costing Dropbox $200k/year each and taking most of a year ~ $1M to develop this system. So this might pay for itself after 200PB*Months, assuming Dropbox's storage costs are the same as Backblaze's prices, and assuming CPU time is free. (TOD…
Dropbox is valued at $10b (regardless of what you think of that number, someone was willing to buy at that price). But they only have 1800 employees. If by some miracle 80% of them are engineers that is $7M per engineer. All this for a "pure software" business. No inventory, no manufacturing, they only recently started doing their own operations.
Investors would be telling them "more engineers" => "more software" => "better Dropbox". So now you have to go out and snap up engineers in Silicon Valley, which is very very hard. But you don't offer the perk of a Google/Apple line on a resume nor the compensation of e.g. Microsoft. What do you do?
Answer: you offer people the chance to work on cutting edge breakthrough technology. Not only that but it's all open source! This is a dream job for some people. They will turn down every other gig for this one.
It doesn't matter that PB/mo is chump change because an investor doesn't know that, they only know the engineering headcount. If by some miracle an investor does know this is a waste of time, then you just point to this HN post and observe how many developers are interested in this technology and how it is attracting developer mindshare that can be exploited for additional hires down the road.
I'm not saying it's rational–it's not. But I think there were strong incentives to greenlight a project like this, even if the actual cost savings were zero.
I want this working seamlessly on the file system. That means: I see JPGs but they are lepton compressed in reality. Would be a great use case for existing file servers. How could this be (theoretically) achieved on a linux machine?
Earlier quoted context omitted.
The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG. Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format. You wouldn't see the same leap for videos because people have been working hard t…
I'd love to have something like this for archiving DVD ISOs though, where the VOBs are compressed with old-school MPEG.
It's mostly to let you fast forward, but there is a technical issue there. MPEG2 decoders aren't all mathematically identical, so what happens is the picture tends to drift away from the real thing after a while, and there's hacks like frequent keyframes and flipping the smallest DCT coefficient to get around it…
Earlier quoted context omitted.
IMO, it shows how they don't plan to make any money from this, don't consider it a large competitive advantage, and think that the largest advantage of open sourcing it are the 'free' open source volunteers will improve the software. It's interesting that they don't consider it a large enough competitive advantage. Either that or they are using this to attract engineers.
You also open source if it's in your best interest for something to become the 'standard'. In google there is a regret that they didn't open source a lot of stuff, because it ends up being reproduced outside of google in some form. The open source companies have an advantage in hiring and overall advancement of their product by using the open source version whatever their internal version was. You also see it in face…
What regret? A lot of the open-source clones of Google's internal projects come from Google themselves.
This is amazing, we've been struggling with JPG storage and fast delivery at my lab (terabytes and petabytes of microscopy images). We'll be running tests and giving this a shot!
This is amazing, we've been struggling with JPG storage and fast delivery at my lab (terabytes and petabytes of microscopy images). We'll be running tests and giving this a shot!
This is very impressive for archiving images. For a quick test, I run it over ~1.3GB JPEG pictures I had locally, the finally result is 810M, that's 66% of the original size, very impressive considering it's lossless. It only deals with jpg file though, no png, no iso, no zip, no any formats other than JPG. If someone can do this over video files that will be PiedPiper comes into real life.
The reason this can be done with JPEGs is the compression hasn't been updated. There are folks that have used h264 for compressing images and webp uses vp8 compression, both with much better results than JPEG. Lepton is cool because it helps make existing technology a whole lot better, but what we actually need is a better image format. You wouldn't see the same leap for videos because people have been working hard t…
This is amazing, we've been struggling with JPG storage and fast delivery at my lab (terabytes and petabytes of microscopy images). We'll be running tests and giving this a shot!
jpg is a weird format to be storing microscopy images, no? Usually end up in some sort of bitmap TIFF (or their Zeiss/etc. proprietary format) from what I've seen.
Eg. just tested on a small 10MP Sony ARW raw image - the raw file is 7MB, the camera jpeg is 2.3MB, and a tiff compressed with LZW is 20MB (uncompressed 29MB). The raw tiff run through lzma is 9.3MB. But either way, if ~7MB is the likely lossless size, if the JPEG is good enough, at ~2.3MB it's a pretty big difference, if we're talking petabytes, not megabytes.
(I'll get around to testing lepton on the jpegs shortly)