Berkeley Deep Drive Dataset • Hacker News Board

Re: Berkeley Deep Drive Dataset

#51

Post by Isamu » Wed, Jun 27, 2018, 6:22 PM UTC

sytelus wrote: post #39

adpirz wrote: post #2

In case you don't want to register and are curious about some metadata: Videos: 100K video clips: Size:1.8TB Info: The GPS/IMU information recorded along with the videos: Size: 3.9GB Images It has two subfolders. 1) 100K labeled key frame images extracted from the videos at 10th second 2) 10K key frames for full-frame semantic segmentation.: Size: 6.5GB Labels: Annotations of road objects, lanes, and drivable areas i…

While this is largest public annotated dataset, it's still only from 2 or 3 cities if I'm reading this right.

4 regions - see page 7 of the paper. New York, Berkeley, San Francisco, and "Bay Area"

https://arxiv.org/pdf/1805.04687.pdf

Re: Berkeley Deep Drive Dataset

#52

Post by derefr » Thu, Jun 28, 2018, 2:10 AM UTC

ori_b wrote: post #37

derefr wrote: post #20

Earlier quoted context omitted.

It's a really big download (1.8TB). One where both you and they would be really rather perturbed if the download failed at 90%. In fact, they'd probably be perturbed by the bandwidth costs even if everyone who wanted the dataset was only downloading it once. Maybe it uses WebTorrent? (Not sure why it couldn't just fall back to giving you a .torrent file in that case, though.)

Thankfully, Firefox has the ability to resume broken downloads. As does nearly every other method of downloading, other than (apparently) Chrome.

Broken downloads, yes. Corrupted downloads, no. Given that files served from CDNs are still usually served without HTTPS, there aren't many checksums between the two ends of the pipe to protect it from on-the-wire corruption. Doesn't matter much for video streaming ala Netflix; matters a lot for a structured dataset.

BitTorrent and related protocols handle this automatically by breaking the file into large (megabyte-range) chunks, and then putting the cryptographic hashes of all the chunks in the manifest. As long as you've received the manifest, you can protect against both passive corruption and active MITMing in the same way you resume broken downloads: by just discarding chunks that failed to complete to a state of "has all the bytes and hashes correctly", and trying those chunks again.

(Sadly, HTTP doesn't support a digest response header that applies to each chunk of a "Transfer-Encoding: chunked" response stream, or it could vaguely compete with this. The Content-MD5 header could have done this, but it was removed precisely because implementations were in conflict on whether it was for this, or for hashing the document as a whole.)

Re: Berkeley Deep Drive Dataset

#53

Post by tomcooks » Thu, Jun 28, 2018, 2:04 PM UTC

quickben wrote: post #3

The website doesn't work in Firefox. "FAQ: The download buttons do not work". "The website is fully supported by Chrome now" The current state of the internet, has reached a very low point. One would expect more from berkley. edu . EDIT: I find the downvotes preposterous. Are we somehow supposed to expect requiring a proprietary browser to simply download a file from an educational institution now?

Works great on FF60.0.1

Re: Berkeley Deep Drive Dataset

#54

Post by arielbaz » Mon, Jul 09, 2018, 7:18 PM UTC

First, thanks for sharing this data. Second - why on earth would anyone create a 1.8TB zip file of 100k videos? Likely the video encoder already compressed every possible bit out of these videos, zip is not going to make it smaller. It is, however, going to make it mandatory for everyone to download the full 1.8TB file even to get a single video out of this archive. Makes me wonder what else is happening here (like the chrome only download link which is hosted on another domain, and the non-https login, and that escalator to nowhere..*)