TensorFlow Datasets • Hacker News Board

Re: TensorFlow Datasets

#11

Post by throwaway20222 » Wed, Dec 21, 2022, 7:30 PM UTC

yeldarb wrote: post #7

For computer vision, there are 100k+ open source classification, object detection, and segmentation datasets available on Roboflow Universe: https://universe.roboflow.com

So many of those have tiny datasets - like 30 images that are seemingly of low quality. I love roboflow, but those are really hard to work with. I wish there was an open platform for generating the datasets that was cost effective.

Re: TensorFlow Datasets

#12

Post by jamesblonde » Wed, Dec 21, 2022, 8:08 PM UTC

soraki_soladead wrote: post #10

dbish wrote: post #8

Earlier quoted context omitted.

HF datasets have a much better UX for this

UX preferences vary. Imo, hf is too verbose and their pages try to cram in too much information with poor information hierarchy. For example: https://huggingface.co/datasets/glue https://www.tensorflow.org/datasets/catalog/glue

In this case, I don't see UIs, i see HF have a 1GB+ dataset that is curated. TF, in contrast, has 10 glue datasets varying from KBs! to 100s of MBs in size.

For Glue, HF wins, hands-down, IMO.

Re: TensorFlow Datasets

#13

Post by albertzeyer » Wed, Dec 21, 2022, 8:24 PM UTC

soraki_soladead wrote: post #4

jamesblonde wrote: post #3

I actually prefer Hugging Face Datasets - there's 16k+ of them today: https://huggingface.co/datasets

Quantity of datasets doesn’t seem like the right metric. The library just needs the datasets you care about and both libraries have the popular ones. What’s more important is integration and if you’re training custom TF models then tfds will generally integrate more smoothly than huggingface.

I tried Librispeech, a very common dataset for speech recognition, in both HF and TFDS.

TFDS performed extremely bad.

First it failed because the official hosting server only allows 5 simultaneous connections, and TFDS totally ignored that and makes up to 50 simultaneous downloads and that breaks. I wonder if anyone actually tested this?

Then you need to have some computer with 30GB to do the preparation, which might fail on your computer. This is where I stopped. https://github.com/tensorflow/datasets/issues/3887. It might be fixed now but it took them 8 months to respond to my issue.

On HF, it just worked. There was a smaller issue in how the dataset was split up but that is fixed now, and their response was very fast and great.

Re: TensorFlow Datasets

#14

Post by soraki_soladead » Wed, Dec 21, 2022, 8:44 PM UTC

jamesblonde wrote: post #12

soraki_soladead wrote: post #10

Earlier quoted context omitted.

UX preferences vary. Imo, hf is too verbose and their pages try to cram in too much information with poor information hierarchy. For example: https://huggingface.co/datasets/glue https://www.tensorflow.org/datasets/catalog/glue

In this case, I don't see UIs, i see HF have a 1GB+ dataset that is curated. TF, in contrast, has 10 glue datasets varying from KBs! to 100s of MBs in size. For Glue, HF wins, hands-down, IMO.

To each their own. I like that TF separates them since they are separate tasks and combining them is only one use case. At the end of the day we should just use what works best. The ML landscape is far from settled.