Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Hyperparam: OSS tools for exploring datasets locally in the browser

hyperparam.app

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Show HN: Hyperparam: OSS tools for exploring datasets locally in the browser

#11

Post by platypii »

abeppu wrote: post #10

Though these tools might be interesting, I wish they had called this something else. This isn't at all related to the concept of hyperparameters which people commonly refer to as hyperparams. And in their copy, the only reference to hyperparameters seems to be misusing the term. > This stems from an industry-wide realization that model performance is ultimately bounded by data quality, not just model architecture or…

That's fair criticism... to be honest when I started the project it was more focused on hyperparameters, and it evolved into this javascript-for-ai mission. But now I just kind of liked the name.

Re: Show HN: Hyperparam: OSS tools for exploring datasets locally in the browser

#12

Post by abeppu »

platypii wrote: post #5
wbradmoore wrote: post #4

Why not WASM? Seems like something like duckdb-wasm or datafusion-wasm can do the same thing?

Duckdb and datafusion are super cool! But they are VERY large wasm blobs (30-40mb each). This is often larger than the data you’re trying to load. And they add complexity with serving and deploying wasm files. Hyparquet is 10kb of pure js, and so its trivial to deploy on a modern webapp, and wins hands down on time-to-first-data metric.

> Duckdb and datafusion are super cool! But they are VERY large wasm blobs (30-40mb each). This is often larger than the data you’re trying to load.

I don't know how to reconcile this with the emphasis in the page on interacting with datasets relevant to AI which are commonly several orders of magnitude larger than this. What's an AI problem where the data data involved has been less than 10s of mb? I think that only toy problems and datasets could plausibly be smaller (e.g. the training images for the classic MNIST dataset are 47MB, and the whole dataset is 55 https://www.kaggle.com/datasets/hojjatk/mnist-dataset?select... ).

Re: Show HN: Hyperparam: OSS tools for exploring datasets locally in the browser

#14

Post by platypii »

abeppu wrote: post #12
platypii wrote: post #5

Earlier quoted context omitted.

Duckdb and datafusion are super cool! But they are VERY large wasm blobs (30-40mb each). This is often larger than the data you’re trying to load. And they add complexity with serving and deploying wasm files. Hyparquet is 10kb of pure js, and so its trivial to deploy on a modern webapp, and wins hands down on time-to-first-data metric.

> Duckdb and datafusion are super cool! But they are VERY large wasm blobs (30-40mb each). This is often larger than the data you’re trying to load. I don't know how to reconcile this with the emphasis in the page on interacting with datasets relevant to AI which are commonly several orders of magnitude larger than this. What's an AI problem where the data data involved has been less than 10s of mb? I think that only…

Yea except with parquet you don't need to load the entire file, the parquet metadata let's you do http range requests for just the data you need.

For example this parquet is the entire english wikipedia (400mb) but loads less than 4mb including html and all js to display the first rows:

https://hyperparam.app/files?key=https%3A%2F%2Fs3.hyperparam...

This way you can have huge AI datasets in cloud storage, and still have a nice interface for looking at your data.

In particular, a lot of modern AI datasets are huge walls of text (web scrapes, chains of thought, or agentic conversation histories), and most datasets on huggingface are in parquet. So you can much more quickly look at your data this way versus say jupyter notebooks.

Here's the glaive reasoning dataset on the Hyperparam hugging face space:

https://huggingface.co/spaces/hyperparam/hyperparam?url=http...

Re: Show HN: Hyperparam: OSS tools for exploring datasets locally in the browser

#15

Post by platypii »

klntsky wrote: post #13

That's a lot of names for a bunch of tools that do a single task each. What I would really benefit of is a hypothetical LLM chat app that is focused on data migration or processing pipelines.

Funny you say that, because I built these tools because I wanted to build something very much like what you're describing!

I was trying to look at, filter, and transform large AI datasets, and I was frustrated with how bad the existing tool was for working with datasets with huge amounts of text (web scrapes, github dumps, reasoning tokens, agent chat logs). Jupyter notebook is woefully bad at helping you to look at your data.

So I wanted to build better browser tools for working with AI datasets. But to do that I had to build these tools (there was no working parquet implementation in JS when I started).

Anyway I'm still working on building an app for data processing using LLM chat assistant to help a single user curate entire datasets singlehandedly. But for now I'm releasing these components to the community as open source. And having them "do a single task each" was very much intentional. Thanks for the comment!

Re: Show HN: Hyperparam: OSS tools for exploring datasets locally in the browser

#17

Post by platypii »

cyrdax wrote: post #16

Anyone benchmark this vs. duckdb-wasm?

I don’t have benchmarks specifically against duckdb. I’m sure native C++ will run faster than JavaScript.

But whats important is that with Hyperparam you can do it in the browser, where the bottleneck will always be network-bound not cpu-bound.

Re: Show HN: Hyperparam: OSS tools for exploring datasets locally in the browser

#18

Post by xoofoog »

platypii wrote: post #14
abeppu wrote: post #12

Earlier quoted context omitted.

> Duckdb and datafusion are super cool! But they are VERY large wasm blobs (30-40mb each). This is often larger than the data you’re trying to load. I don't know how to reconcile this with the emphasis in the page on interacting with datasets relevant to AI which are commonly several orders of magnitude larger than this. What's an AI problem where the data data involved has been less than 10s of mb? I think that only…

Yea except with parquet you don't need to load the entire file, the parquet metadata let's you do http range requests for just the data you need. For example this parquet is the entire english wikipedia (400mb) but loads less than 4mb including html and all js to display the first rows: https://hyperparam.app/files?key=https%3A%2F%2Fs3.hyperparam... This way you can have huge AI datasets in cloud storage, and still h…

Wow - that's super clever. How do you get away with loading part of the file? Which part do you load?

Re: Show HN: Hyperparam: OSS tools for exploring datasets locally in the browser

#20

Post by chatmasta »

xoofoog wrote: post #18
platypii wrote: post #14

Earlier quoted context omitted.

Yea except with parquet you don't need to load the entire file, the parquet metadata let's you do http range requests for just the data you need. For example this parquet is the entire english wikipedia (400mb) but loads less than 4mb including html and all js to display the first rows: https://hyperparam.app/files?key=https%3A%2F%2Fs3.hyperparam... This way you can have huge AI datasets in cloud storage, and still h…

Wow - that's super clever. How do you get away with loading part of the file? Which part do you load?

I’m not OP but as this is a common pattern…

Parquet stores the metadata in the footer so first request is effectively a negative byte range (content length minus footer length). This metadata includes table statistics like “column ‘date_sold’ has minimum date 1-1-1970 and maximum date 12-31-2024,” and row group statistics like “the row group at byte offset X has minimum ‘date_sold’ value of 1-1-2023 and maximum ‘1-1-2024’.”

So if your query tool gets a SQL query with a predicate like “WHERE date_sold > ‘3-1-2024’ AND date_sold My colleague Artjoms (and co-founder of Splitgraph with me) gave a great presentation [0] on how we achieved this with DataFusion, including visualization of the pruning.

[0] https://youtube.com/watch?v=D_phetiS-4w

Post reply on HN