Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

381–390 of 475 posts

Re: Big data is dead (2023)

#381

Post by thesz »

datadeft wrote: post #204
thesz wrote: post #188

Earlier quoted context omitted.

Parquet is underdesigned. Some parts of it do not scale well. I believe that Parquet files have rather monolithic metadata at the end and it has 4G max size limit. 600 columns (it is realistic, believe me), and we are at slightly less than 7.2 millions row groups. Give each row group 8K rows and we are limited to 60 billion rows total. It is not much. The flatness of the file metadata require external data structures…

Nobody is forcing you to use a single Parquet file.

Of course.

But nobody tells me that I can hit a hard limit and then I need a second Parquet file and should have some code for that.

The situation looks to me as if my "Favorite DB server" supports, say, only 1.9 billions records per table and if I hit that limit I need a second instance of my "Favorite DB server" just for that unfortunate table. And it is not documented anywhere.

Re: Big data is dead (2023)

#382

Post by wenc »

riku_iki wrote: post #379
geysersam wrote: post #378

Earlier quoted context omitted.

The entire csv doesn't have to fit in memory, but the entire csv has to pass through memory at some point during the processing. The parquet file has metadata that allows duckdb to only read the parts that are actually used, reducing total amount of data read from disk/network.

> The parquet file has metadata that allows duckdb to only read the parts that are actually used, reducing total amount of data read from disk/network. this makes sense, and what I hoped to have. But in reality looks like parsing CSV string works faster than bloated and overengineered parquet format with libs.

>But in reality looks like parsing CSV string works faster than bloated and overengineered parquet format with libs.

Anecdotally having worked with large CSVs and large on-disk Parquet datasets, my experience is the opposite of yours. My DuckDB queries operate directly on Parquet on disk and never load the entire dataset, and is always much faster than the equivalent operation on CSV files.

I think your experience might be due to -- what it sounds like -- parsing the entire CSV into memory first (CREATE TABLE) and then processing after. That is not an apples-to-apples comparison because we usually don't do this with Parquet -- there's no CREATE TABLE step. At most there's a CREATE VIEW, which is lazy.

I've seen your comments bashing Parquet in DuckDB multiple times, and I think you might be doing something wrong.

Re: Big data is dead (2023)

#383

Post by kwillets »

dventimi wrote: post #37

Question for the Big Data folks: where do sampling and statistics fit into this, if at all? Unless you're summing to the penny, why would you ever need to aggregate a large volume of data (the population) rather than a small volume of data (a sample)? I'm not saying there isn't a reason. I just don't know what it is. Any thoughts from people who have genuine experience in this realm?

I've done it both ways. Look into Data Sketches also if you want to see applications.

The pros:

-- Samples are small and fast most of the time.

-- can be used opportunistically, eg in queries against the full dataset.

-- can run more complex queries that can't be pre-aggregated (but not always accurately).

The cons:

-- requires planning about what to sample and what types of queries you're answering. Sudden requirements changes are difficult.

-- data skew makes uniform sampling a bad choice.

-- requires ETL pipelines to do the sampling as new data comes in. That includes re-running large backfills if data or sampling changes.

-- requires explaining error to users

-- Data sketches can be particularly inflexible; they're usually good at one metric but can't adapt to new ones. Queries also have to be mapped into set operations.

These problems can be mitigated with proper management tools; I have built frameworks for this type of application before -- fixed dashboards with slow-changing requirements are relatively easy to handle.

Re: Big data is dead (2023)

#384

Post by metadat »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Can you get a single machine with more than 6TiB of memory these days?

That's quite a bit..

Re: Big data is dead (2023)

#385

Post by jjgreen »

nostrademons wrote: post #364
jjgreen wrote: post #288

Earlier quoted context omitted.

Please promote the use of .parquet files! apt-cache search parquet Maybe later

It's more like "sudo pip install pandas" and then Pandas comes with Parquet support.

Pandas cannot read parquet files itself, it uses 3rd party "engines" for that purpose and those are not available in Debian

Re: Big data is dead (2023)

#386

Post by data-ottawa »

nostrademons wrote: post #345
data-ottawa wrote: post #321

Earlier quoted context omitted.

Even if I'm doing a one off, depending on the task it can be easier/faster/more reliable to load 6TiB into a big query table than waiting hours for some task to complete and fiddling with parallelism and memory management. It's a couple hundred bucks a month and $36 to query the entire dataset, after partitioning thats not terrible.

A 6T hard drive and Pandas will cost you a couple hundred bucks, one time purchase, and then last you for years (and several other data analysis jobs). It also doesn't require that you be connected to the Internet, doesn't require that you trust 3rd-party services, and is often faster (even in execution time) than spooling up BigQuery. You can always save an intermediate data set partitioned and massaged into whateve…

I wouldn't make a purchase of either without knowing a bit more about the lifecycle and requirements.

If you only needed this once, the BQ approach requires very little setup and many places already have a billing account. If this is recurring then you need to figure out what the ownership plan of the hard drive is (what's it connected to, who updates this computer, what happens when it goes down, etc.).

Re: Big data is dead (2023)

#387

Post by thesz »

apwell23 wrote: post #208
thesz wrote: post #188

Earlier quoted context omitted.

Parquet is underdesigned. Some parts of it do not scale well. I believe that Parquet files have rather monolithic metadata at the end and it has 4G max size limit. 600 columns (it is realistic, believe me), and we are at slightly less than 7.2 millions row groups. Give each row group 8K rows and we are limited to 60 billion rows total. It is not much. The flatness of the file metadata require external data structures…

some critiques of parquet by andy pavlo https://www.vldb.org/pvldb/vol17/p148-zeng.pdf

Thanks, very insightful.

"Dictionary Encoding is effective across data types (even for floating-point values) because most real-world data have low NDV ratios. Future formats should continue to apply the technique aggressively, as in Parquet."

So this is not critique, but assessment. And Parquet has some interesting design decisions I did not know about.

So, let me thank you again. ;)

Re: Big data is dead (2023)

#388

Post by vegabook »

bartart wrote: post #309
vegabook wrote: post #6

my experience is that while data keeps growing at an exponential rate, its information content does not. In finance at least, you can easily get 100 million data points per series per day if you want everything, and you might be dealing with thousands of series. That sample rate, and the number of series, is usually 99.99% redundant, because the eigenvalues drop off almost to zero very quickly after about 10 dimensio…

That’s very interesting, so thank you — how do you handle if the eigenvectors change over time?

you can store the main eigenvectors for a set rolling period and see how the space evolves along them, all the while also storing the new ones. In effect the whole idea is to get away from "individual security space" and into "factor space", which is much smaller, and see how the factors are moving. Also, a lot of the time you just care about the outliers -- those (small numbers of) instruments or clusters of instruments that are trading in an unusual way -- then you either try to explain it.... or trade against it. Also keep in mind that lower-order factors tend to be much more stationary so there's a lot of alpha there -- if you can execute the trades efficiently (which is why most successful quant shops like citadel and jane street are market MAKERS, not takers, btw).

Re: Big data is dead (2023)

#389

Post by riku_iki »

wenc wrote: post #382
riku_iki wrote: post #379

Earlier quoted context omitted.

> The parquet file has metadata that allows duckdb to only read the parts that are actually used, reducing total amount of data read from disk/network. this makes sense, and what I hoped to have. But in reality looks like parsing CSV string works faster than bloated and overengineered parquet format with libs.

>But in reality looks like parsing CSV string works faster than bloated and overengineered parquet format with libs. Anecdotally having worked with large CSVs and large on-disk Parquet datasets, my experience is the opposite of yours. My DuckDB queries operate directly on Parquet on disk and never load the entire dataset, and is always much faster than the equivalent operation on CSV files. I think your experience mi…

> I think your experience might be due to -- what it sounds like -- parsing the entire CSV into memory first (CREATE TABLE) and then processing after. That is not an apples-to-apples

original discussion was about CSV vs parquet "reader" part, so this is exactly apple to apple testing, easy to benchmark and I stand my ground. What you are doing downstream, it is another question which is not possible to discuss because no code for your logic is available.

> I've seen your comments bashing Parquet in DuckDB multiple times, and I think you might be doing something wrong.

like running one command from DuckDB doc.

Also, I am not "bashing", I just state that CSV reader is faster.

Re: Big data is dead (2023)

#390

Post by nostrademons »

jjgreen wrote: post #385
nostrademons wrote: post #364

Earlier quoted context omitted.

It's more like "sudo pip install pandas" and then Pandas comes with Parquet support.

Pandas cannot read parquet files itself, it uses 3rd party "engines" for that purpose and those are not available in Debian

Ah yes, that's true though a typical Anaconda installation will have them automatically installed. "sudo pip install pyarrow" or "sudo pip install fastparquet" then.
Post reply on HN