Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

301–310 of 475 posts

Re: Big data is dead (2023)

#301

Post by xnx »

riku_iki wrote: post #271
mrtimo wrote: post #168

Earlier quoted context omitted.

.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…

> They are 50x faster to read than CSV I actually benchmarked this and duckdb CSV reader is faster than parquet reader.

For how many rows?

Re: Big data is dead (2023)

#302

Post by bee_rider »

whamlastxmas wrote: post #234
kmarc wrote: post #209

Earlier quoted context omitted.

FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.

Is this like interviewing for a chef position for a fancy restaurant and when asked how to perfectly cook a steak, you preface it with “well you can either go to McDonald’s and get a burger, or…” It may not be reasonable to suggest that in a role that traditionally uses big data tools

I’m not sure if you are referencing it intentionally or not, but some chefs (Gordon Ramsey for one) will ask an interviewee to make some scrambled eggs; something not super niche or specialized but enough to see what their technique is.

It is a sort of “interview hack” example that’s been used to emphasize the idea of a simple unspecialized skill-test that went around a while ago. I guess upcoming chefs probably practice egg scrambling nowadays, ruining the value of the test. But maybe they could ask to make a bit of steak now.

Re: Big data is dead (2023)

#304

Post by [deleted] »

jandrewrogers wrote: post #295
the_real_cher wrote: post #91

Earlier quoted context omitted.

How would six terabytes fit into memory? It seems like it would get a lot of swap thrashing if you had multiple processes operating on disorganized data. I'm not really a data scientist and I've never worked on data that size so I'm probably wrong.

6 TB does not fit in memory. However, with a good storage engine and fast storage this easily fits within the parameters of workloads that have memory-like performance. The main caveat is that if you are letting the kernel swap that for you then you are going to have a bad day, it needs to be done in user space to get that performance which constrains your choices.

[deleted]

Re: Big data is dead (2023)

#305

Post by jandrewrogers »

gbin wrote: post #10

IMHO the main driver for big data was company founders egos. Of course your company will explode and will be a planet scale success!! We need to design for scale! This is really a tragic mistake while your product only needs one SQLite DB until you reach series C.... All the energy should be focused on the product, not its scale yet.

It depends on the kind of data you work with. Many kinds of important data models -- geospatial, sensing, telemetry, et al -- can hit petabyte volumes at "hello world".

Data models generated by intentional human action e.g. clicking a link, sending a message, buying something, etc are universally small. There is a limit on the number of humans and the number of intentional events they can generate per second regardless of data model.

Data models generated by machines, on the other hand, can be several orders of magnitude higher velocity and higher volume, and the data model size is unbounded. These are often some of the most interesting and under-utilized data models that exist because they can get at many facts about the world that are not obtainable from the intentional human data models.

Re: Big data is dead (2023)

#306

Post by riku_iki »

wenc wrote: post #298
riku_iki wrote: post #271

Earlier quoted context omitted.

> They are 50x faster to read than CSV I actually benchmarked this and duckdb CSV reader is faster than parquet reader.

I would love to see the benchmarks. That is not my experience, except in the rare case of a linear read (in which CSV is much easier to parse). CSV underperforms in almost every other domain, like joins, aggregations, filters. Parquet lets you do that lazily without reading the entire Parquet dataset into memory.

> That is not my experience, except in the rare case of a linear read (in which CSV is much easier to parse).

Yes, I think duckdb only reads CSV, then projects necessary data into internal format (which is probably more efficient than parquet, again based on my benchmarks), and does all ops (joins, aggregations) on that format.

Re: Big data is dead (2023)

#307

Post by fredstar »

I am in a software services company for more than 15 years. And to be honest, a lot of these big topics have always been some kind of sales talk or door opener. You write a white paper, nominate an 'expert' in your team and use these things in conversations with clients. Sure some trends are way more real and useful then others. But for me the article hits the nail on its head.

Re: Big data is dead (2023)

#308

Post by [deleted] »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

[deleted]

Re: Big data is dead (2023)

#309

Post by bartart »

vegabook wrote: post #6

my experience is that while data keeps growing at an exponential rate, its information content does not. In finance at least, you can easily get 100 million data points per series per day if you want everything, and you might be dealing with thousands of series. That sample rate, and the number of series, is usually 99.99% redundant, because the eigenvalues drop off almost to zero very quickly after about 10 dimensio…

That’s very interesting, so thank you — how do you handle if the eigenvectors change over time?

Re: Big data is dead (2023)

#310

Post by buremba »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

I can’t really think of a product with the requirement of max 6TiB data. If the data is big as TiB, most products have 100x TiB rather than a few ones.
Post reply on HN