Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

311–320 of 475 posts

Re: Big data is dead (2023)

#311

Post by wenc »

riku_iki wrote: post #306
wenc wrote: post #298

Earlier quoted context omitted.

I would love to see the benchmarks. That is not my experience, except in the rare case of a linear read (in which CSV is much easier to parse). CSV underperforms in almost every other domain, like joins, aggregations, filters. Parquet lets you do that lazily without reading the entire Parquet dataset into memory.

> That is not my experience, except in the rare case of a linear read (in which CSV is much easier to parse). Yes, I think duckdb only reads CSV, then projects necessary data into internal format (which is probably more efficient than parquet, again based on my benchmarks), and does all ops (joins, aggregations) on that format.

Yes, it does that, assuming you read in the entire CSV, which works for CSVs that fit in memory.

With Parquet you almost never read in the entire dataset and it's fast on all the projections, joins, etc. while living on disk.

Re: Big data is dead (2023)

#313

Post by riku_iki »

wenc wrote: post #311
riku_iki wrote: post #306

Earlier quoted context omitted.

> That is not my experience, except in the rare case of a linear read (in which CSV is much easier to parse). Yes, I think duckdb only reads CSV, then projects necessary data into internal format (which is probably more efficient than parquet, again based on my benchmarks), and does all ops (joins, aggregations) on that format.

Yes, it does that, assuming you read in the entire CSV, which works for CSVs that fit in memory. With Parquet you almost never read in the entire dataset and it's fast on all the projections, joins, etc. while living on disk.

> which works for CSVs that fit in memory.

what? Why CSV is required to fit in memory in this case? I tested CSVs which are far larger than memory, and it works just fine.

Re: Big data is dead (2023)

#314

Post by zaphar »

pdimitar wrote: post #267
zaphar wrote: post #215

Earlier quoted context omitted.

"blows my mind" is similar in tone yes. But I wasn't replying to the OP. Further the OP actually goes into some detail about how he would approach the problem. You do not. You are assuming you know what the OP meant by tricky question. And your assumption contradicts the rest of the OP's post regarding what he considered good answers to the question and why.

Honest question: was "blows my mind" so offensive? Thought it was quite obvious I meant that "it blows my mind people don't try the simpler stuff first, especially having in mind that it works for much bigger percentage than cloud providers would have you believe"? I guess it wasn't but even if so, it would be legitimately baffling how people manage to project so much negativity in three words that are slightly tongu…

I generally don't find that sort of thing offensive when combined with useful alternative approaches like your post provided. However the phrase does come with a connotation that you are surprised by a lack of knowledge or skill in others. That can be taken as smug or elitist by someone in the wrong frame of mind.

Re: Big data is dead (2023)

#315

Post by seabass-labrax »

jjgreen wrote: post #288
mrtimo wrote: post #168

Earlier quoted context omitted.

.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…

Please promote the use of .parquet files! apt-cache search parquet Maybe later

Parquet is a file format, not a piece of software. 'apt install csv' doesn't make any sense either.

Re: Big data is dead (2023)

#316

Post by pizzafeelsright »

apwell23 wrote: post #122
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

What kind of business just has a static set of 6TiB data that people are loading on their laptops. You tricked candidates with your nonsensical scenario. Hate smartass interviewers like this that are trying some gotcha to feel smug about themselves. Most candidates don't feel comfortable telling ppl 'just load on your laptops' even if they think thats sensible. They want to present a 'professional solution', esp when…

Big data companies or those that work with lots of data.

The largest dataset I worked with was about 60TB

While that didn't fit in ram most people would just load the sample data into the cluster when I told them it would be faster to load 5% locally and work off that.

Re: Big data is dead (2023)

#317

Post by dralley »

xethos wrote: post #300
crowcroft wrote: post #230

Earlier quoted context omitted.

I wonder if it's fair to revise this to 'your data set fits on NVME drives' these days. Astonishing how fast and how much storage you can get these days.

Based on a very brief search: Samsung's fastest NVME drives [0] could maybe keep up with the slowest DDR2 [1]. DDR5 is several orders of magnitude faster than both [2]. Maybe in a decade you can hit 2008 speeds, but I wouldn't consider updating the phrase before then (and probably not after, either). [0] https://www.tomshardware.com/reviews/samsung-980-m2-nvme-ssd... [1] https://www.tomshardware.com/reviews/ram-speed…

The statement was "fits on", not "matches the speed of".

Re: Big data is dead (2023)

#318

Post by internet101010 »

paulryanrogers wrote: post #44
threeseed wrote: post #29

Earlier quoted context omitted.

HN is the worst echo chamber around. Obsessed with this "you must use PostgreSQL for every use case" nonsense. And that anyone who actually has unique data needs is simply doing it for their resume or are over-engineering.

> Obsessed with this "you must use PostgreSQL for every use case" nonsense. Pg fans are certainly here asking "why not PG?". Yet so are fans of other DBs; like DuckDB, CouchDB, SQLite, etc.

I don't see so much DuckDB and CouchDB proselytizing but the SQLite force always out strong. I tend to divide the Postgres vs. SQLite decision on if the data in question is self-contained. Like am I pulling data from elsewhere (Postgres) or am I creating data within the application that is only used for the functionality of said application (SQLite).

Re: Big data is dead (2023)

#319

Post by giantrobot »

devsda wrote: post #221
L-four wrote: post #35

Earlier quoted context omitted.

Airline booking numbers used to just be the sector number of your booking record on the mainframes HDD.

This is such a simple scheme. I wonder how they dealt with common storage issues like backups and disks having bad sectors.

They're likely record based formatting rather than file based. At the high level the code is just asking for a record number from a data set. The data set is managed including redundancy/ECC by the hardware of that storage device.

Re: Big data is dead (2023)

#320

Post by hirsin »

palata wrote: post #266
kmarc wrote: post #209

Earlier quoted context omitted.

FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.

Sure, but as you said yourself: it's a trick question. How often does the employee have to answer trick questions without having any time to think in the actual job? As an interviewer, why not asking: "how would you do that in a setup that doesn't have much data and doesn't need to scale, and then how would you do it if it had a ton of data and a big need to scale?". There is no trick here, do you feel you lose infor…

Trick questions (although not known as such at the time) are the basis of most of the work we do? XY problem is a thing for a reason, and I cannot count the number of times my teams and I have ratholed on something complex only to realize we were solving for the wrong problem, i.e. A trick question.

As a sibling puts it though, it's a matter of level. Senior/staff and above? Yeah, that's mostly what you do. Lower than that, then you should be able to mostly trust those upper folks to have seen through the trick.

Post reply on HN