Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

431–440 of 475 posts

Re: Big data is dead (2023)

#431

Post by lelanthran »

hipadev23 wrote: post #261
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Huh? How are you proposing loading a 6TB CSV into memory multiple times? And then processing with awk, which generally streams one a line at a time. Obviously we can get boxes with multiple terabytes of RAM for $50-200/hr on-demand but nobody is doing that and then also using awk. They’re loading the data into clickhouse or duckdb (at which point the ram requirement is probably 64-128GB) I feel like this is an anecdo…

> How are you proposing loading a 6TB CSV into memory multiple times? And then processing with awk, which generally streams one a line at a time.

Ramdisk would work.

Re: Big data is dead (2023)

#432

Post by assmanreturns »

hipadev23 wrote: post #261
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Huh? How are you proposing loading a 6TB CSV into memory multiple times? And then processing with awk, which generally streams one a line at a time. Obviously we can get boxes with multiple terabytes of RAM for $50-200/hr on-demand but nobody is doing that and then also using awk. They’re loading the data into clickhouse or duckdb (at which point the ram requirement is probably 64-128GB) I feel like this is an anecdo…

Awk doesn't load things into memory. It processes one line at a time. So memory usage is basically zero. That said awk isn't that fast. I mean your looking at "query" times in the range of at least 30 minutes if not more.

Awk is imo a poor solution. I use awk all the time and I would never use it for something like this. Why not just use postgres. Its a lot more powerful, easy to setup and you get SQL which is extremely powerful. Normally I might even go with sqllite but for me 6TB is too much for sqllite.

Re: Big data is dead (2023)

#433

Post by assmanreturns »

internet101010 wrote: post #318
paulryanrogers wrote: post #44

Earlier quoted context omitted.

> Obsessed with this "you must use PostgreSQL for every use case" nonsense. Pg fans are certainly here asking "why not PG?". Yet so are fans of other DBs; like DuckDB, CouchDB, SQLite, etc.

I don't see so much DuckDB and CouchDB proselytizing but the SQLite force always out strong. I tend to divide the Postgres vs. SQLite decision on if the data in question is self-contained. Like am I pulling data from elsewhere (Postgres) or am I creating data within the application that is only used for the functionality of said application (SQLite).

I think differently. If multiple users need to query the database directly then postgres. If no one is ever going to look at the db except one person or one application then sqllite. It's an interesting question as to whether micro services should be sqllite.

Postgres is recommended because it's free, easy to maintain, install and it's has excellent query features.

Re: Big data is dead (2023)

#434

Post by justsomehnguy »

xethos wrote: post #300
crowcroft wrote: post #230

Earlier quoted context omitted.

I wonder if it's fair to revise this to 'your data set fits on NVME drives' these days. Astonishing how fast and how much storage you can get these days.

Based on a very brief search: Samsung's fastest NVME drives [0] could maybe keep up with the slowest DDR2 [1]. DDR5 is several orders of magnitude faster than both [2]. Maybe in a decade you can hit 2008 speeds, but I wouldn't consider updating the phrase before then (and probably not after, either). [0] https://www.tomshardware.com/reviews/samsung-980-m2-nvme-ssd... [1] https://www.tomshardware.com/reviews/ram-speed…

980 is an M.2 drive, PCIe 3.0 x4, 3 years old, up to 3500MB/s sequential read.

You want something like PM1735: PCIe 4.0 x8, up to 8000 MB/s sequential read.

And while DDR5 is surely faster the question is what the data access patterns are there.

In almost all cases (ie mix of random access, occasional sequential reads) just reading from the NVMe drive would be faster than loading to RAM and reading from there. In some cases you would spend more time processing the data than reading it.

PS all these RAM bandwidth rates are good for the sequential access, as you go random access the bandwidth drops.

https://semiconductor.samsung.com/ssd/enterprise-ssd/pm1733-...

Re: Big data is dead (2023)

#435

Post by therealdrag0 »

snakeyjake wrote: post #344
zurfer wrote: post #8

This is not fully correct. Originally big data was defined by 3 dimensions: - Volume (mostly what the author talks about) [solved] - Velocity, how fast data is processed etc [solved, but expensive] - Variety [not solved] Big Data today is not: I don't have enough storage or compute. It is: I don't have enough cognitive capacity to integrate and make sense of it.

>Big Data today is not: I don't have enough storage or compute. It is for me. Six times per year I go out to the field for two weeks to do data acquisition. In the field we do a dual-aircraft synthetic aperture radar collection over four bands and dual polarities. That means two aircraft each with one radar system containing eight 20TiB 16-drive RAID-0 SSD storage devices. We don't usually fill up the RAIDs so we gen…

Thanks for sharing. That was interesting.

Re: Big data is dead (2023)

#436

Post by [deleted] »

int_19h wrote: post #392
xethos wrote: post #300

Earlier quoted context omitted.

Based on a very brief search: Samsung's fastest NVME drives [0] could maybe keep up with the slowest DDR2 [1]. DDR5 is several orders of magnitude faster than both [2]. Maybe in a decade you can hit 2008 speeds, but I wouldn't consider updating the phrase before then (and probably not after, either). [0] https://www.tomshardware.com/reviews/samsung-980-m2-nvme-ssd... [1] https://www.tomshardware.com/reviews/ram-speed…

I think the point is that if it fits on a single drive, you can still get away with a much simpler solution (like a traditional SQL database) than any kind of "big data" stack.

[deleted]

Re: Big data is dead (2023)

#437

Post by otabdeveloper4 »

ufo wrote: post #186
palata wrote: post #167

Earlier quoted context omitted.

One thing that may have an impact on the answers: you are hiring them, so I assume they are passing a technical interview. So they expect that you want to check their understanding of the technical stack. I would not conclude that they over-engineer everything they do from such an answer, but rather just that they got tricked in this very artificial situation where you are in a dominant position and ask trick questio…

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

You don't need to guess, you can just ask the interviewer. (Shocker, I know.)

Re: Big data is dead (2023)

#438

Post by ddalex »

sdenton4 wrote: post #202
ddalex wrote: post #185

Earlier quoted context omitted.

Why is .parquet better than protobuf?

Parquet is columnar storage, which is much faster for querying. And typically for protobuf you deserialize each row, which has a performance cost - you need to deserialize the whole message, and can't get just the field you want. So, of you want to query a giant collection of protobufs, you end up reading and deserializing every record. For parquet, you get much closer to only reading what you need.

Thank you.

Re: Big data is dead (2023)

#439

Post by weebull »

andoando wrote: post #353
ufo wrote: post #186

Earlier quoted context omitted.

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

Its great if the interviewer actually takes time to sort out the questions you have, cause seemingly simple questions to you have a lot of assumptions you made. I had an interview "design an app store". I tried asking, ok an app store has a ton of components, which part of the app store are you asking exactly? The response I got was "Have you ever used an app store? Design an app store". Umm ok.

Red flag - walk away. Interview did it's job.

Re: Big data is dead (2023)

#440

Post by weebull »

int_19h wrote: post #392
xethos wrote: post #300

Earlier quoted context omitted.

Based on a very brief search: Samsung's fastest NVME drives [0] could maybe keep up with the slowest DDR2 [1]. DDR5 is several orders of magnitude faster than both [2]. Maybe in a decade you can hit 2008 speeds, but I wouldn't consider updating the phrase before then (and probably not after, either). [0] https://www.tomshardware.com/reviews/samsung-980-m2-nvme-ssd... [1] https://www.tomshardware.com/reviews/ram-speed…

I think the point is that if it fits on a single drive, you can still get away with a much simpler solution (like a traditional SQL database) than any kind of "big data" stack.

I always heard it as "if the database index fits in the RAM of a single machine, it's not big data". The reason being that this makes random access fast. You always know where a piece of data is.

Once the index is too big to have in one place, thing get more complicated.

Post reply on HN