Storing data in object and querying from compute caching what you can basically scales until your queries are too expensive for a single node.
Big data is dead (2023)
291–300 of 475 posts
Re: Big data is dead (2023)
#292When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
As a point of reference, I routinely do fast-twitch analytics on tens of TB on a single, fractional VM. Getting the data in is essentially wire speed. You won't do that on Spark or similar but in the analytics world people consistently underestimate what their hardware is capable of by something like two orders of magnitude.
That said, most open source tools have terrible performance and efficiency on large, fast hardware. This contributes to the intuition that you need to throw hardware at the problem even for relatively small problems.
In 2024, "big data" doesn't really start until you are in the petabyte range.
Re: Big data is dead (2023)
#293Previous years I would have completely agreed with this post. Nowadays with the AI and ML craze I'm not so sure. I've seen plenty of companies using using vast amounts of data to train ML models to incorporate to their products. Definitely more data that can be handle by a traditional DB, and well into Big Data territory. This isn't a value judgement about whether that's a good idea, just an observation from talking…
Can you explain? It seems like you’re just using certain words without context.
Re: Big data is dead (2023)
#294I love how the way he talks about this. His paycheque from big-data is what is dead. His service offering was always bullshit.
Re: Big data is dead (2023)
#295When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…
How would six terabytes fit into memory? It seems like it would get a lot of swap thrashing if you had multiple processes operating on disorganized data. I'm not really a data scientist and I've never worked on data that size so I'm probably wrong.
6 TB does not fit in memory. However, with a good storage engine and fast storage this easily fits within the parameters of workloads that have memory-like performance. The main caveat is that if you are letting the kernel swap that for you then you are going to have a bad day, it needs to be done in user space to get that performance which constrains your choices.
Re: Big data is dead (2023)
#296Earlier quoted context omitted.
FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.
That's great, but it's really just desiderata about you and your personal situation. E.g., if a HN'er takes this as advice they're just as likely to be gated by some other interviewer who interprets hedging as a smell. I believe the posters above are essentially saying: you, the interviewer, can take the 2.5 seconds to ask the follow up, "... and if we're not immediately optimizing for scalability?" Then take that da…
> E.g., if a HN'er takes this as advice they're just as likely to be gated by some other interviewer who interprets hedging as a smell.
If people in high stakes environments interpret hedging as a smell - run from that company as fast as you can.
Hedging is a natural adult reasoning process. Do you really want to work with someone who doesn't understand that?
Re: Big data is dead (2023)
#297Earlier quoted context omitted.
https://x.com/garybernhardt/status/600783770925420546 (Gary Bernhardt of WAT fame): > Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000. This is from 2015...
I wonder if it's fair to revise this to 'your data set fits on NVME drives' these days. Astonishing how fast and how much storage you can get these days.
You can always check available ram: https://yourdatafitsinram.net/
Re: Big data is dead (2023)
#298Earlier quoted context omitted.
.parquet files are completely underrated, many people still do not know about the format! .parquet preserves data types (unlike CSV) They are 10x smaller than CSV. So 600GB instead of 6TB. They are 50x faster to read than CSV They are an "open standard" from Apache Foundation Of course, you can't peek inside them as easily as you can a CSV. But, the tradeoffs are worth it! Please promote the use of .parquet files! Ma…
> They are 50x faster to read than CSV I actually benchmarked this and duckdb CSV reader is faster than parquet reader.
I would love to see the benchmarks. That is not my experience, except in the rare case of a linear read (in which CSV is much easier to parse).
CSV underperforms in almost every other domain, like joins, aggregations, filters. Parquet lets you do that lazily without reading the entire Parquet dataset into memory.
Re: Big data is dead (2023)
#299The funny thing about "big data" was that it came with a perverse incentive to avoid even the most basic and obvious optimizations on the software level, because the hardware requirement was how you proved how badass you were. Like: "Look, boss, I can compute all those averages for that report on just my laptop, by ingesting a SAMPLE of the data, rather than making those computations across the WHOLE dataset". Boss:…
This is a pretty snarky outside view and just not actually true (I spent the first part of my career trying to reduce compute spend as a data engineer).
It was extremely difficult to get > 64gb on a machine for a very long time, and implementation complexity gets hard FAST when you have a hard cap.
And it's EXTREMELY disruptive to have a process that fails every 1/50 times, when data is slightly too large, because your team will be juggling dozens of these routine crons, and if each of them breaks regularly, you do nothing but dumb oncall trying to trim bits off of each strong.
No, Hadoop and MapReduce were not hyperefficient, but it was OK if you write it correctly, and having something that ran reliably is WAY more valuable than boutique bit-optimized C++ crap that nobody trusts or can maintain and fails every thursday with insane segfaults.
(nowdays, just use Snowflake. but it was a reasonable tool for the time).
Re: Big data is dead (2023)
#300Earlier quoted context omitted.
https://x.com/garybernhardt/status/600783770925420546 (Gary Bernhardt of WAT fame): > Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000. This is from 2015...
I wonder if it's fair to revise this to 'your data set fits on NVME drives' these days. Astonishing how fast and how much storage you can get these days.
Based on a very brief search: Samsung's fastest NVME drives [0] could maybe keep up with the slowest DDR2 [1]. DDR5 is several orders of magnitude faster than both [2]. Maybe in a decade you can hit 2008 speeds, but I wouldn't consider updating the phrase before then (and probably not after, either).
[0] https://www.tomshardware.com/reviews/samsung-980-m2-nvme-ssd...
[1] https://www.tomshardware.com/reviews/ram-speed-tests,1807-3....