Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

441–450 of 475 posts

Re: Big data is dead (2023)

#441

Post by stakhanov »

bpodgursky wrote: post #299
stakhanov wrote: post #48

The funny thing about "big data" was that it came with a perverse incentive to avoid even the most basic and obvious optimizations on the software level, because the hardware requirement was how you proved how badass you were. Like: "Look, boss, I can compute all those averages for that report on just my laptop, by ingesting a SAMPLE of the data, rather than making those computations across the WHOLE dataset". Boss:…

This is a pretty snarky outside view and just not actually true (I spent the first part of my career trying to reduce compute spend as a data engineer). It was extremely difficult to get > 64gb on a machine for a very long time, and implementation complexity gets hard FAST when you have a hard cap. And it's EXTREMELY disruptive to have a process that fails every 1/50 times, when data is slightly too large, because yo…

Why do you assume it's an "outside" view? I was freelancing for many years on projects with "Data Scientist" / "Data Engineer" in my job title, having been hired by managers who heard about "Big Data" at an event by KPMG or Accenture or whatever.

And I don't understand why you're reading "boutique bit-optimized C++ crap" into "most basic and obvious optimizations".

One of those most basic and obvious optimizations is to avoid reading a dataframe into memory in its entirety, when the math that you want to do on top of it can actually be done as a running accumulator while reading the data from a stream. This is possible in 90% of realistic use cases, but the fraction of software written back then that took advantage of this was shockingly small. Solving the problem by buying more machines, chopping up the dataframe into smaller pieces, and farming out the payload through Hadoop had management buy-in. Yet, for some reason, doing the sane thing, namely rewriting poorly-written software, didn't.

Re: Big data is dead (2023)

#442

Post by rldjbpin »

"_ is dead" titles rarely represent reality. same case here.

more accurately, "big data" or its main paradigms are not for everyone. on the other hand, most teams are still unable to reasonably consolidate their data to leverage the benefits of these tools.

"AI" will have a similar fate in a few years time for similar reasons.

Re: Big data is dead (2023)

#443

Post by pdinny »

fock wrote: post #148
throwaway_20357 wrote: post #129

Earlier quoted context omitted.

It depends on what you want to do with the data. It can be easier to just stick nicely-compressed columnar Parquets in S3 (and run arbitrarily complex SQL on them using Athena or Presto) than to try to achieve the same with shell-scripting on CSVs.

how exactly is this solution easier than putting the very Parquet files on a classic filesystem. Why does the easy solution require an amazon-subscription?

This is adjacent to "why would I need EC2 when I can serve from my laptop?"

In terms of maturity of solution and diversity of downstream applications you'll go much further with BigQuery/Athena (at comically low cost) for this amount of data than some cobbled together "local" solution.

I thoroughly agree with the author but the comments in this thread are an indication of people who haven't actually had to do meaningful ongoing work with modest amounts of data if they're suggesting just storing it as plain text on personal devices.

I'm not advocating for complicated or expensive solutions here, BigQuery and Athena are very low complexity compared to any of the Hadoop et-al tooling (yes Athena is Trino is in the family, but it is managed and dirt cheap).

Re: Big data is dead (2023)

#444

Post by meetingthrower »

coffeebeqn wrote: post #260
jancsika wrote: post #247

Earlier quoted context omitted.

That's great, but it's really just desiderata about you and your personal situation. E.g., if a HN'er takes this as advice they're just as likely to be gated by some other interviewer who interprets hedging as a smell. I believe the posters above are essentially saying: you, the interviewer, can take the 2.5 seconds to ask the follow up, "... and if we're not immediately optimizing for scalability?" Then take that da…

This is the crux of it. Another interviewer would’ve marked “run on a local machine with a big SSD” - as: this fool doesn’t know enough about distributed systems and just runs toy projects on one machine

Exactly!!! This is an interviewer issue. You should be guiding them. Get the answers for each scenario out of both sets of interviewees and see if they are facile enough to roll with it. Doing a one shot "quiz" is not that revealing....

Re: Big data is dead (2023)

#445

Post by dredmorbius »

davedx wrote: post #19
blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because peop…

A/K/A "premature optimisation is the root of all evil".

http://c2.com/cgi/wiki?PrematureOptimization>

https://news.ycombinator.com/item?id=5673015>

Re: Big data is dead (2023)

#446

Post by fatnoah »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Somewhat ironically, I'm pretty sure I failed a system design interview at a big tech last year for not drastically over building to solve a problem that probably had far less data (movie showtimes, and IIRC there are less than 50k screens in the US).

Re: Big data is dead (2023)

#447

Post by foobarian »

ufo wrote: post #186
palata wrote: post #167

Earlier quoted context omitted.

One thing that may have an impact on the answers: you are hiring them, so I assume they are passing a technical interview. So they expect that you want to check their understanding of the technical stack. I would not conclude that they over-engineer everything they do from such an answer, but rather just that they got tricked in this very artificial situation where you are in a dominant position and ask trick questio…

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

But the interview is for a data science position. Why play games with "Ha ha 6 TiB can fit on my PC we actually don't need you?"

Re: Big data is dead (2023)

#448

Post by datadrivenangel »

crowcroft wrote: post #230
_ugfj wrote: post #80

Earlier quoted context omitted.

https://x.com/garybernhardt/status/600783770925420546 (Gary Bernhardt of WAT fame): > Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000. This is from 2015...

I wonder if it's fair to revise this to 'your data set fits on NVME drives' these days. Astonishing how fast and how much storage you can get these days.

Your data access patterns are fast enough in NVME. You own me $10,000 for saving you $250,000 (in ram).

The value of our data skills are getting eroded!

Re: Big data is dead (2023)

#449

Post by taylodl »

To me, the best thing to come out of the Big Data era was Spark Streaming. That tool provides a nice architecture for processing a stream of data in near real-time, from multiple data sources no less, that we didn't have before.

Spark in general was a great tool to get out of the Big Data era.

Re: Big data is dead (2023)

#450

Post by hnuser123456 »

__alexs wrote: post #85
mattbillenstein wrote: post #76

Earlier quoted context omitted.

I can appreciate the vertical scaling solution, but to be honest, this is the wrong solution for almost all use cases - consumers of the data don't want awk, and even if they did, spooling over 6TB for every kinda of query without partitioning or column storage is gonna be slow on a single cpu - always. I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you c…

A moderately powerful desktop processor has memory bandwidth of over 50TB/s so yeah it'll take a couple of minutes sure.

A high-end regular desktop is around 100 GB/s to DRAM, 750 GB/s to L3, 1.5 TB/s to L2, 4 TB/s to L1. 50 TB/s would require about 1000 channels of RAM.
Post reply on HN