Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

111–120 of 475 posts

Re: Big data is dead (2023)

#111

Post by Spooky23 »

stakhanov wrote: post #48

The funny thing about "big data" was that it came with a perverse incentive to avoid even the most basic and obvious optimizations on the software level, because the hardware requirement was how you proved how badass you were. Like: "Look, boss, I can compute all those averages for that report on just my laptop, by ingesting a SAMPLE of the data, rather than making those computations across the WHOLE dataset". Boss:…

It came with a few cohorts of Xooglers cashing their options out.

The amount of salesman hype and chatter about big data, followed by the dick measuring contests about whose data was big enough to be worthy was intense for awhile.

Re: Big data is dead (2023)

#112

Post by fifilura »

mattbillenstein wrote: post #76
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

I can appreciate the vertical scaling solution, but to be honest, this is the wrong solution for almost all use cases - consumers of the data don't want awk, and even if they did, spooling over 6TB for every kinda of query without partitioning or column storage is gonna be slow on a single cpu - always. I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you c…

I agree with this. BigQuery or AWS s3/Athena.

You shouldn't have to set up a cluster for data jobs these days.

And it kind of points out the reason for going with a data scientist with the toolset he has in mind instead of optimizing for a commandline/embedded programmer.

The tools will evolve in the direction of the data scientist, while the embedded approach is a dead end in lots of ways.

You may have outsmarted some of your candidates, but you would have hired a person not suited for the job long term.

Re: Big data is dead (2023)

#113

Post by fijiaarone »

__alexs wrote: post #85
mattbillenstein wrote: post #76

Earlier quoted context omitted.

I can appreciate the vertical scaling solution, but to be honest, this is the wrong solution for almost all use cases - consumers of the data don't want awk, and even if they did, spooling over 6TB for every kinda of query without partitioning or column storage is gonna be slow on a single cpu - always. I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you c…

A moderately powerful desktop processor has memory bandwidth of over 50TB/s so yeah it'll take a couple of minutes sure.

The slow part of using awk is waiting for the disk to spin over the magnetic head.

And most laptops have 4 CPU cores these days, and a multiprocess operating system, so you don’t have to wait for random access on a spinning plate to find every bit in order, you can simply have multiple awk commands running in parallel.

Awk is most certainly a better user interface than whatever custom BrandQL you have to use in a textarea in a browser served from localhost:randomport

Re: Big data is dead (2023)

#114

Post by civilized »

lokimedes wrote: post #49

I was a researcher at the Large Hadron Collider around the time “Big Data” became a thing. We had one of the use cases where analyzing all the data made sense, since it boiled down to frequentist statistics, the more data, the better. Yet even with a global network of supercomputers at our disposal, we funnily figured out that fast local storage was better than waiting for huge jobs to finish. So, surprise, surprise,…

I think there is a law of convenience, and it also explains why many technologies improve at a consistent exponential rate. People are very good at finding convenient ways to make something a little better each year, but every idea takes some minimal time to execute.

Re: Big data is dead (2023)

#115

Post by fijiaarone »

WesolyKubeczek wrote: post #74
pdimitar wrote: post #71

Earlier quoted context omitted.

That's included in CLI tools. Also duckdb and clickhouse-local are amazing.

I need to learn more about the latter for some log processing...

Log files aren’t data. That’s your first problem. But that’s the only thing that most people have that generates more bytes than can fit on screen in a single spreadsheet.

Re: Big data is dead (2023)

#116

Post by iamleppert »

What he means to say, is the grift is dead. All the best fish have been fished in that data lake (pun intended), leaving most waiting on the line to truly catch an appealing mackerel. Most of the big data people I know have moved on to more lucrative grifts like crypto and (more recently) AI.

There are still bags to be made if you can scare up a CTO or one of his lieutenants working for a small to medium size Luddite company. Add in storage on the blockchain and a talking AI parrot if you want some extra gristle in your grift.

Long live the tech grifters!

Re: Big data is dead (2023)

#117

Post by IneffablePigeon »

blagie wrote: post #16
brtkdotse wrote: post #11

Earlier quoted context omitted.

> 95% of businesses never become unicorns, but that's the goal for most Is it really the general case or is it just a HN echo chamber meme? My pet peeve is that patterns used by companies that in theory could become global unicorns are mimicked by companies where 5000 paying customers would mean an immense success

It's neither. Lifestyle companies are fine, if that's what you're aiming for. I know plenty of people who run or work at ≈1-30 person companies with no intention to grow. However, if you're going for high-growth, you need to plan for success. I've seen many potential unicorns stopped by simple lack of planning early on. Despite all the pivots which happen, if you haven't outlined a clear path from 1-3 people in a met…

If you’re in b2b 5000 customers can be a lot more revenue than that. 10-100x, depending hugely on industry and product.

Re: Big data is dead (2023)

#118

Post by randomtoast »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Now, you have to consider the cost it takes for you whole team to learn how to use AWK instead of SQL. Then you do these TCO calculations and revert back to the BigQuery solution.

Re: Big data is dead (2023)

#119

Post by zemo »

blagie wrote: post #4

Overall, I agree with much of this post, but there are several caveats: 1) Mongo is a bad point of reference. The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc. 2) Plan for success 95% of businesses never become unicorns, but that's the goal for most (for th…

> The reason to architect for scalability when you have 5 customers is so if that exponential growth cycle hits, you can capitalize on it.

If you have a product gaining that much traction, it’s usually because of some compound effect based on the existence and needs of its userbase. If on the way up you stumble to add new users, the userbase that’s already there is unlikely to go back to the Old Thing or go somewhere else (because these events are actually rare). For a good while using Twitter meant seeing the fail whale every day. Most people didn’t just up and leave, and nothing else popped up that could scale better that people moved to. Making a product that experiences exponential growth in that way is pretty rare, and struggling to scale those cases and having a period of availability degradation is common. What products hit an exponential growth situation failed because they couldn’t scale?

Re: Big data is dead (2023)

#120

Post by bee_rider »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

There’d still have to be some further questions, right? I guess if you store it on the interview group’s cellphones you’ll have to plan on what to do if somebody leaves or the interview room is hit by a meteor, if you plan to store it in ram on a server you’ll need some plan for power outages.
Post reply on HN