Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

131–140 of 475 posts

Re: Big data is dead (2023)

#131

Post by thfuran »

fijiaarone wrote: post #101

The problem with big data is that people don’t have data, they have useless noise. For lack of data, they generate random bytes collected on every mouse movement on every page, and every packet that moves through their network. It doesn’t tell them anything because the only information that means anything is who clicks that one button on their checkout page after filling out the form with their information or that on…

It absolutely would. You can't build a useful model off of occasionally looking outside.

Re: Big data is dead (2023)

#132

Post by tomrod »

randomtoast wrote: post #118
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Now, you have to consider the cost it takes for you whole team to learn how to use AWK instead of SQL. Then you do these TCO calculations and revert back to the BigQuery solution.

About $20/month for chatgpt or similar copilot, which really they should reach for independently anyhow.

Re: Big data is dead (2023)

#133

Post by filleokus »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

I think I've written about it here before, but I imported ≈1 TB of logs into DuckDB (which compressed it to fit in RAM of my laptop) and was done with my analysis before the data science team had even ingested everything into their spark cluster.

(On the other hand, I wouldn't really want the average business analyst walking around with all our customer data on their laptops all the time. And by the time you have a proper ACL system with audit logs and some nice way to share analyses that updates in real time as new data is ingested, the Big Data Solution™ probably have a lower TCO...)

Re: Big data is dead (2023)

#134

Post by renegade-otter »

Big Data is not dead - it has been reborn as AI, which is essentially Big Data 2.0.

And just in the same fashion, there was massive hype around Big Data 1.0. From 2013: https://hbr.org/2013/12/you-may-not-need-big-data-after-all

Everyone has so much data that they must use AI in order to tame it. The reality is, however, is that most of their data is crap and all over the place, and no amount of Big Data 1.0 or 2.0 is ever going to fix it.

Re: Big data is dead (2023)

#135

Post by thunky »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

> requirements of "6 TiB of data"

How could anyone answer this without knowing how the data is to be used (query patterns, concurrent readers, writes/updates, latency, etc)?

Awk may be right for some scenarios, but without specifics it can't be a correct answer.

Re: Big data is dead (2023)

#136

Post by renegade-otter »

quonn wrote: post #12
silvestrov wrote: post #7

Earlier quoted context omitted.

It has just been rebranded as AI. AI also use all the data, just with a magick neural network to figure out what it all means.

The overlap in terms of the used technologies, the required skills, the actual products and the target market is minimal. AI is not mostly Hadoop, it's not MapReduce, the hardware is different, the software is different, the skillset is very different and a chatbot or image generator is very different from a batch job producing an answer to a query.

But the underlying problem is the same - companies that use Big Data tech are clueless about data management. You can use unicorns - it's not going to do anything. "Garbage in, garbage out" is a timeless principle.

Re: Big data is dead (2023)

#137

Post by donatj »

The article only touches on it for a moment but GDPR killed big data. The vast majority of the data that any regular business would have and could be considered big almost certainly contained PII in one form or another. It became too much of a liability to keep that around.

With GDPR, we went from keeping everything by default unless a customer explicitly requested it gone to deleting it all automatically after a certain number of days after their license expires. This makes opaque data lakes completely untenable.

Don't get me wrong, this is all a net positive. The customers data is physically removed and they don't have to worry about future leaks or malicious uses, and we get a more efficient database. The only people really fussed were the sales team trying to lure people back with promises that they could pick right back up where they left off.

Re: Big data is dead (2023)

#138

Post by apwell23 »

pdimitar wrote: post #62
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Blows my mind. I am a backend programmer and a semi-decent sysadmin and I would have immediately told you: "make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs, I know dozens that work on CSV and XML, what's the problem?". And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of…

> make a ZFS or BCacheFS pool with 20-30% redundancy bits and just go wild with CLI programs

Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with.

Re: Big data is dead (2023)

#139

Post by clwg »

randomtoast wrote: post #118
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Now, you have to consider the cost it takes for you whole team to learn how to use AWK instead of SQL. Then you do these TCO calculations and revert back to the BigQuery solution.

Not necessarily. I always try to write to disk first, usually in a rotating compressed format if possible. Then, based on something like a queue, cron, or inotify, other tasks occur, such as processing and database logging. You still end up at the same place, and this approach works really well with tools like jq when the raw data is in jsonl format.

The only time this becomes an issue is when the data needs to be processed as close to real-time as possible. In those instances, I still tend to log the raw data to disk in another thread.

Re: Big data is dead (2023)

#140

Post by boxed »

CuriouslyC wrote: post #104
boxed wrote: post #57

Earlier quoted context omitted.

I see people planning for success to the point of guaranteeing failure, much more than people who suddenly must try to handle success in panic. It's a second system syndrome + survivor bias thing I think: people who had to clean up the mess of a good MVP complaining about what wasn't done before. But the companies that DID do that planning and architecting before did not survive to be complained about .

It's not either or. There are best practices that can be followed regardless with no time cost up front, and there is taking some time to think about how your product might evolve (which you really should be doing anyhow) then making choices with your software that don't make the evolution process harder than it needs to be. Layers of abstraction make code harder to reason about and work with, so it's a lose lose whe…

I'm a big fan of "optimize for deletion" (aka leaf-heavy) code. It's good for reasoning when the system is big, and it's good for growing a code base.

It's a bit annoying how the design of Django templates works against this by not allowing free functions...

Post reply on HN