Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead (2023)

motherduck.com

Post reply on HN
Print view View original

231–240 of 475 posts

Re: Big data is dead (2023)

#231

Post by marcosdumay »

lokimedes wrote: post #49

I was a researcher at the Large Hadron Collider around the time “Big Data” became a thing. We had one of the use cases where analyzing all the data made sense, since it boiled down to frequentist statistics, the more data, the better. Yet even with a global network of supercomputers at our disposal, we funnily figured out that fast local storage was better than waiting for huge jobs to finish. So, surprise, surprise,…

Let me try one:

"If you can't do your statistical analysis in 1 to 5 TB of data, your methodology is flawed"

This is probably more about human limitations than math. There's a clear ceiling in how much flexibility we can use. That will also change with easier ways to run new kinds of analysis, but it increases with the logarithm of the amount of things we want to do.

Re: Big data is dead (2023)

#232

Post by mk67 »

CoastalCoder wrote: post #99
vegabook wrote: post #6

my experience is that while data keeps growing at an exponential rate, its information content does not. In finance at least, you can easily get 100 million data points per series per day if you want everything, and you might be dealing with thousands of series. That sample rate, and the number of series, is usually 99.99% redundant, because the eigenvalues drop off almost to zero very quickly after about 10 dimensio…

Could you point to something explaining that eigenvalue / dimensions topic? It sounds interesting, but it's totally new to me.

https://en.wikipedia.org/wiki/Principal_component_analysis

Re: Big data is dead (2023)

#233

Post by michaelcampbell »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

My smartphone cannot store 1TiB.

Re: Big data is dead (2023)

#234

Post by whamlastxmas »

kmarc wrote: post #209
ufo wrote: post #186

Earlier quoted context omitted.

From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.

Is this like interviewing for a chef position for a fancy restaurant and when asked how to perfectly cook a steak, you preface it with “well you can either go to McDonald’s and get a burger, or…”

It may not be reasonable to suggest that in a role that traditionally uses big data tools

Re: Big data is dead (2023)

#235

Post by lizknope »

boppo1 wrote: post #51
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

You have 6 TiB of ram?

I personally don't but our computer cluster at work as around 50,000 CPU cores. I can request specific configurations through LSF and there are at least 100 machines with over 4TB RAM and that was 3 years ago. By now there are probably machines with more than that. Those machines are usually reserved for specific tasks that I don't do but if I really needed it I could get approval.

Re: Big data is dead (2023)

#236

Post by deepsquirrelnet »

marginalia_nu wrote: post #83
kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

Problem is possibly that most people with that sort of hands-on intuition for data don't see themselves as data scientists and wouldn't apply for such a position. It's a specialist role, and most people with the skills you seek are generalists.

Yeah it’s not really what you should be hiring a data scientist to do. I’m of the opinion that if you don’t have a data engineer, you probably don’t need a data scientist. And not knowing who you need for a job causes a lot of confusion in interviews.

Re: Big data is dead (2023)

#237

Post by qaq »

apwell23 wrote: post #128
qaq wrote: post #75

Earlier quoted context omitted.

You can have 8TB RAM in a 2U box for under 100K. grab a couple and it will save you millions a year compared to over-engineered bigdata setup.

Bigquery and snowflake are software. They come with a sql engine, data governance, integration with your ldap, auditing. Loading data into snowflake isn't overegineering. What you described is over-engineering. No business is passing 6tb data around on their laptops.

So is ClickHouse your point being ? Please point out what a server being able to have 8TB of RAM has to do with laptops.

Re: Big data is dead (2023)

#238

Post by apwell23 »

photonthug wrote: post #226
apwell23 wrote: post #205

Earlier quoted context omitted.

> No need to act smug and superior I responded with the same tone that gp responded with. "blows my mind" ( that people can be so stupid) .

Another comment mentions this classic meme: > Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000. A lot of industry work really does fall into this category, and it's not controversial to say that going the wrong way on this thing is mind-blowing. More than not being controversial, it's not confrontational , because his comment was…

i repurposed the parent comment

"not understanding the scale of "real" big data was a no-go in my eyes when hiring." , "real winner" ect.

But yea you are right. I shouldn't have directed it at commenter. I was miffed at interviewers who use "tricky questions" and expect people to read their minds and come up with their preconceived solution.

Re: Big data is dead (2023)

#239

Post by beng-nl »

CoastalCoder wrote: post #99
vegabook wrote: post #6

my experience is that while data keeps growing at an exponential rate, its information content does not. In finance at least, you can easily get 100 million data points per series per day if you want everything, and you might be dealing with thousands of series. That sample rate, and the number of series, is usually 99.99% redundant, because the eigenvalues drop off almost to zero very quickly after about 10 dimensio…

Could you point to something explaining that eigenvalue / dimensions topic? It sounds interesting, but it's totally new to me.

Nog OP, but i think they are referring to the fact that you can use PCA (principal component analysis) on a matrix of datapoints to approximate it. Works out of the box in scikit-learn.

You can do (lossy) compression on rows of vectors (treated like a matrix) by taking the top N eigenvectors (largest N eigenvalues) and using them to approximate the original matrix with increasing accuracy (as N grows) by some simple linear operations. If the numbers are highly correlated, you can get a huge amount of compression with minor loss this way.

Personally I like to use it to visualize linear separability of a high dimensioned set of vectors by taking a 2-component PCA and plotting them as x/y values.

Re: Big data is dead (2023)

#240

Post by dfgdfg34545456 »

kmarc wrote: post #32

When I was hiring data scientists for a previous job, my favorite tricky question was "what stack/architecture would you build" with the somewhat detailed requirements of "6 TiB of data" in sight. I was careful not to require overly complicated sums, I simply said it's MAX 6TiB I patiently listened to all the big query hadoop habla-blabla, even asked questions about the financials (hardware/software/license BOM) and…

The problem with your question is that they are there to show off their knowledge. I failed a tech interview once, question was build a web page/back end/db that allows people to order let's say widgets, that will scale huge. I went the simpleton answer route, all you need is Rails, a redis cache and an AWS provisioned relational DB, solve the big problems later if you get there sort of thing. Turns out they wanted to hear all about microservices and sharding.
Post reply on HN