Big data is dead • Hacker News Board

Re: Big data is dead

#251

Post by LarryMullins » Tue, Feb 07, 2023, 9:05 PM UTC

eganist wrote: post #200

LarryMullins wrote: post #176

Earlier quoted context omitted.

Years before wolfram alpha, we had TI-89s with computer algebra systems for cheating your way through highschool math.

Oh yeah, it's why most of my classes restricted us to TI-83s. The TI-89 was restricted in schools to basically calc and above, and the TI-92 was just banned. Lol

In my school TI-89s were unrestricted, but I think that was mostly because teachers were only trained with 83s and assumed the 89 had equivalent capabilities. The SAT permitting the 89 probably had something to do with it too, since the 92 was banned (because qwerty as I understand it.)

Re: Big data is dead

#252

Post by Loughla » Tue, Feb 07, 2023, 9:06 PM UTC

pcthrowaway wrote: post #197

Loughla wrote: post #150

Earlier quoted context omitted.

What a weird assumption you made there. In what way does what I wrote sound panicked? Because it was a week? Yes it was fast, but it was a massive effort of the entire english faculty. It's integrated into existing assignments, modifying processes that are super well established already. It was like integrating a new person into the class. Also, it was before semester, so the students literally saw nothing weird; tha…

> don't assume things. 9/10 times you're going to be incorrect Isn't that... an assumption?

No, it's an assertion. It's like the bastard cousin of an assumption, in that it's only incorrect 8.67 times out of 10.

Re: Big data is dead

#253

Post by threeseed » Tue, Feb 07, 2023, 9:07 PM UTC

lucidguppy wrote: post #13

Some of mongo's leveling off is the adoption of good jsonb columns in postgres. mongo's got sharding out of the box - which is nice - but you have to get your key right or it will suck. Also no one should want to host a mongo db - unless that's your business.

MongoDB grew revenue 52.8% in the previous financial year [1].

And if there is any levelling off it's going to be because of the move towards cloud managed options e.g. Snowflake, DocumentDB rather than because PostgreSQL decided to add JSONB support.

[1] https://www.macrotrends.net/stocks/charts/MDB/mongodb/revenu...

Re: Big data is dead

#254

Post by olyjohn » Tue, Feb 07, 2023, 9:12 PM UTC

mritchie712 wrote: post #72

fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

Size isn't the real problem, it's time. Are you going to take the time / money to set up a warehouse, get all the data into with an ETL product, set up dbt or some other transformation layer, set up a BI tool and build the reports and dashboards, etc. Regardless the size of your data, you still need to get it in one place and model it in a way it's actually usable.

Can't we just give it to that one IT guy down in the basement?

Re: Big data is dead

#255

Post by ravivooda » Tue, Feb 07, 2023, 9:17 PM UTC

ilyt wrote: post #192

ravivooda wrote: post #163

Earlier quoted context omitted.

It never sat well with me that none of the production services could leverage my local computation and storage power. I don't need to store my contacts on a remote server that could index my contacts when mixed with every other contact in a single table. That’s a blatantly oversimplified example but you get the gist.

Developing apps as local-mostly with remote being "just storage" might've been interesting approach but oh so many stuff moved to webshit from native apps and browsers still don't even have decent data management.

Well said! I wonder if Web3 could solve such a problem (or a zero trust solution). Where you provide your service that can run in a special container

Re: Big data is dead

#256

Post by hermitcrab » Tue, Feb 07, 2023, 9:19 PM UTC

karmajunkie wrote: post #61

slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

"Data launderer" would be a good job title...

Most data is very dirty.

Re: Big data is dead

#257

Post by I_complete_me » Tue, Feb 07, 2023, 9:20 PM UTC

happymellon wrote: post #96

pjmorris wrote: post #90

Earlier quoted context omitted.

> I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. So, a synonym for 'consultant?' :)

My experience with consultants normally ended up with them asking why they are there and what report should they present to upper management. I've always used them as "independent 3rd parties" who were listened to.

Here's my take on this not listening to the "expert":

A few years ago there was a problem with storm-water infiltration into my (elderly at the time) mother's property from her neighbor. I, being a dutiful son and a civil engineer, investigated it and came up with the probable cause, the likely effects of non-action and the most cost-effective solution. I presented it to my mother in the most layman-like terms that I could. She said she'd think about it – meaning she'd refer i.e. defer – to her daughters. In the meantime I had a very layman-like chat with my mother's carer and told her the situation in layman's terms. The carer listened and said that what I said I made total sense to her. Later on, one of my sisters accosted me and stated that it was completely obvious what the problem and the solution was – "even the carer could see it". Human foreheads don't have the real estate for where my eyebrows wanted to ascend.

My advice is to consider whether the message should be separated from the messenger somehow.

Re: Big data is dead

#258

Post by peatmoss » Tue, Feb 07, 2023, 9:21 PM UTC

slt2021 wrote: post #36

fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

> Data Scientist's job is to launder management's intuition using quantitative methods

Ouch. This is savage, but sadly correct in many cases.

HOWEVER, to play devil's advocate here, I've also seen corporate data scientists overstate the conclusions / generalizability of their analysis. I've also seen data scientists fall prey to believing that their analysis proves would should be done, rather than what is likely to happen.

The role of an executive or decision maker is to apply a normative lens to problems. The role of the data scientist / economist / whatever is to reduce the uncertainty that an action will have the desired effect.

Re: Big data is dead

#259

Post by stcroixx » Tue, Feb 07, 2023, 9:22 PM UTC

Loughla wrote: post #150

stcroixx wrote: post #108

Earlier quoted context omitted.

Sounds like they simply panicked and threw something together with little thought or preparation, what, right in the middle of an actual course? And they want to charge kids for this kind of 'expert instruction'? I'd be pissed as a student.

What a weird assumption you made there. In what way does what I wrote sound panicked? Because it was a week? Yes it was fast, but it was a massive effort of the entire english faculty. It's integrated into existing assignments, modifying processes that are super well established already. It was like integrating a new person into the class. Also, it was before semester, so the students literally saw nothing weird; tha…

I simply don’t buy that they had anywhere close to enough time to integrate this into an existing curriculum in a way that would meet the high standards paying students should have for a university education. I don’t have to ask how an English department became experts in using a just released AI in the classroom in a week because I don’t think that’s what they actually achieved.

Re: Big data is dead

#260

Post by ryguyrg » Tue, Feb 07, 2023, 9:23 PM UTC

boh wrote: post #242

fdgsdfogijq wrote: post #14

"For more than a decade now, the fact that people have a hard time gaining actionable insights from their data has been blamed on its size." The real issue is that business people usually ignore what the data says. Wading through data takes a huge amount of thought, which is in short supply. Data Scientists are commonly disregarded by VPs in large corporations, despite the claims about being "data driven". Most corpo…

The other thing to consider is that data simply has nothing of value. Part of the marketing of big data is the almost fairy tale belief in "insights" existing in any data set if you just look hard enough.

Correct, much data has no value. The cost for storing the data, maintaining the data [in the day-and-age of privacy requirements especially], and combing through the data is often much greater than the value obtained from the data itself.

The expertise we need in the industry is people who understand applications in-and-out and make great decisions on what data is worth keeping for present and future applications. And what data is needed to be kept, but only in aggregates (or anonymized, which reduces costs of maintenance)