Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

381–390 of 444 posts

Re: Big data is dead

#381

Post by miguelazo »

Loughla wrote: post #292
stcroixx wrote: post #259

Earlier quoted context omitted.

I simply don’t buy that they had anywhere close to enough time to integrate this into an existing curriculum in a way that would meet the high standards paying students should have for a university education. I don’t have to ask how an English department became experts in using a just released AI in the classroom in a week because I don’t think that’s what they actually achieved.

Again, stop putting words in my mouth, please. I never said they became experts. I said they integrated it into existing processes. I said that they were doing something other than just scanning for chatgpt hits like plagiarism checkers. >It's part of the collaborative idea building and development process now for every student enrolled in creative writing and writing analysis classes >It's integrated into existing a…

It sounds like they’re chasing hype, like every idiotic modern US movement to “reform” education.

Re: Big data is dead

#382

Post by noisy_boy »

Loughla wrote: post #292
stcroixx wrote: post #259

Earlier quoted context omitted.

I simply don’t buy that they had anywhere close to enough time to integrate this into an existing curriculum in a way that would meet the high standards paying students should have for a university education. I don’t have to ask how an English department became experts in using a just released AI in the classroom in a week because I don’t think that’s what they actually achieved.

Again, stop putting words in my mouth, please. I never said they became experts. I said they integrated it into existing processes. I said that they were doing something other than just scanning for chatgpt hits like plagiarism checkers. >It's part of the collaborative idea building and development process now for every student enrolled in creative writing and writing analysis classes >It's integrated into existing a…

Not who you are replying to. I found your example fascinating - would you be able to share one or two concrete examples of this integration you mentioned? I would like a low-level peek or two into how the teaching landscape is changing in light of the rise of LLMs like chatGPT.

Re: Big data is dead

#383

Post by miguelazo »

nickersonm wrote: post #181
drowsspa wrote: post #179

Earlier quoted context omitted.

Pretty sure there's a fallacy named after this whole "hey this is just exactly like before so we have nothing to be concerned about".

The point is that all technology is a tool. Whether it be writing, calculators, or various narrow AI software. We can either bemoan the loss of a now-less-useful skill (memorization, long division, longform writing), or learn how to use these tools to better achieve our goals.

Technologies develop for different purposes and have different effects. Modern digital technology has a very inhumane origin story, as noted in “New Dark Age” by James Bridle, and other works. https://jamesbridle.com/books/new-dark-age

Re: Big data is dead

#384

Post by hackernewds »

wongarsu wrote: post #375
noobermin wrote: post #371

The less than a terabyte datasets being common had me awestruck. I, singular post-doctoral scientist noobermin[0], have processed terabytes of data at a time on HPC systems. Sure, a lot of it was garbage and I had to wade through it, but no one paid me millions to do it, I just did it to publish the papers. Sure, I needed the system which cost someone a lot of money, I suppose. But, I considered myself a small fry co…

> I'm just awestruck, I could tell anyone that "large data" isn't really a bottleneck, but making sense of it is the very difficult part Especially now that 1TB datasets fit in memory on off-the-shelf servers and 100GB fits in memory on consumer hardware. You need a lot of data to run into real technical challenges that can't be solved by throwing a couple hundreed dollars a month (amortized cost) at hardware. And of…

Not to mention if you're physically storing it (not in the cloud) 1TB can fit on your pinky finger

Re: Big data is dead

#385

Post by raindropm »

gffrd wrote: post #105
travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

I've observed the same in manufacturing … and fitness trackers a la FitBit. There's initial value from training yourself on what something looks/feels like … but diminishing returns after that. Whether there is more value to be found doesn't seem to matter. Factories would sensor up, go nuts with data, find one or two major insight, tire of data, and then just continue operating how they were before … but with a few…

Your comment made me took my Fitbit out of drawer again :p At least I can have some fun number and graph about my health to look at when I want to.

Re: Big data is dead

#386

Post by lelanthran »

neuronic wrote: post #243
happymellon wrote: post #96

Earlier quoted context omitted.

My experience with consultants normally ended up with them asking why they are there and what report should they present to upper management. I've always used them as "independent 3rd parties" who were listened to.

As a consultant with roots in backend dev, I fully understand the scrutiny that we receive because unfortunately, it is often very warranted... It feels a bit refreshing to read your comment and see someone articulate what I am trying to convey to my clients. I am a tool, and yes, this pun is intended.

Hello

I want to make the move from development into consultancy and would appreciate hearing how you did it.

I cannot DM you, but if you have the time to type a small paragraph, I'm all ears.

If you don't want to make that public, you can directly email me (@gmail.com)

Re: Big data is dead

#387

Post by yMEyUyNE1 »

karmajunkie wrote: post #61
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

"Data launderer" would be a good job title...

Data Torturer? https://en.wiktionary.org/wiki/if_you_torture_the_data_long_...>

Re: Big data is dead

#388

Post by somenameforme »

koolba wrote: post #143
slt2021 wrote: post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

> if you want to succeed as a Data Scientist and be praised by management - you got to provide data analysis that supports managements ideas (however wrong or ineffective they might be). > Data Scientist's job is to launder management's intuition using quantitative methods :) It’s no different than the days when grey bearded wisemen would read the stars and weave a tale about the great glory that awaits the king if h…

If you [Croesus] go to war, a great empire will fall.

Re: Big data is dead

#389

Post by nicklaf »

lucidguppy wrote: post #13

Some of mongo's leveling off is the adoption of good jsonb columns in postgres. mongo's got sharding out of the box - which is nice - but you have to get your key right or it will suck. Also no one should want to host a mongo db - unless that's your business.

"Shards are the secret ingredient in the webscale sauce": https://www.youtube.com/watch?v=b2F-DItXtZs
Post reply on HN