Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big data is dead

motherduck.com

Post reply on HN
Print view View original

371–380 of 444 posts

Re: Big data is dead

#371

Post by noobermin »

The less than a terabyte datasets being common had me awestruck. I, singular post-doctoral scientist noobermin[0], have processed terabytes of data at a time on HPC systems. Sure, a lot of it was garbage and I had to wade through it, but no one paid me millions to do it, I just did it to publish the papers. Sure, I needed the system which cost someone a lot of money, I suppose. But, I considered myself a small fry compared to some of the things others did on the system, particularly, hyrdrodynamics modellers. Moreover, I know I can probably process 100GB datasets on my own home PC, which isn't too impressive, it would just take longer (say a day or so instead of a hour or a few minutes). And this is with idk, python scripts using MPI. Yes, MPI because I'm a computational scientist and that's what HPC systems use, nothing fancy and likely the "legacy systems" he railed against in his pitches, but it worked.

I'm just awestruck, I could tell anyone that "large data" isn't really a bottleneck, but making sense of it is the very difficult part. My mentors kept pushing me to mention the sheer size of the datasets I process in talks because it sounds impressive, and I do do so, but I always knew it didn't matter because the interpretation and analysis is the hard part, not just the "sheer size."

[0] not going to use my real name

Re: Big data is dead

#372

Post by sortalongo »

> Customer data sizes followed a power-law distribution. The largest customer had double the storage of the next largest customer, the next largest customer had half of that, etc

I’m no statistician, but I’m like 99% sure that’s an exponential, not a power law

There’s a world of difference. The point of an exponential is that you can ignore big things. The point of a power law is that you can’t.

Re: Big data is dead

#373

Post by defrost »

fatneckbeard wrote: post #342
HybridCurve wrote: post #332

Earlier quoted context omitted.

Authoritarian types consider any information derived by science which is contrary to their position as invalid or irrelevant because facts challenge their authority and ability to exercise control.

yes. I used to think the Church had a honest disagreement with Galileo about heliocentricity. When I grew up I realized the Church never cared about orbits at all, what they care about is maintenance of status quo. And then when I got old, I realized, there is even a reason that some people want status quo... because they have usually been around long enough to see society fall apart into anarchy and mass murder, so…

"The Church" wasn't then and isn't now a monolith of opinion.

A modern characterisation of "The Galileo Affair" would be that he was SWAT'ed by someone he was really really mean online to.

    Thus the whole "Galileo affair" starts as a conflict initiated by a secular Aristotelian philosopher, who, unable to silence Galileo by philosophical arguments, uses religion to achieve his aim.  [1]
and

    While delle Colombe was almost alone in arguing publicly against Galileo, there was a group of scholars and churchmen who supported his Aristotelian views. After Galileo referred disparagingly to delle Colombe as 'pippione' ('pigeon'), his close friend the painter Lodovico Cigoli coined the nickname 'Lega del Pippione' ('The Pigeon League') for delle Colombe's group.[2]
Galileo literally refered to delle Colombe (and friends) as Simplicio (simple minded) and worse in his highly popular Dialogue Concerning the Two Chief World Systems [3] and within a year or so the Pigeon League got their revenge, using their influence to have religuous charges bought against Galileo.

The affair was complex since very early on Pope Urban VIII had been a patron to Galileo and had given him permission to publish on the Copernican theory .. this was very much a case of personal vendettas and internal politics rather than a straight up case of "The Church Versus Galileo".

[1] https://en.wikipedia.org/wiki/Galileo_affair#cite_note-Spell...

[2] https://en.wikipedia.org/wiki/Lodovico_delle_Colombe

[3] https://en.wikipedia.org/wiki/Dialogue_Concerning_the_Two_Ch...

Re: Big data is dead

#374

Post by anthomtb »

pradn wrote: post #190
gffrd wrote: post #105

Earlier quoted context omitted.

I've observed the same in manufacturing … and fitness trackers a la FitBit. There's initial value from training yourself on what something looks/feels like … but diminishing returns after that. Whether there is more value to be found doesn't seem to matter. Factories would sensor up, go nuts with data, find one or two major insight, tire of data, and then just continue operating how they were before … but with a few…

I went on a diet a few years ago. I obsessively recorded every food I ate in MyFitnessPal. To this day, I know roughly how many calories pretty much everything I eat is. So, I've learned from the process and don't need the process as much any more. (I'm kidding about that - it's easy to underestimate how much you eat, and an extra 200 cal a day adds up over the years.)

I did something similar in my early 20’s. 15 years later the weight is gone and hasn’t come back. It is pretty nice being able to eyeball my plate of food at a BBQ knowing it will not add any flesh to my mid section.

I definitely did not need a multi-PB storage array to reach that goal. Or did I? I am sure someone out there knows roughly the human brain storage capacity in MB.

Re: Big data is dead

#375

Post by wongarsu »

noobermin wrote: post #371

The less than a terabyte datasets being common had me awestruck. I, singular post-doctoral scientist noobermin[0], have processed terabytes of data at a time on HPC systems. Sure, a lot of it was garbage and I had to wade through it, but no one paid me millions to do it, I just did it to publish the papers. Sure, I needed the system which cost someone a lot of money, I suppose. But, I considered myself a small fry co…

> I'm just awestruck, I could tell anyone that "large data" isn't really a bottleneck, but making sense of it is the very difficult part

Especially now that 1TB datasets fit in memory on off-the-shelf servers and 100GB fits in memory on consumer hardware. You need a lot of data to run into real technical challenges that can't be solved by throwing a couple hundreed dollars a month (amortized cost) at hardware. And often you can get by with much, much less than even that.

Re: Big data is dead

#377

Post by wongarsu »

bombcar wrote: post #356
maliker wrote: post #191

Earlier quoted context omitted.

It appears that it's hard to detect AI generated content. E.g. true detection rates are only around 25% and there are also techniques to further mask output [1]. [1] https://www.nbcnews.com/tech/innovation/chatgpt-can-help-foo...

You only need enough to flag suspect content, and then the teacher calls the student in for a quick oral exam - the fakers will flounder, the reals will pass.

Even if you understand the material well, doing assigned tasks takes a lot of time. Especially if it's free form text. And "I want to do something more interesting right now" is at least as powerful a motivator to cheat assignments as "I don't know how to do this".

Re: Big data is dead

#378

Post by wongarsu »

fijiaarone wrote: post #370
criddell wrote: post #41

Earlier quoted context omitted.

Is it crazy to think that instead of stepping up in the war against AI we instead try to figure out a way to teach kids assuming they will use AIs?

Learn your times tables and then learn to use a calculator. If you don’t do it that way, you’ll always be fooled by whatever the screen tells you.

That's how we teach today, but that doesn't mean it's the only way. As a counter example, in chess, AI makes a great learning partner, and from beginners to world champions the level of proficiency has increased immensely in the last 30 years due to better chess engines.

Re: Big data is dead

#379

Post by timfsu »

qup wrote: post #60
criddell wrote: post #41

Earlier quoted context omitted.

Is it crazy to think that instead of stepping up in the war against AI we instead try to figure out a way to teach kids assuming they will use AIs?

The average human hates changes of things they've grown used to. People are very attached to school being just like it was when they went.

Isn't the problem that kids (like all of us) are lazy and would rather use tools than their brain, rather than people trying to keep school "just like it was"?

Re: Big data is dead

#380

Post by daveguy »

ff317 wrote: post #134
travisgriggs wrote: post #22

I've made anecdotal observations similiar to this over the last 10 years. I work in AgTech. A big push for a while here has been "more and more more data". Sensor-the-heck out of your farm, and We'll Tell You Things(tm). Most of what we as an industry are able to tell growers is stuff they already know or suspect. There is the occasional suprise or "Aha" moment where some correlation becomes apparent, but the thing a…

I tend to think the problem is the "random digging for correlations" part. Having tons of data is a Good Thing, so long as you can afford the marginal cost of gathering and managing all that data so that it's ready at hand when you need it later. It's how you use the data that makes all the difference. If you're facing an issue you don't understand at all, don't go digging for random correlations in your mountain of…

Even better... Once you have ~20 hypotheses, you'll get one right!
Post reply on HN