Live data from Hacker News

Big data is dead

motherduck.com

411–420 of 444 posts

Re: Big data is dead

#411
post #277

While I get that they're sometimes useful to trigger debate, I don't really subscribe to very bold statements. We are drowning in data, it's all around us. Information overload is real. Data enables most of our daily digital experiences, from operational data to insights in the form of user facing analytics. Data systems are the backbone of the digital life. It's is an ocean and it's all about the vessel you pick to…

The heading is definitely “clickbait-ey” but the quality of the content was worth it. I probably would have missed the article without the headline. And I am already applying the insights gained.

Re: Big data is dead

#412

Earlier quoted context omitted.

As a consultant with roots in backend dev, I fully understand the scrutiny that we receive because unfortunately, it is often very warranted... It feels a bit refreshing to read your comment and see someone articulate what I am trying to convey to my clients. I am a tool, and yes, this pun is intended.

Hello I want to make the move from development into consultancy and would appreciate hearing how you did it. I cannot DM you, but if you have the time to type a small paragraph, I'm all ears. If you don't want to make that public, you can directly email me ( @gmail.com)

Sure, it is actually not very complicated in my case. I did backend development for a short while during and after university and then moved into IT consulting fairly quickly.

It was a LinkedIn recruiter message which I usually ignore. However, my SO did not (she is in IT as well) and convinced me to join a hiring event. I ended up liking it a lot and went through the hiring process. Soon, I started out on the most junior level and joined my first project with 3 very senior colleagues after a few weeks.

The learning curve was very steep both on the technical level and also regarding the consulting aspect - at first there was nothing I could 'consult' on due to lack of experience. This changed with growing experience, with the guidance of senior colleagues and my private efforts to gain skills and expertise.

Let me know if this was helpful!

Re: Big data is dead

#413
Unlike quantum which cracks computationally complex algorithms, BigData was just about costs.

SSDs we’re limited in capacity and still expensive.

Parallelizing work with MapReduce allowed using cheap fault-prone commodity hardware and disks.

If you’re dealing with terabytes rather than petabytes of data, you probably don’t need BigData

Re: Big data is dead

#414
post #75

Earlier quoted context omitted.

One may follow the other, but not vice versus. It's a pretty strong argument to say that Microsoft under Gates was technically obsessed, but that really faltered under Balmer. Microsoft continued to win profits, but they made major strategic missteps that cost the revenue. Amazon feels like it's going down the same path: empowering the tree-gazers without remembering that the forest also matters.

You can also “coast upwards” for quite awhile - as an example we now pay Microsoft less than we were for periodic upgrades to Office for the entire Microsoft 365 suite (including email hosting, etc) but all the machines are now Macs. They make more from us in one way, but less in total dollars.

Exactly. I can't imagine some accountant hasn't come up with a way to quantify this.

{Revenue attributable to previous R&D} (aka coast) vs {Revenue attributable to current R&D} (aka acceleration)

Re: Big data is dead

#415
post #36

Earlier quoted context omitted.

I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…

> Data Scientist's job is to launder management's intuition using quantitative methods Ouch. This is savage, but sadly correct in many cases. HOWEVER, to play devil's advocate here, I've also seen corporate data scientists overstate the conclusions / generalizability of their analysis. I've also seen data scientists fall prey to believing that their analysis proves would should be done, rather than what is likely to…

Good point. Data is one aspect of making a decision. The other aspect is understanding the industry and environment. Often data scientists give just one variable needed to make a decision. In health care for example you need to factor in a whole host of legislation. You also need to factor aspects of the industry not reflected in the data. As an example doctors not wanting to use iPads is something you can't measure and can't force as company. Even though data analysis might suggest this is the way to go.

Re: Big data is dead

#416

Earlier quoted context omitted.

Hello I want to make the move from development into consultancy and would appreciate hearing how you did it. I cannot DM you, but if you have the time to type a small paragraph, I'm all ears. If you don't want to make that public, you can directly email me ( @gmail.com)

Sure, it is actually not very complicated in my case. I did backend development for a short while during and after university and then moved into IT consulting fairly quickly. It was a LinkedIn recruiter message which I usually ignore. However, my SO did not (she is in IT as well) and convinced me to join a hiring event. I ended up liking it a lot and went through the hiring process. Soon, I started out on the most j…

This almost reads like my trajectory so far, but I'm at the point where I can't really consult due to the lack of experience, but I did make a good impression so far. Can I ask you, into what efforts should I put my private time? More technical knowledge? Into very fine details, or brief insights into different areas? Any good resources?

Thanks a lot!

Re: Big data is dead

#417

Earlier quoted context omitted.

Again, stop putting words in my mouth, please. I never said they became experts. I said they integrated it into existing processes. I said that they were doing something other than just scanning for chatgpt hits like plagiarism checkers. >It's part of the collaborative idea building and development process now for every student enrolled in creative writing and writing analysis classes >It's integrated into existing a…

Not who you are replying to. I found your example fascinating - would you be able to share one or two concrete examples of this integration you mentioned? I would like a low-level peek or two into how the teaching landscape is changing in light of the rise of LLMs like chatGPT.

Sure - I'll use one of the creative writing classes.

In the past, the class would be centered around ideas and themes the class came up with together during the first week of the semester. They would then read and discuss short stories from various authors centered around that theme, preferably from different eras and/or cultures. From there, they would work in pairs/small groups to flush out original ideas they came up with for their own stories based on the themes and styles from the first month of reading. Then they would work individually to write the stories. Finally, they would come back together to edit and work through that process as a group, with a reading and discussion in the last week or two.

Now, the class works alone the first day or two to discuss themes with chatgpt, to identify relevant and appropriate literature (if possible), and to flush out initial ideas for the readings and discussion topics. Then we come back together for group work to figure out themes for the semester and possible readings, like it used to be, but primed with whatever information they had already discussed with Chatgpt. From there they work through the readings, using chatgpt to bounce ideas for discussion off of before coming to class. Then they work in teams, like before, to flush out their own stories, but again, using chatgpt as if it were another member of the group. They also have to track their question/comment and response, to evaluate their own thought processes and look for weak links in their reasoning and logic. Students are then free to use the software to edit their writing before coming together with their creative assignments.

We haven't made it past the step of reading, as it's still the first month of the semester. But,the discussion for the first section has been much more in-depth and (to use a word that is impossible to quantify) vibrant. The students had already aired the ideas they thought may be dumb, and would therefore be less willing to voice in a public setting; this allows them to really dive further into whatever is in each story, and connect dots between stories that generally took a week or two. Because they have another 'person' to talk to whenever they want, and however much they want, they tend to really get into the work. Further, and unexpectedly (and anecdotally sadly) it has allowed a couple of students I know personally who would not have been willing to participate in public discussion (anxiety disorder and TBI) a stronger voice, because they know their ideas are flushed out already, and it provides them with a 'script' of ideas they know are novel and valuable for the class. In other words, if they were able to get the idea from Chatgpt, they knew they had more work to do to build that thought, because that is really just a baseline.

I'm interested to see how the actual writing process goes. Chatgpt is okay at creative writing, but not at the level that we expect to see. Some faculty expressed concerns that students would just have the software do the work for them, but I'm not really bothered by that. First off, if someone can figure out how to make a career out of publishing chatgpt prompts, well, good for them. Second, if the software is able to write better than the students are, they don't really belong in this class in particular (graduate level writing class).

Anyway, like I said earlier in the thread - we're really just treating this as another 'person' in the class, but who is available to everyone, all the time. It's beneficial for the brainstorming sections, but not for the hard skills, from what I can tell. I am most excited to watch them evaluate their own thought processes when working with chatgpt to flush out their ideas. In the past, this has been a barrier, because every single person I’ve ever met never gives a partner 100% of their thoughts all the time. They always hold back. My theory is that, because it’s a dead-end conversation, they will be more willing to push into new topics, and really think hard about what they’re trying to say. We’ll see if that stands.

Re: Big data is dead

#418
post #105

Earlier quoted context omitted.

I've observed the same in manufacturing … and fitness trackers a la FitBit. There's initial value from training yourself on what something looks/feels like … but diminishing returns after that. Whether there is more value to be found doesn't seem to matter. Factories would sensor up, go nuts with data, find one or two major insight, tire of data, and then just continue operating how they were before … but with a few…

Where as a Garmin is a useful ongoing tool for me a runner. I have got through about six watches. I don't use gimmicks like step counters though, turned it off. "Congratulations you completed 10,000 steps!" - well duh, I ran 10 miles this morning.

if youre running 10 miles per morning, even half that, on the regular you're an extreme outlier. the vast, vast majority of folks don't get anywhere near that, and a step counter that gamifies making them move is a good thing.

Re: Big data is dead

#419

Earlier quoted context omitted.

You only need enough to flag suspect content, and then the teacher calls the student in for a quick oral exam - the fakers will flounder, the reals will pass.

Even if you understand the material well, doing assigned tasks takes a lot of time. Especially if it's free form text. And "I want to do something more interesting right now" is at least as powerful a motivator to cheat assignments as "I don't know how to do this".

Aye, if I mostly know a subject why bother to take the time to write 10 pages when I can outsource that to a bot, and if they question me I can do a 20 min oral interview and save myself the trouble?

seems like a no-brainer in the long term.

Re: Big data is dead

#420

Earlier quoted context omitted.

Is it crazy to think that instead of stepping up in the war against AI we instead try to figure out a way to teach kids assuming they will use AIs?

Are we trying to produce adults who are able to think critically and creatively, and who reach their full intellectual potential, or are we trying to produce adults who can push a few buttons and blindly believe what the machine tells them?

The short story MANNA about AI directed folks and the future that creates comes to mind:

https://marshallbrain.com/manna1

Post reply on HN