While I get that they're sometimes useful to trigger debate, I don't really subscribe to very bold statements. We are drowning in data, it's all around us. Information overload is real. Data enables most of our daily digital experiences, from operational data to insights in the form of user facing analytics. Data systems are the backbone of the digital life. It's is an ocean and it's all about the vessel you pick to…
Big data is dead
411–420 of 444 posts
Re: Big data is dead
#412Earlier quoted context omitted.
As a consultant with roots in backend dev, I fully understand the scrutiny that we receive because unfortunately, it is often very warranted... It feels a bit refreshing to read your comment and see someone articulate what I am trying to convey to my clients. I am a tool, and yes, this pun is intended.
Hello I want to make the move from development into consultancy and would appreciate hearing how you did it. I cannot DM you, but if you have the time to type a small paragraph, I'm all ears. If you don't want to make that public, you can directly email me ( @gmail.com)
It was a LinkedIn recruiter message which I usually ignore. However, my SO did not (she is in IT as well) and convinced me to join a hiring event. I ended up liking it a lot and went through the hiring process. Soon, I started out on the most junior level and joined my first project with 3 very senior colleagues after a few weeks.
The learning curve was very steep both on the technical level and also regarding the consulting aspect - at first there was nothing I could 'consult' on due to lack of experience. This changed with growing experience, with the guidance of senior colleagues and my private efforts to gain skills and expertise.
Let me know if this was helpful!
Re: Big data is dead
#413SSDs we’re limited in capacity and still expensive.
Parallelizing work with MapReduce allowed using cheap fault-prone commodity hardware and disks.
If you’re dealing with terabytes rather than petabytes of data, you probably don’t need BigData
Re: Big data is dead
#414Earlier quoted context omitted.
One may follow the other, but not vice versus. It's a pretty strong argument to say that Microsoft under Gates was technically obsessed, but that really faltered under Balmer. Microsoft continued to win profits, but they made major strategic missteps that cost the revenue. Amazon feels like it's going down the same path: empowering the tree-gazers without remembering that the forest also matters.
You can also “coast upwards” for quite awhile - as an example we now pay Microsoft less than we were for periodic upgrades to Office for the entire Microsoft 365 suite (including email hosting, etc) but all the machines are now Macs. They make more from us in one way, but less in total dollars.
{Revenue attributable to previous R&D} (aka coast) vs {Revenue attributable to current R&D} (aka acceleration)
Re: Big data is dead
#415Earlier quoted context omitted.
I used to joke that Data Scientists exist not to uncover insights or provide analysis, but merely to provide factoids that confirm senior management's prior beliefs. I did several experiments, and noticed that whenever I produced analysis that was in line with what management expected - my analysis was praised and widely disseminated. Nobody would even question data completeness, quality, whatever. They would pick so…
> Data Scientist's job is to launder management's intuition using quantitative methods Ouch. This is savage, but sadly correct in many cases. HOWEVER, to play devil's advocate here, I've also seen corporate data scientists overstate the conclusions / generalizability of their analysis. I've also seen data scientists fall prey to believing that their analysis proves would should be done, rather than what is likely to…
Re: Big data is dead
#416Earlier quoted context omitted.
Hello I want to make the move from development into consultancy and would appreciate hearing how you did it. I cannot DM you, but if you have the time to type a small paragraph, I'm all ears. If you don't want to make that public, you can directly email me ( @gmail.com)
Sure, it is actually not very complicated in my case. I did backend development for a short while during and after university and then moved into IT consulting fairly quickly. It was a LinkedIn recruiter message which I usually ignore. However, my SO did not (she is in IT as well) and convinced me to join a hiring event. I ended up liking it a lot and went through the hiring process. Soon, I started out on the most j…
Thanks a lot!
Re: Big data is dead
#417Earlier quoted context omitted.
Again, stop putting words in my mouth, please. I never said they became experts. I said they integrated it into existing processes. I said that they were doing something other than just scanning for chatgpt hits like plagiarism checkers. >It's part of the collaborative idea building and development process now for every student enrolled in creative writing and writing analysis classes >It's integrated into existing a…
Not who you are replying to. I found your example fascinating - would you be able to share one or two concrete examples of this integration you mentioned? I would like a low-level peek or two into how the teaching landscape is changing in light of the rise of LLMs like chatGPT.
In the past, the class would be centered around ideas and themes the class came up with together during the first week of the semester. They would then read and discuss short stories from various authors centered around that theme, preferably from different eras and/or cultures. From there, they would work in pairs/small groups to flush out original ideas they came up with for their own stories based on the themes and styles from the first month of reading. Then they would work individually to write the stories. Finally, they would come back together to edit and work through that process as a group, with a reading and discussion in the last week or two.
Now, the class works alone the first day or two to discuss themes with chatgpt, to identify relevant and appropriate literature (if possible), and to flush out initial ideas for the readings and discussion topics. Then we come back together for group work to figure out themes for the semester and possible readings, like it used to be, but primed with whatever information they had already discussed with Chatgpt. From there they work through the readings, using chatgpt to bounce ideas for discussion off of before coming to class. Then they work in teams, like before, to flush out their own stories, but again, using chatgpt as if it were another member of the group. They also have to track their question/comment and response, to evaluate their own thought processes and look for weak links in their reasoning and logic. Students are then free to use the software to edit their writing before coming together with their creative assignments.
We haven't made it past the step of reading, as it's still the first month of the semester. But,the discussion for the first section has been much more in-depth and (to use a word that is impossible to quantify) vibrant. The students had already aired the ideas they thought may be dumb, and would therefore be less willing to voice in a public setting; this allows them to really dive further into whatever is in each story, and connect dots between stories that generally took a week or two. Because they have another 'person' to talk to whenever they want, and however much they want, they tend to really get into the work. Further, and unexpectedly (and anecdotally sadly) it has allowed a couple of students I know personally who would not have been willing to participate in public discussion (anxiety disorder and TBI) a stronger voice, because they know their ideas are flushed out already, and it provides them with a 'script' of ideas they know are novel and valuable for the class. In other words, if they were able to get the idea from Chatgpt, they knew they had more work to do to build that thought, because that is really just a baseline.
I'm interested to see how the actual writing process goes. Chatgpt is okay at creative writing, but not at the level that we expect to see. Some faculty expressed concerns that students would just have the software do the work for them, but I'm not really bothered by that. First off, if someone can figure out how to make a career out of publishing chatgpt prompts, well, good for them. Second, if the software is able to write better than the students are, they don't really belong in this class in particular (graduate level writing class).
Anyway, like I said earlier in the thread - we're really just treating this as another 'person' in the class, but who is available to everyone, all the time. It's beneficial for the brainstorming sections, but not for the hard skills, from what I can tell. I am most excited to watch them evaluate their own thought processes when working with chatgpt to flush out their ideas. In the past, this has been a barrier, because every single person I’ve ever met never gives a partner 100% of their thoughts all the time. They always hold back. My theory is that, because it’s a dead-end conversation, they will be more willing to push into new topics, and really think hard about what they’re trying to say. We’ll see if that stands.
Re: Big data is dead
#418Earlier quoted context omitted.
I've observed the same in manufacturing … and fitness trackers a la FitBit. There's initial value from training yourself on what something looks/feels like … but diminishing returns after that. Whether there is more value to be found doesn't seem to matter. Factories would sensor up, go nuts with data, find one or two major insight, tire of data, and then just continue operating how they were before … but with a few…
Where as a Garmin is a useful ongoing tool for me a runner. I have got through about six watches. I don't use gimmicks like step counters though, turned it off. "Congratulations you completed 10,000 steps!" - well duh, I ran 10 miles this morning.
Re: Big data is dead
#419Earlier quoted context omitted.
You only need enough to flag suspect content, and then the teacher calls the student in for a quick oral exam - the fakers will flounder, the reals will pass.
Even if you understand the material well, doing assigned tasks takes a lot of time. Especially if it's free form text. And "I want to do something more interesting right now" is at least as powerful a motivator to cheat assignments as "I don't know how to do this".
seems like a no-brainer in the long term.
Re: Big data is dead
#420Earlier quoted context omitted.
Is it crazy to think that instead of stepping up in the war against AI we instead try to figure out a way to teach kids assuming they will use AIs?
Are we trying to produce adults who are able to think critically and creatively, and who reach their full intellectual potential, or are we trying to produce adults who can push a few buttons and blindly believe what the machine tells them?