Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What it feels like to work in AI right now

robotic.substack.com

Post reply on HN
Print view View original

731–740 of 768 posts

Re: What it feels like to work in AI right now

#731

Post by zarzavat »

nullsense wrote: post #709
ChatGTP wrote: post #553

Earlier quoted context omitted.

What actually is the trajectory we're on and what will we do once there?

So, it seems the trajectory is one of increasing generality and capability of models and increasing reliance on them. If it's at all possible to improve our technology then we will. If we improve it it increases in utility. If it increases in utility we use it more. What other thesis is there?

The model architecture has stayed roughly the same since the original AIAYN transformer in 2017. That’s 6 years of nothing fundamental happening.

Now, obviously the models have got hugely better in capabilities since BERT. Everything else has advanced. Tweaking, tuning and scaling have delivered true intelligence, albeit sub-human. But it seems unlikely that transformers are what take us to human-parity AGI and beyond, because the more we optimize these word predictors the more we find their limitations.

The lack of architecture changes over the last 6 years creates a huge amount of “potential energy”. A new model architecture might well push us over the human-parity threshold. It wouldn’t surprise me if I wake up one day to find that transformers are obsolete and Google has trained a human-parity AGI with a new arch.

This could happen tomorrow or in 20 years, transformers had an easy discovery path from RNNs, to RNNs with attention mechanisms, to Transformers. Architecture X seems to have a much more obscure discovery path.

Re: What it feels like to work in AI right now

#732

Post by zone411 »

Analog24 wrote: post #684
zone411 wrote: post #677

Earlier quoted context omitted.

No, I have no plans to openly publish any of it. Some of my researcher employees have published their own stuff. I've previously written about how it was a huge commercial mistake for Google and others to openly publish their research, and they should stop. Indeed, now OpenAI has not published a meaningful GPT-4 paper, and DeepMind has also become more cautious. This mistake has cost them billions, and for what? Recr…

The fact that you think creating a writing assistant plugged into Word is equivalent to building a general purpose, always-on voice assistant tells me all I need to know.

What? We were talking about making a language model. I mentioned the plugins in relation to the question of commercializing. I'm very clear about what my project was and was not doing. I get that you're bitter because Alexa became a joke with how little progress was made and the struggles of Amazon in getting the top talent are well known. How much did it cost Amazon to fail like this? Is the whole team gone? Is that the billions you mentioned?

Re: What it feels like to work in AI right now

#733

Post by teaearlgraycold »

throwaway2037 wrote: post #444
teaearlgraycold wrote: post #372

Earlier quoted context omitted.

For me it’s a perfect replacement for Stack Overflow. Except every solution is tailored to my exact code and situation. I’ve even gotten it to walk me through things like installing WSL 2 without using the Microsoft Store after I nuked all Appx packages. Maybe my favorite was pasting in pages of documentation describing all of the error codes for a library (the docs are the only source of truth) and getting it to out…

Please write a blog post about this! That is an amazing story.

https://news.ycombinator.com/item?id=35489587

Re: What it feels like to work in AI right now

#734

Post by sashank_1509 »

sealeck wrote: post #725
sashank_1509 wrote: post #594

Earlier quoted context omitted.

There’s too much insanity over building new tech these days, and I find a lot of hate directed on Autopilot, to stem from hate directed on Elon. Let’s go over the basics for Autopilot: 1. It costs thousands of dollars a year + 15k one time and it’s very easy to get banned from autopilot for life. They have a 3 strike rule + 1000 miles driven on your Tesla with a good safety before you’re allowed to access autopilot 2…

A cynical person might remark upon the fact that your being a Tesla employee might have some bearing on your position, but I am not such a person. > In essence, this is not tech that is being used by regular people who have a chance of misusing it. Now this is true in one sense – people who can't afford a Tesla and then aren't willing to spent an additional $15k on a piece of software, which has (many, many, many) ti…

PS., I have left Tesla but sure I might be biased since I have friends there and worked there for a while.

> I could be wrong (this is a genuine statement, please don't take it as a passive aggressive one, it's not intended that way) but doesn't this rely on Tesla first finding a failure, then diagnosing a symptom, writing a fix, etc. The fact is though that this initial failure might be one of several crashes which have occured in a Tesla on AutoPilot, which isn't great?

Failures are generally user disengagements not a crash. We measure user disengagements, classify them and try to drive the egregious ones to zero. FSD has had one major crash, no injuries that is being investigated by NHTSA, and a few minor bumps (I went in more detail below).

> what if someone else's Tesla crashes into me_?

I think that is a very fair point. It happened when a Uber self driving car crashed and killed a pedestrian which was a major incident in this industry. The problem with DL models is they are unexplainable and we cannot tell when they fail (Though in Uber case it was not exactly DL model failing). Tesla took this risk and has managed fine with no injuries to date. And now the main reason I made this post, the tech keeps getting better, we have this model from Meta that just literally segments everything in an image (even ones you take from your phone). It honestly feels we are leaving the risky DL territory and reaching the "we can't understand how but it just works" territory where you can rely on a Deep Learning to do what you expect it to do.

Re: What it feels like to work in AI right now

#735

Post by Analog24 »

zone411 wrote: post #732
Analog24 wrote: post #684

Earlier quoted context omitted.

The fact that you think creating a writing assistant plugged into Word is equivalent to building a general purpose, always-on voice assistant tells me all I need to know.

What? We were talking about making a language model. I mentioned the plugins in relation to the question of commercializing. I'm very clear about what my project was and was not doing. I get that you're bitter because Alexa became a joke with how little progress was made and the struggles of Amazon in getting the top talent are well known. How much did it cost Amazon to fail like this? Is the whole team gone? Is that…

"It's six years later and Siri, Alexa, and Google Home are still nearly as dumb as they were back then". You can't even keep a coherent discussion and you are delusional about the significance of your work. You shared a slide with nothing but generic pie-in-the-sky use-cases and you act like it gives you some credibility on the subject ("let's make an AI system that can do the work of your non-professional employees!"). And to top it off you act like you've been successful here! Again, you shared nothing but a slide with generic use-cases that a 12 year old could think up. I don't know what you think you proved. Enjoy your imaginary pedestal.

Re: What it feels like to work in AI right now

#736

Post by copperx »

jacquesm wrote: post #368
copperx wrote: post #357

Earlier quoted context omitted.

Understanding the code of a fully fledged application is hard. But with ChatGPT you can build it piece by piece.

With a brain you can also build it piece by piece, in fact I don't know of any other way of writing a large software system than doing it piece by piece.

Sure, but the argument was that reading an entire application's code is hard, therefore GPT-4 is counterproductive.

Re: What it feels like to work in AI right now

#737

Post by girvo »

SanderNL wrote: post #606
girvo wrote: post #491

Earlier quoted context omitted.

How do you deal with it straight up lying? My problem with this whole system is, if I’m asking those questions it’s because I don’t understand the field well enough to answer it myself, which means I can’t pick up on if ChatGPT is lying…

Fair, but not completely true. The Thailand examples gives a detailed reasoning. You can use those building blocks to check. If it says Thailand is a cold country and uses that in its argumentation, it's shaky. You don't have to be an expert climatologist to make this judgement. It's not just one clean answer and we're done. In my experience it is helpful in breaking the problem down into stuff you can Google.

> In my experience it is helpful in breaking the problem down into stuff you can Google.

Yeah I can see that being useful. I’ve also seen a lot of non-technical people straight up accept whatever comes out of it, so that’s a little worrying. It’s true of Google searches too, of course, but at least a google search gives N results someone can check rather than 1.

Re: What it feels like to work in AI right now

#738

Post by SanderNL »

int_19h wrote: post #685
SanderNL wrote: post #605

Earlier quoted context omitted.

The same reason why you don't let Mr Musk do all the work. He can't. One LLM is limited, one obvious limitation is its context window. Using a swarm of LLMs that each do a little task can alleviate that. We do it too and it's called delegation. Edit: BTW, "swarm" is meaningless with LLMs. It can be the same instance, but prompted differently each time.

Context window is a limitation, but have we actually hit the ceiling wrt scaling that? For GPT, you need O(N^2) VRAM to handle larger context sizes, but that is a "I need more hardware" problem ultimately; as I understand, the reason why they don't go higher is because of economic viability of it, not because it couldn't be done in principle. And there are many interesting hardware developments in the pipeline now th…

I am sure the context window can go up, maybe into the MB range. But I still see delegation as a necessary part of the solution.

For the same reason one genius human does not suddenly need less support staff, they actually need more.

Edit: and why it isn’t here yet is because it’s new and hard.

Re: What it feels like to work in AI right now

#739

Post by kolinko »

alexvoda wrote: post #557
kolinko wrote: post #452

Earlier quoted context omitted.

9 things is considered a standard for working memory (kind of like processor registers), for people with ADHD it's even less - 3-5. Try writing a number from one piece of paper to another. If it's more than 7-9 numbers, you won't do it in one shot, unless you spend extra time memorizing it.

The equivalent for computers would be L1 cache on the CPU which is tiny.

More like cpu registers I would say :)

Re: What it feels like to work in AI right now

#740

Post by kolinko »

JohnFen wrote: post #713
kolinko wrote: post #452

Earlier quoted context omitted.

9 things is considered a standard for working memory (kind of like processor registers), for people with ADHD it's even less - 3-5. Try writing a number from one piece of paper to another. If it's more than 7-9 numbers, you won't do it in one shot, unless you spend extra time memorizing it.

That can be increased quite a bit with practice. But it's also not important. It's just the cache memory -- it isn't the limit of what can be learned and recalled.

It is a limit on what you can reason about without a piece of paper.

I’m proficient at math, but my working memory is around 6, so I cannot add two three digit numbers to each other in my head (unless I see numbers to be added in front of me).

Post reply on HN