Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees

cnbc.com

Post reply on HN
Print view View original

71–80 of 348 posts

Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees

#71

Post by varispeed »

I noticed that I pretty much stopped using Google for coding queries and spend most of the time with ChatGPT. It's so helpful getting information to the point so you don't have to browse through dozens of spam sites etc.

So for instance I can say. I have such and such file and I need this and that extracted and plotted on a graph. Then I see the graph and I can tell it - discard values above this and that threshold and calculate standard deviation and median.

Together with copilot, it's quite neat. I am excited how it gets developed.

It's really boring spending time finding how to code something in this and that library. I'd rather tell the machine to code me this and that and spend my time in more useful way.

ChatGPT helps get rid of a lot of "busy" unnecessary work.

Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees

#72

Post by brhsagain »

6c737133 wrote: post #2

why does chatGPT passing tests or interviews continue to make headlines? all they're proving is that tests, and interviews, are bullshit constructs that merely attempt to evaluate someone's ability to retain and regurgitate information

Sure, but in our present reality those tests and interviews are how we currently gatekeep upper middle class jobs, so it is at least of some practical interest.

Also, I think this is a bit overstated. Programmers (and smart people in general) like to think that their real job is high level system design or something, and that "mere regurgitation" is somehow the work of lesser craftsmen. When in reality what GPT shows is that high-dimensional regurgitation actually gets you a good fraction of the way down the road of understanding (or at least prediction). If there is a "buried lede" here it's that human intelligence is less impressive than we think.

Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees

#73

Post by nzoschke »

22SAS wrote: post #5

So, a LLM, trained extensively on StackOverflow and other data (possibly the plethora of LC solutions out there), is fed a bunch of LC questions and spits out the correct solutions? In other news, water is blue. It is one thing to train an AI on megatons of data, for questions which have solutions. The day ChatGPT can build a highly scalable system from scratch, or an ultra-low latency trading system that beats the c…

Agree LeetCode is one of the least surprising starting points.

Any human that reads the LeetCode books and practices and remembers the fundamentals will pass a LeetCode test.

But there is also a ton of code out there for highly scalable client/servers, low latency processing, performance optimizations and bug fixing. Certainly GPT it is being trained on this too.

“Find a kernel bug from first principles” maybe not, but analyze a file and suggest potential bugs and fixes and other optimizations absolutely. Particularly when you chain it into a compiler and test suite.

Even the best human engineers will look at the code in front of them, consult Google and SO and papers and books and try many things iteratively until a solution works.

GPT speedruns this.

Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees

#74

Post by scandum »

22SAS wrote: post #5

So, a LLM, trained extensively on StackOverflow and other data (possibly the plethora of LC solutions out there), is fed a bunch of LC questions and spits out the correct solutions? In other news, water is blue. It is one thing to train an AI on megatons of data, for questions which have solutions. The day ChatGPT can build a highly scalable system from scratch, or an ultra-low latency trading system that beats the c…

I've been most impressed with ChatGPT's ability to analyze source code.

It may be able to tell you what a compiled binary does, find flaws in source code, etc. Of course it would be quite idiotic in many respects.

It also appears ChatGPT is trainable, but it is a bit like a gullible child, and has no real sense of perspective.

I also see utility as a search engine, or alternative to Wikipedia, where you could debate with ChatGPT if you disagree with something to have it make improvements.

Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees

#75

Post by 22SAS »

ericmcer wrote: post #63
echelon wrote: post #35

Earlier quoted context omitted.

> The rest of us aren't going to care that much. If you don't adapt, you'll be out of a job in ten years. Maybe sooner. Or maybe your salary will drop to $50k/yr because anyone will be able to glue together engineering modules. I say this as an engineer that solved "hard problems" like building distributed, high throughput, active/active systems; bespoke consensus protocols; real time optics and photogrammetry; etc.…

You really think in <10 years AI will be able to take a loose problem like: "our file uploader is slow" and write code that fixes the issue in a way that doesn't compromise maintainability? And be trustworthy enough to do it 100% of the time?

My point exactly.

If I interpret OP's statement correctly, that chatGPT can build complex systems from scratch in 10 years. Then according to that statement, the only adaptation is to choose a new career because it has made almost all SWE jobs go the way of the dinosaurs.

Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees

#76

Post by phoehne »

squarefoot wrote: post #55
22SAS wrote: post #5

So, a LLM, trained extensively on StackOverflow and other data (possibly the plethora of LC solutions out there), is fed a bunch of LC questions and spits out the correct solutions? In other news, water is blue. It is one thing to train an AI on megatons of data, for questions which have solutions. The day ChatGPT can build a highly scalable system from scratch, or an ultra-low latency trading system that beats the c…

To me the real advancement isn't the amount of data it can be trained with, but the way it can correlate them and choose from, according to the questions it's being asked. The first is culture, the second intelligence, or a good approximation of it. Which doesn't mean it could perform the job; that probably means the tests are flawed.

It doesn’t really have a model for choosing. It’s closer to pattern matching. Essentially the pattern is encoded in the training of the networks. So your query most closely matches the stuff about X, where there’s a lot of good quality training data for X. If you want Y, which is novel or rarely used, the quality of the answers varies.

Not to say they’re nothing more than pattern matching. It’s also synthesizing the output, but it’s based on something akin to the most likely surrounding text. It’s still incredibly impressive and useful, but it’s not really making any kind of decision any more than a parrot makes a decision when it repeats human speech.

Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees

#77

Post by pixl97 »

morelisp wrote: post #36
mensetmanusman wrote: post #18

Earlier quoted context omitted.

Don’t understand this take. If it was easy to make an LLM that quickly parsed all of StackOverflow and described new answers that most of the time worked in the timeframe of an interview, it would have been done by now. ChatGPT is clearly disruptive being the first useful chatbot in forever.

While I think the jury is still out on whether ChatGPT is truly useful or not, passing an L3 hiring test is not evidence of that one way or another.

If it doesn't point out that ChatGPT is useful, especially if its proven it is not, then maybe the hiring tests are not useful.

Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees

#78

Post by dpedu »

My experience with asking ChatGPT to write code is that is produces code that LOOKS like it will work and solve the question asked but it actually doesn't. For example, I've asked it to create code examples of how to use different features in some Python libraries. The samples it produces make me think "ok, that's exactly how I would expect X feature in this library to work", but upon more a detailed inspection, I find that it references library methods that don't even exist! You can complain to the bot that it is lying and it will apologize and spit out another sample with calls to nonexistent methods.

This experience makes me question if something else is going on here. It's easy to overlook a mistake in a whiteboard coding exercise.

Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees

#79

Post by Bukhmanizer »

tbalsam wrote: post #16
22SAS wrote: post #5

So, a LLM, trained extensively on StackOverflow and other data (possibly the plethora of LC solutions out there), is fed a bunch of LC questions and spits out the correct solutions? In other news, water is blue. It is one thing to train an AI on megatons of data, for questions which have solutions. The day ChatGPT can build a highly scalable system from scratch, or an ultra-low latency trading system that beats the c…

If it helps, this likely is coming. I think we have a tendency to mentally move the goalposts when it comes to this kind of thing as a self-defense mechanism. Years ago this would have been a similar level of impossibility. Since all a codebase like that is is a kind of directed graph, then augmentations to the processing of the network to allow for the simultaneous parsing of and generation of this kind of code may…

I agree this would have been thought to be impossible a few years ago, but I don't think it's necessarily moving the goalposts. I don't think software engineers are really paid for their labour exactly. FAANG is willing to pay top dollar for employees, because that's how they retain dominance over their markets.

Now you could say that LLMs enable Google to do what it does now with fewer employees, but the same thing is true for every other competitor to Google. So the question is how will Google try and maintain dominance over it's competitors now? Likely they will invest more heavily in AI and probably make some riskier decisions but I don't see them suddenly trying to cheap out on talent.

I also think that it's not a zero sum game. The way that technology development has typically gone is the more you can deliver, the more people want. We've made vast improvements in efficiency and it's entirely possible that what an entire team's worth of people was doing in 2005 could be managed by a single person today. But technology has expanded so much since then that you need more and more people just to keep up pace.

Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees

#80

Post by nzoschke »

ipnon wrote: post #20
22SAS wrote: post #5

So, a LLM, trained extensively on StackOverflow and other data (possibly the plethora of LC solutions out there), is fed a bunch of LC questions and spits out the correct solutions? In other news, water is blue. It is one thing to train an AI on megatons of data, for questions which have solutions. The day ChatGPT can build a highly scalable system from scratch, or an ultra-low latency trading system that beats the c…

Until ChatGPT can slack my PM, attend my sprint plannings, read my Jira tickets, and synthesize all of this into actionable tasks on my codebase, I think we have job security. To be clear, we are starting to see this capability on the horizon.

Perhaps an engineering manager can use one trained on entire Slack history, all Jira tickets, and all PRs to stub out some new tickets and even first PR drafts themselves…

We will always need humans to prompt, prioritize, review, ship and support things.

But maybe far less of them for many domains. Support and marketing are coming first, but I don’t think software development is exempt.

Post reply on HN