This puts the stake in the ground that coding interviews today are more about memorization than testing for understanding.
I felt like this was the whole premise of the book Cracking the Coding Interview though I only read the first bit of the book.
Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees
321–330 of 348 posts
Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees
#322Earlier quoted context omitted.
Ah the HN echo chamber again! Please visit your local non FAAAM (or what it is now?) fortune 1000, pick a senior dev randomly and work with them for week. Chatgpt is vastly better now , today. Faster, does not need sleep, rest, politeness or handholding, can explain itself (sure it’s wrong often but less wrong than the dev you picked while actually being able to use proper syntax and grammar, unlike the dev you picke…
Doesn’t this only work for relatively contrived situations? I can tell a jr dev to go and add some minor feature in a codebase, put it behind a flag, and add tracking/analytics to it. I can point to the part of the application I want the feature to be added on the screen and the jr devs are often able to find it on their own. I haven’t seen chatGPT do anything like that and I don’t think there is a way to provide it…
For me it works for small stand alone utility scripts. But the most impressive thing I was able to get it to do was.
“Given an XML file with the format {[1]} and a DynamoDB table with two fields “Key”, “Value”, write a Python script that replaces the Value in the xml file when the corresponding key is found. Use argparse to let me specify both the input xml file and the output XML”
It spit out perfect Python code. I hadn’t used XML in well over a decade and I definitely didn’t know how to read xml in Python. I didn’t want to bother about learning.
I actually pasted an XML sample like the link below.
[1] https://learn.microsoft.com/en-us/troubleshoot/developer/vis...
Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees
#323Earlier quoted context omitted.
Ah, the good ol "A(G)I will arrive in 10 years!" --For the past 50+ years, basically. It's a cautionary tale to people who are working in ML to be not too optimistic on "the future", but in my opinion being cautiously optimistic(not on AGI though) isn't harmful by itself, and I stand by that. Well at least until we hit the next wall and plunge everyone into another AI winter(fourth? fifth?) again. As a plus, we do ac…
Kurzweil gets a lot of flack for this sort of thing, he's generally presented as the ridiculous hype man for AI. And yet, he bet in 2002 that an AI would pass the Turing test by 2029. (And this is actually a more conservative prediction than "we will have AGI by 2029.") And looking at GPT3 it seems like he is probably going to win that bet.
Good point, I do appreciate this comment. Thanks for adding this. It is is interesting in how it very much appears that he will be correct, but instead in a different way maybe than most of us would reasonably have guessed at the time.
Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees
#324why does chatGPT passing tests or interviews continue to make headlines? all they're proving is that tests, and interviews, are bullshit constructs that merely attempt to evaluate someone's ability to retain and regurgitate information
You're not wrong, but until you actually play with ChatGPT yourself, you just don't understand how _dumb_ it is. All people see is the cheating, and possibly this scary new AI that's going to get smarter than humans in a short period of time. I suspect the best way to educate people on both the powers and limits of the technology is to get them to sit down for 15 minutes with it.
Its ability to correct itself is very impressive when given the feedback. I imagine with the proper feedback loop it can advance very fast. E.g., when asked to write a piece of html markup, if it could "see" how the rendered layout is different from what was asked for, it could adjust its solution without human involvement. If it could run the deployment script and see where it fails, it could apply all the fixes itself until it works. If it could run the unit tests and see where its solution breaks the other parts of the system, it would need much less handholding.
Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees
#325Earlier quoted context omitted.
Reading that I can only assume you’ve made really poor use of Jr developers. Which ok I get why you think ChatGPT is more useful.
You really think that Jr devs could crank out the same code faster than ChatGPT? I couldn’t crank out the same code and you couldn’t either. The most you can hope from JR devs (even the ones I have met at BigTech) is that they don’t eat the chalk during the first 3-6 months.
Always hire a senior developer without experience for junior role. By that I mean hire a developer who knows how to program but lacks specific experience or has no formal experience at all.
Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees
#326Earlier quoted context omitted.
If it helps, this likely is coming. I think we have a tendency to mentally move the goalposts when it comes to this kind of thing as a self-defense mechanism. Years ago this would have been a similar level of impossibility. Since all a codebase like that is is a kind of directed graph, then augmentations to the processing of the network to allow for the simultaneous parsing of and generation of this kind of code may…
I don't think ChatGPT or its successors will be able to do large-scale software development, defined as 'translating complex business requirements into code', but the actual act of programming will become more one of using ML tools to create functions, and writing code to link them together with business logic. It'll still be programming, but it will just start at a higher level, and a single programmer will be vastl…
I could see IDEs for AI, where you manipulate ways to input prompts (natural Landis language, weighted keywords, audio..) and selection of methods (chatgpt, whatever model will come for diagrams, visual models, audio ones..). Then basically visually program outputs, add tests you want to use to validate and feed back, multimodal output views..
Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees
#327Earlier quoted context omitted.
I see a lot of criticism towards modern software engineer interviewing techniques, but never a solution offered that would catch and reject BS. Your comment comes off as "hire a person because you get along with them, don't worry if they can't write a function that accomplishes a simple task".
In the google case, the interview should be considering that the best candidates will be using chat ai to augment their work, and assign harder tasks and allow use of the ai. Somebody who can't use the chat tools no longer meets the bar
Harder tasks are not like "Generate a code for an express API and add a user endpoint". Harder tasks would be "A stupid bug that sometimes happens when a user clicks a button in a funny way."
ChatGPT isn't an artificial general intelligence. You can't tell it about a bug and expect it to 1) understand it, 2) come up with a solution.
So you have to actually know what you're doing.
Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees
#328So, a LLM, trained extensively on StackOverflow and other data (possibly the plethora of LC solutions out there), is fed a bunch of LC questions and spits out the correct solutions? In other news, water is blue. It is one thing to train an AI on megatons of data, for questions which have solutions. The day ChatGPT can build a highly scalable system from scratch, or an ultra-low latency trading system that beats the c…
> The day ChatGPT can build a highly scalable system from scratch, or an ultra-low latency trading system that beats the competition, or find bugs in the Linux kernel and solve them; then I will worry.
The bar for “then I will worry!” when talking about AI is getting hilarious. You’re now expecting an AI to do things that can take highly skilled engineers decades to learn or require outright a large team to execute?
Remind me where the people who years ago were saying “when an AI will respond in natural language to anything I ask it then I will worry” are now.
Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees
#329I think this says more about the Google interview process than about ChatGPT. That a machine learning model can "bullshit" its way through an interview that is heavily leaning on recall of memorized techniques, algorithms and "stock" problems that have solutions (of various quality) all over Internet is not exactly surprising. Machines will always be able to "cram" better than humans. In practice these questions are…
I see a lot of criticism towards modern software engineer interviewing techniques, but never a solution offered that would catch and reject BS. Your comment comes off as "hire a person because you get along with them, don't worry if they can't write a function that accomplishes a simple task".
You seem to imply that the interview process actually works in the sense that it rejects bad candidates and selects good candidates. There's actually very little evidence for that. And if you look at companies like Google, they obviously have issues with hiring lots of people that aren't getting a whole lot done. Case in point: OpenAI. That company has been operating completely in the open for years. And yet Google got caught by surprise. Why is that? The company collectively lacks imagination and leadership. They've self selected out of hiring people that have those traits.
In my experience, companies using this style of interviewing are actually incapable through process of hiring the type of people that are qualified and experienced enough to know that this process is bullshit. I.e. the type of people that have 0 need to drop down on their knees and beg for the job. Leaders, not followers. It's a problem. If you want to hire the best, insulting them with a silly coding interview is not a great way to do it. Companies like this self select into hiring people that at best are as good as what they already have. It's the old A's hire A's, B's hire C's kind of thing.
The solution is to trust your people more to take good decisions rather than allowing them to defer to some HR process. The process at the startup I run is very simple. We don't subject people to coding interviews. If you pass our initial filters (CV screen and common sense), you first talk to somebody senior enough to make a good judgment call. Anyone recommended by anyone we care about gets priority. We trust our people to have good judgment. Big companies hide behind process because they don't trust their people to have good judgment and/or their people don't want to take the responsibility for having good judgment. Both are bad. I don't want such people in my company. It works. We get some amazing people walking in through the front door that are actually excited about working for us.
Re: Google testing ChatGPT-like chatbot 'Apprentice Bard' with employees
#330So, a LLM, trained extensively on StackOverflow and other data (possibly the plethora of LC solutions out there), is fed a bunch of LC questions and spits out the correct solutions? In other news, water is blue. It is one thing to train an AI on megatons of data, for questions which have solutions. The day ChatGPT can build a highly scalable system from scratch, or an ultra-low latency trading system that beats the c…
If it helps, this likely is coming. I think we have a tendency to mentally move the goalposts when it comes to this kind of thing as a self-defense mechanism. Years ago this would have been a similar level of impossibility. Since all a codebase like that is is a kind of directed graph, then augmentations to the processing of the network to allow for the simultaneous parsing of and generation of this kind of code may…
[deleted]