Earlier quoted context omitted.
>What we're seeing is the first instance, still very limited and imperfect, of AGI absolutely not. language-model text generation is actually about as non-general as it gets -- they are fundamentally incapable of understanding anything at all, ever. they can't do math, work through basic logic problems, or produce any output that isn't just an assumed logical continuation of the input.
People seem so convinced of this and I just don't get it. I'm seeing this comment through my eyeballs, generating some pertinent text in my brain, and outputting it back out. But so many people seem convinced convinced this process is something radically, fundamentally, irreducibly different than what ChatGPT is doing internally, and I don't get why. Is it because I have a consciousness with an internal narrative and…
CheatGPT
461–470 of 544 posts
Re: CheatGPT
#462Earlier quoted context omitted.
> Will this filter cut many of the best engineers? It won't cut out a single good engineer, let alone the best. > I'd assert that the engineer who doesn't assume an unspoken implication, but instead qualifies their answer, or tells you when they cannot answer, or asks for clarification... is more likely to be the one who can make a system that works, and tell you when a system will not work. You grouped the one optio…
You seem to be emphasizing this, and I'd like to understand why. Have you seen an engineer that "shut down" on some problem, and you believe it was due to the kind of situation for which you're now trying to test?
But specific to the "shutting down because the requirements weren't 100% totally perfect" I see it all the time, and it's even what we're seeing people attribute to Google's slow decline
On one hand many hardcore engineers think we're seeing the slow and steady decline of software because of bootcamp kiddies ready to hack together any mess with a ball of Leftpad inspired libraries.
But on the other, so so many engineers struggle to see past the tip of their nose in larger organizations. There's this antagonistic co-existence with those outside of engineering where little effort is put into disseminating requirements if they don't agree with them to start.
Which ironically we're watching unfold here! People jumped to the conclusion the interviewer is in fact asking you to select from "every bug ever", but in doing so refuse to interpret that the interviewer might be asking "things you remember"... because that would be jumping to conclusions?
-
For example: when estimating how long tasks take and finding that there's a disconnect between what the larger org expected and what an engineer produced, there's rarely any deep inclination of many otherwise brilliant engineers to find out why because it's assumed "non-engineers just don't know."
They might try to shave some time here or there, they might try and bake in some crunch time because they seem themselves as being that brilliant and dedicated that they can make it work.
But rarely will they try discarding the notion that there was a disconnect on the non-engineering side, and self-directedly throwing out their entire proposed solution to try something that fits on the assumption that their solution was what was wrong in the equation.
Because when they made the design: they designed it with all of their intelligence and skill and experience. And that's what they were hired for, to make brilliant things. So why should they cheapen all that? If that's what management wants they should go hire some junior devs or something.
And unfortunately, if the reality really is that majority of the business value could be produced with orders of magnitude less effort, it's the engineering side that has to enable that kind of discovery. The engineering side is source of the plays in the playbook.
-
The reality is not every engineer can ever reach that. There are brilliant brilliant people who will never have the communication skills or the inclination, or the patience for any of this, and a good interview process doesn't require 1 person to ace every single signal.
Also some people will jump at me for implying engineers should need to zoom out, because in their minds management should be enabling them to stay complete heads down writing code.. but to me that mentality is not generally compatible with being a top of field company for the long haul.
Yes you might catch lightning in a bottle by just enabling very smart people to do build marvels in their silos, but business is more than having marvels to stare at.
I personally worked at a company that essentially succumbed to exactly this. A culture of exceptional engineering, hiring technically brilliant people at all costs... and dying a slow death because the engineers wouldn't leave room for business in their engineering.
-
I guess the tl;dr of all this is: A CEO will say "It's no use if we take 10 years to make a perfect product, if our competitor makes it to market with a decent product next year". And engineers will expect as much from business types.
But what they often forget is that the same is true for customers. No one benefits from your engineering if it never reaches the field. No one benefits from your answer if you willingly get stuck on every single speed bump.
Being a good engineer is being able to efficiently categorize which speed bumps are "just" bumps, and which ones are chasms that will swallow the ship whole if you don't change direction.
If the engineers at Boeing had the mentality that I see often in our field, each 727 would have cost a billion dollars, and would no one would fly today.
Re: CheatGPT
#463As a CS professor myself, I don't think ChatGPT really creates entirely new problems for assessment, but rather makes problems that we had been sweeping under the carpet more obvious and harder to ignore. Any assignment that can be cheated by using ChatGPT could already be cheated before by asking a friend, an expert or paying someone else to do it. But most teachers assumed this doesn't happen often, and thus acted…
As a first year grad student, I had a professor in material science who gave the same assignments and mostly same exams every year. Of course a) I didn't know this b) using previous works was prohibited c) cheating was rampant. I only found this out after getting heavily marked down on the first HW and joining a study group which was moderately chaste (eg only used the answers to check our work before turning it in).…
I tend to really appreciate 'open everything but people' exams, since they're a good way to actually check that people know how to complete a task. I think that a good one is probably harder for a professor to write, but the ones that I've taken have been a much better test of my understanding of the topic than ones where I was concerned with memorizing what the API for POSIX threads was.
Not so sure about infinite time, though. Seems like it makes it really easy for someone who has the time to dedicate a whole day (or the obsessiveness to do it even if they don't have the time) to do way better than someone who, for whatever reason, can't.
To me, the gold standard for computer science exams would be ~90m-3h with an open everything policy, including both a "do you know/can you apply the theory?" and a "can you do something practical with that?" section.
Re: CheatGPT
#464Earlier quoted context omitted.
the other week i gave chatgpt a simple multiplication problem that it got wrong. very simple problem like 86 * 0.0007 or something. but ive been working with chatgpt for 4-5 weeks now and that wrong answer doesnt make up for all the "good answers" that are usually not perfect. like one day i needed to COALESCE in mysql. i didnt know that, but chatgpt did. theres a few times i would have written a function the complic…
Multiplication problems are not language problems. There is no data in the training set where there is a likely probability of the next token in the "86 * 0.0007 =" sentence being correct. People need to stop treating ChatGPT as a computation engine. It is not wolfram alpha. It is not google. It is fancy autocomplete trained on a large subset of the internet.
Re: CheatGPT
#465Earlier quoted context omitted.
You seem to be emphasizing this, and I'd like to understand why. Have you seen an engineer that "shut down" on some problem, and you believe it was due to the kind of situation for which you're now trying to test?
Ironically the "shutting down" part isn't the real test because it's not a trick question: Discussing an interesting bug really is a great way to dive into technicals and learn about the person's soft skills and learn about their background amongst a bunch of other stuff. But specific to the "shutting down because the requirements weren't 100% totally perfect" I see it all the time, and it's even what we're seeing pe…
I've just been assuming that this kind of product/customer-driven engineering in a business environment can be learned, if it's not already known. And the only questions are whether the org can teach it (with culture, onboarding, consistent messaging) and whether the candidate would be happy with that.
If a candidate came to me with no product/commercial experience (e.g., recent grad, or from a research environment), I'd try to characterize the nature of the work, and see whether I could get an honest discussion with them about how they'd feel about that (and whether they really understood what that means). I'm not wise enough to have figured out tests that will tell me.
And I'd have to hit some team-oriented discussion, too, since that's my biggest concern lately, even more than product-oriented. And it's something a lot of companies seem to do badly (e.g., people focused on their own appearance in sprint tasks or metrics or promotions, rather than the whole of the team's work coming together).
Re: CheatGPT
#466Earlier quoted context omitted.
Knowing how to produce an essay is exactly the same as "analytical thinking, research, and argument skills" with the added challenge of making it legible to a reader — which is what makes those skills useful.
I suppose, but having written plenty of essays as an adult I can say with complete certainty that nothing I learned from my 5 paragraph days was of any use. No one, not you, not your teacher, not any real life audience for any topic you would be presenting on or publishing for, wants to read anything remotely close to what you're forced to write in school.
What you were forced to write in school. I readily admit that I had an quality of education several SDs higher than usual, but the trite "5 paragraph" nonsense is neither universal or (more importantly) inevitable.
Re: CheatGPT
#467Earlier quoted context omitted.
I assumed scheme and lisp were the same here semantically. Bad assumption on my part. My experience is primarily with scheme. Similar Rust and Swift allow let bindings to be shadowed but you need to re-declare your intent. You can't just mutate the binding on a whim.
I'm confused, Scheme also permits mutation of let bindings so Lisp and Scheme do have similar semantics here. Neither language provides immutable (const) let bindings. Scheme just makes it more obvious (by convention) when you may have mutation, through function and form names ending in !.
Re: CheatGPT
#468Excercises should not be about evaluating and judging. They should be about learning. If a student uses an AI, or copies someone else's work, it is to their own detriment. (Exams are a whole different question.) Even before AI, you could read a book and copy-paste the excercises, or just skip them, but if you wanted the full learning benefit you would type them out. I think we will have to focus more on teaching how…
So if you don't even try to stop people from cheating you end up doing a disservice to those who do, as you devalue that bit of paper.
Re: CheatGPT
#469Earlier quoted context omitted.
While I'm not complaining that people are realizing they're dead... why is ChatGPT the final straw for those ridiculous "make a binary tree" questions? Why wasn't it the fact that these questions became such a gameable system, that we started referring to them by the copyrighted name of a site where you can access nearly every permutation that will ever be asked of you, along with extremely detailed solutions with ra…
I mean if you can’t make a binary tree, what do you really know about programming? If ChatGPT is so great, why would even need a job interview, why not just eliminate the jobs altogether?
Re: CheatGPT
#470Earlier quoted context omitted.
the other week i gave chatgpt a simple multiplication problem that it got wrong. very simple problem like 86 * 0.0007 or something. but ive been working with chatgpt for 4-5 weeks now and that wrong answer doesnt make up for all the "good answers" that are usually not perfect. like one day i needed to COALESCE in mysql. i didnt know that, but chatgpt did. theres a few times i would have written a function the complic…
Why would you use chatGPT as a calculator? Use a calculator for that.