Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using LLMs at Oxide

rfd.shared.oxide.computer

Post reply on HN
Print view View original

261–270 of 284 posts

Re: Using LLMs at Oxide

#261

Post by girvo »

pmg101 wrote: post #177
mulmboy wrote: post #132

Earlier quoted context omitted.

What do these look like?

1. Take every single function, even private ones. 2. Mock every argument and collaborator. 3. Call the function. 4. Assert the mocks were called in the expected way. These tests help you find inadvertent changes, yes, but they also create constant noise about changes you intend.

You beat me to it, and yep these are exactly it.

“Mock the world then test your mocks”, I’m simply not convinced these have any value at all after my nearly two decades of doing this professionally

Re: Using LLMs at Oxide

#262

Post by girvo »

menaerus wrote: post #147
girvo wrote: post #113

Earlier quoted context omitted.

No, it is qualitatively different because it happens in-line and much faster. If it’s not correct (which it seems it usually isn’t), they spend more time removing whatever garbage it autocompleted.

People do it with the autocomplete as well so I guess there's not that much of a difference wrt LLMs. It likely depends on the language but people who are inexperienced in C++ would be over-relying on autocomplete to the point that it looks hilarious, if you have a chance to sit next to them helping to debug something for example.

For sure, but these new tools spit out a lot more and a lot faster, and it’s usually correct “enough” that the compiler won’t yell. It’s been wild to see its suggestions be wrong far more often than they are right, so I wonder how useful they really are at all.

Normal auto complete plus a code tool like Claude Code or similar seem far more useful to me.

Re: Using LLMs at Oxide

#263

Post by pests »

ambicapter wrote: post #243
aatd86 wrote: post #81

Earlier quoted context omitted.

The issue is that it might look good but an LLM often inserts weird mistakes. Or ellipses. Or overindex on the training data. If someone is not careful it is easy to completely wreck the codebase by piling on seemingly innocuous commits. So far I have developed a good sense for when I need to push the llm to avoid sloppy code. It is all in the details. But a junior engineer would never find/anticipate those issues. I…

If it's such a mind numbing problem it's easy to check it though, and the checking you do after the LLM will be much smaller than you writing every field (implicitly "checking" it when you write it). Obviously if it's anything even minorly complex you can't trust the LLM hasn't found a new way to fool you.

This is exactly it. There wasn't any complex logic. Just making sure the right fields were mapped, some renaming, and sometimes some more complex joins depending on the incoming data source and how it was represented (say multiple duplicate rows or a single field with comma delimited id's from somewhere else). I would have much rather scanned the LLM output line by line (and most would be simple, not very indented) then hand writing from scratch. I do admit it would take some time to review and cross reference, but I have no doubt it would have been a fraction of the time and effort.

Re: Using LLMs at Oxide

#264

Post by venturecruelty »

12300886574321 wrote: post #256
bcantrill wrote: post #230

Earlier quoted context omitted.

I am really surprised that people are surprised by this, and honestly the reference was so casual in the RFD because it's probably the way that I use LLMs the most (so very much coming from my own personal experience). I will add a footnote to the RFD to explain this, but just for everyone's benefit here: at Oxide, we have a very writing-intensive hiring process.[0] Unsurprisingly, over the last six months, we have s…

I debated not writing this, as I planned on re-applying again, as oxide is in many ways a dream company for me, and didn't want this to hurt my chances if I could be identified and it was seen as negative or critical (I hope not, I'm just relaying my experience, as honestly as I can!), but I felt like I needed to make this post (my first on HN, a longtime lurkerj). I applied in the last 6 months, and against my bette…

I mean this nicely: please don't prostrate yourself for these companies. Please have some more respect for yourself.

Re: Using LLMs at Oxide

#265

Post by dgroshev »

12300886574321 wrote: post #256
bcantrill wrote: post #230

Earlier quoted context omitted.

I am really surprised that people are surprised by this, and honestly the reference was so casual in the RFD because it's probably the way that I use LLMs the most (so very much coming from my own personal experience). I will add a footnote to the RFD to explain this, but just for everyone's benefit here: at Oxide, we have a very writing-intensive hiring process.[0] Unsurprisingly, over the last six months, we have s…

I debated not writing this, as I planned on re-applying again, as oxide is in many ways a dream company for me, and didn't want this to hurt my chances if I could be identified and it was seen as negative or critical (I hope not, I'm just relaying my experience, as honestly as I can!), but I felt like I needed to make this post (my first on HN, a longtime lurkerj). I applied in the last 6 months, and against my bette…

Hey fellow failed applicant!

I had a very similar experience, except I got the automated email after two months, not three — you sound like a stronger candidate, so maybe that's why I got rejected sooner, which'd be fair enough. Still, spending about a week's worth of evenings between the suggested materials, reflecting, writing, and editing 15 pages for one job application and having zero human interaction feels uniquely degrading.

I disagree with your point about that being fine. I think it's not good enough to replicate the bare minimum of what the rest of the industry does while asking for so much more from candidates.

A standard custom, well researched cover letter takes an order of magnitude less effort. When it's cookie cutter rejected by someone spending a few seconds on the CV, it's at least understandable: the effort they'd spend writing a rejection (or replying back) is higher than the amount of effort they spent evaluating the application.

With Oxide however, Brian made a point that they "definitely read everyone's materials" [1]. Which means reading at the very least five pages per candidate. If that's still the case, having an actual human on the other side of the rejection would add a very small amount of time to the whole process, but the company decided to do the absolute least possible. It's a choice, and I think this choice goes against their own principle of decency:

"We treat others with dignity, be they colleague, customer, community or competitor."

I wish Oxide best of luck. They have lots of very smart, very driven people that I'd love to work with, and I love what they are doing. Hope this feedback helps them get better.

[1]: https://youtu.be/wN8lcIUKZAU?t=1400

P.S. Don't you dare, dear reader, consider the emdash above an LLM smell.

Re: Using LLMs at Oxide

#266

Post by bcantrill »

12300886574321 wrote: post #256
bcantrill wrote: post #230

Earlier quoted context omitted.

I am really surprised that people are surprised by this, and honestly the reference was so casual in the RFD because it's probably the way that I use LLMs the most (so very much coming from my own personal experience). I will add a footnote to the RFD to explain this, but just for everyone's benefit here: at Oxide, we have a very writing-intensive hiring process.[0] Unsurprisingly, over the last six months, we have s…

I debated not writing this, as I planned on re-applying again, as oxide is in many ways a dream company for me, and didn't want this to hurt my chances if I could be identified and it was seen as negative or critical (I hope not, I'm just relaying my experience, as honestly as I can!), but I felt like I needed to make this post (my first on HN, a longtime lurkerj). I applied in the last 6 months, and against my bette…

Your materials were absolutely read (and indeed, RFD 576 makes clear that LLMs are not a substitute for reading materials). If you have writing samples that were external links, I can't guarantee that they were clicked through though: in part because the materials themselves constitute a galactic writing sample, we may have not clicked through because we were already at a decision point before reading your external writing. As for more specific feedback, if you can DM me, I'll see if I can give you more specific feedback -- but as we explicitly indicate in RFD 3[0], we are very limited in what we can provide.

As for your application getting flagged as LLM-generated: we in fact don't flag such applications (we just reject them), and it's very unlikely that we felt that yours were LLM-generated. (We really, really give applicants the benefit of the doubt on that.)

All of that said: absolutely no one is a shoe-in at Oxide. If you genuinely thought that (and if your materials reflected that kind of overconfidence), it may have well guided our decision. We are very selective in terms of hiring -- and we are very oversubscribed. Bluntly: it's very hard to get a job at Oxide. I know this seems harsh and/or unjust or unfair, but this is the reality. As we told you in the letter we sent you, we already have people at Oxide who prevailed on subsequent applications, because they found a job that's a better fit for them, or they have vastly improved materials (or both). Finally, you can also take solace in knowing that your post here in no way hurts your future chances at Oxide, and we look forward to reading your materials should you choose to apply in the future.

[0] https://rfd.shared.oxide.computer/rfd/0003#_rejection_of_non...

Re: Using LLMs at Oxide

#267

Post by bcantrill »

dgroshev wrote: post #265
12300886574321 wrote: post #256

Earlier quoted context omitted.

I debated not writing this, as I planned on re-applying again, as oxide is in many ways a dream company for me, and didn't want this to hurt my chances if I could be identified and it was seen as negative or critical (I hope not, I'm just relaying my experience, as honestly as I can!), but I felt like I needed to make this post (my first on HN, a longtime lurkerj). I applied in the last 6 months, and against my bette…

Hey fellow failed applicant! I had a very similar experience, except I got the automated email after two months, not three — you sound like a stronger candidate, so maybe that's why I got rejected sooner, which'd be fair enough. Still, spending about a week's worth of evenings between the suggested materials, reflecting, writing, and editing 15 pages for one job application and having zero human interaction feels uni…

I understand your disappointment; we are very explicit about why we provide so little feedback.[0] I disagree that it's indecent; to the contrary, we allow anyone to shoot their shot, with the guarantee that they will be thoughtfully considered.

[0] https://rfd.shared.oxide.computer/rfd/0003#_rejection_of_non...

Re: Using LLMs at Oxide

#268

Post by ahepp »

Yeask wrote: post #197
ahepp wrote: post #24

Earlier quoted context omitted.

It seems like this would be a really interesting field to research. Does AI assisted coding result in fewer bugs, or more bugs, vs an unassisted human? I've been thinking about this as I do AoC with Copilot enabled. It's been nice for those "hmm how do I do that in $LANGUAGE again?" moments, but it's also wrote some nice looking snippets that don't do quite what I want it to. And many cases of "hmmm... that would wor…

This companies have trillions and they are not doing that research. Why?

I don't know. I guess the flip side applies too? Lots of people arguing either side, when it feels like it shouldn't be that difficult to provide some objective data.

Re: Using LLMs at Oxide

#269

Post by dgroshev »

bcantrill wrote: post #267
dgroshev wrote: post #265

Earlier quoted context omitted.

Hey fellow failed applicant! I had a very similar experience, except I got the automated email after two months, not three — you sound like a stronger candidate, so maybe that's why I got rejected sooner, which'd be fair enough. Still, spending about a week's worth of evenings between the suggested materials, reflecting, writing, and editing 15 pages for one job application and having zero human interaction feels uni…

I understand your disappointment; we are very explicit about why we provide so little feedback.[0] I disagree that it's indecent; to the contrary, we allow anyone to shoot their shot, with the guarantee that they will be thoughtfully considered. [0] https://rfd.shared.oxide.computer/rfd/0003#_rejection_of_non...

Indeed, I understand your reasoning, you talk about that in the podcast in the RFD. This is why I wasn't talking about the lack of feedback, but the lack of human interaction. While there is nothing constructive to be done about the disappointment of rejection, this part is very much in your power to change, and that's why I think it's constructive feedback and not just venting.

That said, the RFD does say this:

> Candidates may well respond to a rejection by asking for more specific feedback; to the degree that feedback can be constructive, it should be provided.

Even just replying with refusal to provide feedback would still be more humane and decent.

Re: Using LLMs at Oxide

#270

Post by aatd86 »

ambicapter wrote: post #243
aatd86 wrote: post #81

Earlier quoted context omitted.

The issue is that it might look good but an LLM often inserts weird mistakes. Or ellipses. Or overindex on the training data. If someone is not careful it is easy to completely wreck the codebase by piling on seemingly innocuous commits. So far I have developed a good sense for when I need to push the llm to avoid sloppy code. It is all in the details. But a junior engineer would never find/anticipate those issues. I…

If it's such a mind numbing problem it's easy to check it though, and the checking you do after the LLM will be much smaller than you writing every field (implicitly "checking" it when you write it). Obviously if it's anything even minorly complex you can't trust the LLM hasn't found a new way to fool you.

True. The counterpoint being that back in the days, they could have decided to write a parser if the data was structured and they would have then learnt things that they will never learn by relying on AI.

For a junior in the learning phase that can be useful time spent. Then again, I agree that at times certain menial code tasks are not worth doing and llms are helpful.

It's a bit like a kid not spending time memorizing their time tables since they can use a calculator. They are less likely to become a great mathematician.

Post reply on HN