Viewing profile — lorey

Found this to display the optimal LLM choice while building evalry. It's such a useful tool, not only for thinking about it, but for visualization, too. Example: Which LLM gives me…

What I read from that is that there's a chance claude gets worse using a claude.md and that's a usability issue on their side.

Their response: > The team that made dataroom has stated that they did not use any of papermark’s code and that dataroom was made from scratch with inspiration from existing docume…

That is not what the article argues.

Haha, very true. Exactly as described in the article.

This is true with one caveat. In most cases, e.g. with regular ML, evals are easy and not doing them results in inferior performance. With LLMs, especially frontier LLMs, this has …

This is a very good point. When I came in, the founder did a lot of evaluation based on a few prompts and with manual evaluation, exactly as described. Showing the results helped m…

Doesn't this depend a lot on private vs company usage? There's no way I could spend more than a few hundreds alone, but when you run prompts on 1M entities in some corporate use ca…

It's not you, it's the HN hug of death. There's so much load on the server, I'm barely able to download the redis image I need for caching...

Thanks. Will take a look.

Depends on your remaining budget ;)

I've skipped that in the article, but absolutely!

Fixed, thanks. Not a native speaker.

That's interesting. Similarly, we found out that for very simple tasks the older Haiku models are interesting as they're cheaper than the latest Haiku models and often perform equa…

Pushed a fix. Could you check, please? Any resources you can recommend to properly tackle this going forward?

Will fix, thanks :)

Totally agree with your point. While I can't say specifically, it's a traditional (German) business he's doing vertically integrated with AI. Customer support is really bad in this…

This went straight to prod, even earlier than I'd opted for. What do you mean?

Appreciate the feedback, will work on that.

You're right. We did a few use cases and I have to admit that while customer service is easiest to explain, its where I'd also not choose the cheapest model for said reasons.

Yes, absolutely. This aligns with what we found. It seems to be necessary to be very clear on scoring (at least for Opus 4.5).

Very interesting points. Would you mind sharing a few examples of when cherry-picking is necessary and why atomic changes are a lie? I'm using a monorepo for my company across 3+ p…

Yes, there's the an other specific tags providing far more options than the old favicons. For anyone interested: https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/...

Thanks, this is how I feel about this, too.