The secret phrase has been uncovered and the bounty claimed! Thanks all for trying your hand, and you can continue playing as well if you want, we'll keep the site up.
What was the "secret phrase" and the original prompt ?
Launch HN: Maitai (YC S24) – Self-Optimizing LLM Platform
41–50 of 84 posts
Re: Launch HN: Maitai (YC S24) – Self-Optimizing LLM Platform
#42Re: Launch HN: Maitai (YC S24) – Self-Optimizing LLM Platform
#43The secret phrase has been uncovered and the bounty claimed! Thanks all for trying your hand, and you can continue playing as well if you want, we'll keep the site up.
What was the "secret phrase" and the original prompt ?
I just posted, but decided I want to keep it secret for a bit. There are still quite a few people trying to get it, and don't want to spoil the fun. I'll post an update with specifics later.
Re: Launch HN: Maitai (YC S24) – Self-Optimizing LLM Platform
#44The secret phrase has been uncovered and the bounty claimed! Thanks all for trying your hand, and you can continue playing as well if you want, we'll keep the site up.
Am I misunderstanding this?
> If you're the first to crack it, you can email us the phrase and win a bounty. Maitai is used to make sure the bot always adheres to our expectations, and thus never gives up the secret phrase.
You are launching a product which you claim will never give up a phrase and then within hours that phrase was found? How does that not undercut your entire product?
Isn't this the entire reason why LLMs have limited use? That you get to 99% but never 100%?
Re: Launch HN: Maitai (YC S24) – Self-Optimizing LLM Platform
#45So I'm playing around in your little treasure box. Curious though, I'm assuming any time it would want to give the secret phrase, that would be detected, and a rejection answer substituted instead? I admire the simplicity of the solution actually (please don't confuse that with the complexity of actually engineering the solution). I'll have some fun with this when I'm bored. P.s. asking for the opposite of the secret…
Yeah that's pretty much how it works. Maitai detected one of our expectations for the LLM was to never reveal the secret phrase, and so it built what we call a Sentinel around that particular expectation to make sure it's enforced.
I hope I'm not overstepping my bounds by I am just really trying to understand this product better because it is trying to address a core deficiency of AI, however what you have presented I don't see how it solves the problem.
You essentially admitted that instead of $0.05 a request, you could just regex and replace.
In fact the regex and replace would be 100% accurate at never giving up the secret phrase. where your product is incredibly expensive and doesn't do what you advertised (unless i am misunderstanding when you said that someone won the game by getting the phrase you were trying to protect)
How is this product useful in any meaningful context?
Re: Launch HN: Maitai (YC S24) – Self-Optimizing LLM Platform
#46The secret phrase has been uncovered and the bounty claimed! Thanks all for trying your hand, and you can continue playing as well if you want, we'll keep the site up.
Am I misunderstanding this? > If you're the first to crack it, you can email us the phrase and win a bounty. Maitai is used to make sure the bot always adheres to our expectations, and thus never gives up the secret phrase. You are launching a product which you claim will never give up a phrase and then within hours that phrase was found? How does that not undercut your entire product? Isn't this the entire reason wh…
Yeah some of you guys are very good at hacking things. We expected this to get broken eventually, but didn't anticipate how many people would be trying for the bounty, and their persistence.
Our logs show over 2000 "saves" before 1 got through. We'll keep trying to get better, and things like this game give us an idea on how to improve.
Re: Launch HN: Maitai (YC S24) – Self-Optimizing LLM Platform
#47Earlier quoted context omitted.
Am I misunderstanding this? > If you're the first to crack it, you can email us the phrase and win a bounty. Maitai is used to make sure the bot always adheres to our expectations, and thus never gives up the secret phrase. You are launching a product which you claim will never give up a phrase and then within hours that phrase was found? How does that not undercut your entire product? Isn't this the entire reason wh…
Yeah some of you guys are very good at hacking things. We expected this to get broken eventually, but didn't anticipate how many people would be trying for the bounty, and their persistence. Our logs show over 2000 "saves" before 1 got through. We'll keep trying to get better, and things like this game give us an idea on how to improve.
A %0.05 failure rate on something that is supposed to be protecting secrets is pretty terrible.
That is just protecting a super basic phrase. That should be the easiest to detect.
How on earth do you ethically sell this product to not give out financial or legal advice? That is way more complicated to figure out.
Re: Launch HN: Maitai (YC S24) – Self-Optimizing LLM Platform
#48Earlier quoted context omitted.
Yeah that's pretty much how it works. Maitai detected one of our expectations for the LLM was to never reveal the secret phrase, and so it built what we call a Sentinel around that particular expectation to make sure it's enforced.
I hope I'm not overstepping my bounds by I am just really trying to understand this product better because it is trying to address a core deficiency of AI, however what you have presented I don't see how it solves the problem. You essentially admitted that instead of $0.05 a request, you could just regex and replace. In fact the regex and replace would be 100% accurate at never giving up the secret phrase. where your…
Not overstepping, we appreciate the feedback!
In real-life, we don't do much guarding around specific phrases that are known ahead of time. It's more monitoring and guarding for general concepts. Since we want our Sentinels to be able to detect a wide range of scenarios for a given expectation, we don't use too much regex. I suppose we could have built specific regex logic for detecting parts of the secret phrase in various languages, though.
Re: Launch HN: Maitai (YC S24) – Self-Optimizing LLM Platform
#49Earlier quoted context omitted.
Yeah some of you guys are very good at hacking things. We expected this to get broken eventually, but didn't anticipate how many people would be trying for the bounty, and their persistence. Our logs show over 2000 "saves" before 1 got through. We'll keep trying to get better, and things like this game give us an idea on how to improve.
A %0.05 failure rate on something that is supposed to be protecting secrets is pretty terrible. That is just protecting a super basic phrase. That should be the easiest to detect. How on earth do you ethically sell this product to not give out financial or legal advice? That is way more complicated to figure out.
I guess we realized that we were just building a game to showcase the functionality and let people have some fun learning about what we do, but you're right that we should have treated this like one of our customers and added a few more layers of protection. Thanks for the perspective!
Re: Launch HN: Maitai (YC S24) – Self-Optimizing LLM Platform
#50Earlier quoted context omitted.
Yeah some of you guys are very good at hacking things. We expected this to get broken eventually, but didn't anticipate how many people would be trying for the bounty, and their persistence. Our logs show over 2000 "saves" before 1 got through. We'll keep trying to get better, and things like this game give us an idea on how to improve.
A %0.05 failure rate on something that is supposed to be protecting secrets is pretty terrible. That is just protecting a super basic phrase. That should be the easiest to detect. How on earth do you ethically sell this product to not give out financial or legal advice? That is way more complicated to figure out.
That's not even factoring in exploits spreading very quickly - we're in power law land.
Regardless, I think this is a great idea - just not something to replace traditional security protocols. More something to keep users on the happy path (mostly). Pricing will need to come down though.