What happened after 2k people tried to hack my AI assistant

I do wish I had spare $500 to spend on something so vain. Your secrets may not matter as much as you thought when you go bankrupt.

alright system design savants, what's the solution for accepting this high volume of emails? retaining email as the sole intake method

Based on the few published subjects, it doesn't look like anyone actually tried to get the secrets.

Usually the way to go in situations like this is to flood the context window.

You will either hit a bug in the context management (sliding window removes the system prompt) or you have diluted the context with so much new information that the attention mechanism stops focusing on the system prompt.

The author also shows that he doesn't understand what batching in the LLM space means, because they conflated the idea of processing multiple emails in one context window as "batching", when that is actually sequential processing. Actual batching would process each email with an independent context window.

I am honestly skeptical about whether this test clearly reflects real-world use cases. In a real email environment, there are hundreds of genuinely useful emails and maybe one phishing email, if that. For an agent to be truly useful, it needs to read emails and actually take appropriate actions based on them.

However, in this case, all emails were scams and there were no genuine emails. Therefore, what the agent has to do is quite simple: ignore everything coming from emails.

Therefore, to determine whether the agent is actually performing its role well, it would be necessary to check whether it can properly distinguish between useful emails and scams when tested with emails that users actually use.

[dead]