31–40 of 60 posts
A few basic questions: how reliable is this judge since it’s based on an LLM? What additional measures can an admin or someone with more technical knowledge take to tighten this further if needed? Are the rules in or used by this judge visible to the users or an admin?
> Any call the agent makes to an external connector is reviewed by an LLM judge to prevent dangerous operations. A few basic questions: how reliable is this judge since it’s based on an LLM? What additional measures can an admin or someone with more technical knowledge take to tighten this further if needed? Are the rules in or used by this judge visible to the users or an admin?
Don't take my word for it, watch the news in <6 months and we'll certainly see it get bypassed.
> Any call the agent makes to an external connector is reviewed by an LLM judge to prevent dangerous operations. A few basic questions: how reliable is this judge since it’s based on an LLM? What additional measures can an admin or someone with more technical knowledge take to tighten this further if needed? Are the rules in or used by this judge visible to the users or an admin?
It's not reliable, and presenting it as-such shows a lack of understanding of basic security posture. Don't take my word for it, watch the news in <6 months and we'll certainly see it get bypassed.
I did build a very similar product putting emphasis on integrations and security. It was tough to churn anyone out from Replit, Lovable and then CC and Codex. When I talked to Lovable team, they told me Lovable is used primarily for prototyping, not internal tools and they are having a rough time trying to change the use case. Hope my experience helps!
> Any call the agent makes to an external connector is reviewed by an LLM judge to prevent dangerous operations. A few basic questions: how reliable is this judge since it’s based on an LLM? What additional measures can an admin or someone with more technical knowledge take to tighten this further if needed? Are the rules in or used by this judge visible to the users or an admin?
Earlier quoted context omitted.
It's not reliable, and presenting it as-such shows a lack of understanding of basic security posture. Don't take my word for it, watch the news in <6 months and we'll certainly see it get bypassed.
Maybe that's a hot take, but it's probably still safer than how things are often done in reality when employees want to "get something done" and sign up for random services, download their own software or other variants of shadow IT.
This posture is applicable while we ease out of what I'm calling the "golden era" (loose restrictions on access with limited consequence) but one of the reasons I am pedantic and bearish about solutions like this is that they too have a short lifespan that we can already estimate the expiration date of.
> Any call the agent makes to an external connector is reviewed by an LLM judge to prevent dangerous operations. A few basic questions: how reliable is this judge since it’s based on an LLM? What additional measures can an admin or someone with more technical knowledge take to tighten this further if needed? Are the rules in or used by this judge visible to the users or an admin?
We assume LLMs can always be fooled, so the judge is just one step of the verification process. The deterministic layers, per-tool db roles, read-only-by-default integrations, query timeouts, are the ones an admin can configure. The judge's rules aren't exposed to admins today. I like your idea of letting admins add their own rules or knowledge to the judge, we should add that.
Seems like Superblocks
Awesome launch and good pricing. We built an equivalent version in our org that takes HTML/JSX files and stores/serves them like an S3 bucket would. Works fabulous and we considered turning it into a SaaS product, met a real need in our org. Wishing you guys the very best of luck!!
I am currently building something similar for my company. We are enabling people to host apps with python backends as well, so that people can do more complex tweaking of data for scientific workflows. The out-of-box solutions keep users constrained to an environment that doesn't allow things like importing weird packages on the back and front end for custom apps. The security implications are challenging, but I can…
The ordering thing is harder and currently we don't do it. Per-tool permissions at best ban a combination, they can't say "internet then Slack is ok, the reverse isn't." DeepMind's CaMeL paper (arxiv.org/abs/2503.18813) is the best take I know on this, it tracks data flow between tool calls and enforces policy there. Still an open problem as far as I can tell.
Earlier quoted context omitted.
TBH I was a retool power user for years, but I feel like they completely missed the vibe code train. It's so much more annoying to go drag boxes around and type out little JS snippits in every chart as opposed to just telling claude to make the chart.
Have you looked at their latest product? You describe your app in a text box and the AI builds it, with all the enterprise goodies like RBAC baked in.