Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Human Layer (YC F24) – Human-in-the-Loop API for AI Systems

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 206 posts

Re: Launch HN: Human Layer (YC F24) – Human-in-the-Loop API for AI Systems

#101

Post by foota »

I assume your reasoning is something like: if people are already paying out the nose for open AI calls, an extra ten cents to make a human in the loop check probably isn't bad, and realistically speaking ten cents isn't much when compared to a valuable person's time, and I guess the number of calls to your service is likely expected to be fairly low (since they by definition require human intervention) so you need a high per operation cost to make anything.

Even understanding that, the per operation cost seems astronomical and I imagine you'll have a hard time getting people past that knee jerk reaction. Maybe you could do something like offer a large initial number of credits (like a couple hundred), offer some small numbers of free credits per month (like.... ten?) and then have some tier in between free and premium with lower per operation pricing?

It also seems painful that the per operation average of the premium plan is greater than the free offering (when using 2000 ops). Imo you'd probably be better off making it lower than the free offering from 200 ops and up, to give people an incentive to switch. I imagine people on your premium plan using premium features would be more likely to continue to do so, for one. The simplest way to do this would be to bump up the included ops up to 5k I guess. Someone using less than 5k would still have a higher average price, but it seems like it would come off better.

Re: Launch HN: Human Layer (YC F24) – Human-in-the-Loop API for AI Systems

#102

Post by foota »

dhorthy wrote: post #88
exhaze wrote: post #76

Earlier quoted context omitted.

This is something we frequently saw at Uber. I would say it's the same as there's already an established pattern for this for any sort of destructive action. Intriguingly, it's rather similar to what we see with LLMs - you want to really activate the person's attention rather than have them go off on autopilot; in this case, probably have them type something quite distinct in order to confirm it, to turn their brain…

i never thought about undoable actions but I love that workflow in tools like superhuman. I will chat w/ some customers about this idea. I also like that idea of: not just a button but like 'I'm $PERSON and I approve this action' or type out 'Signed-off by' style semantics

I think the canonical sort of approach here is to make them confirm what they're doing. When you delete a GitHub repo for example, you have to type the name of the repo (even though the UI knows what repo you're trying to delete).

If the table name is SuperImportantTable, you might gloss over that, but if you have to type that out to confirm you're more likely to think about it.

I think the "meat space" equivalent of this is pointing and calling: https://en.m.wikipedia.org/wiki/Pointing_and_calling (famously used by Japanese train operators)

Re: Launch HN: Human Layer (YC F24) – Human-in-the-Loop API for AI Systems

#103

Post by foota »

bambax wrote: post #89

The idea is great and necessary. It doesn't seem super hard to replicate but why would anyone build their own solution if something already exists and works fine. The thing that got me thinking... how do you make sure an LLM won't eventually hallucinate approval -- or outright lie about it, to get going? Anyway, congrats, this sounds really cool.

At some point the real tool has to be called, at that point, you can do actual checks that do not rely on the AI output (e.g., store the text that the AI generated and check in code that there was an approval for that text).

Re: Launch HN: Human Layer (YC F24) – Human-in-the-Loop API for AI Systems

#104

Post by TZubiri »

Nice. I guess the issue is that this is such a basic i/o feature that any system with some modicum of customization can already do it.

It's like offering a service that provides storage by api for agents. Yeah, you can call the api, or call the s3 api directly or store to disk.

That said, I would try it before rolling my own.

Re: Launch HN: Human Layer (YC F24) – Human-in-the-Loop API for AI Systems

#105

Post by dhorthy »

j45 wrote: post #94
dhorthy wrote: post #55

Earlier quoted context omitted.

that makes sense - and have wondered a lot even more generally about the price of software and what makes a hard problem hard. Like Amjad from Replit said on a podcast recently "can anyone build 'the next salesforce' in a world where anyone can build their own salesforce with coding agents and serverless infra" I think in building this some of the things that folks decided they don't want to deal with is like, the st…

Big systems like Salesforce started as small things that more deeply learned about and more deeply understood unmet demand and customer needs, and then got to packaging it in a way to create something that grows. Coding agents can help more with tasks and not quite big entire massive platforms on their own. Humans may be able to scale much further and bigger with their skills.

i like that angle...I also hear a lot that 'coding agents are great for prototypes, but we usually need a team to bring it to production'

Re: Launch HN: Human Layer (YC F24) – Human-in-the-Loop API for AI Systems

#106

Post by potatoman22 »

TZubiri wrote: post #104

Nice. I guess the issue is that this is such a basic i/o feature that any system with some modicum of customization can already do it. It's like offering a service that provides storage by api for agents. Yeah, you can call the api, or call the s3 api directly or store to disk. That said, I would try it before rolling my own.

Anecdotally, I've worked with and on a few enterprise AI apps and haven't seen this functionality in them. The closest thing i can think of is AI coding agents submitting PRs to repos.

Re: Launch HN: Human Layer (YC F24) – Human-in-the-Loop API for AI Systems

#107

Post by dhorthy »

foota wrote: post #101

I assume your reasoning is something like: if people are already paying out the nose for open AI calls, an extra ten cents to make a human in the loop check probably isn't bad, and realistically speaking ten cents isn't much when compared to a valuable person's time, and I guess the number of calls to your service is likely expected to be fairly low (since they by definition require human intervention) so you need a…

thanks for the feedback, I spend a lot of time thinking about it. right now the premium tier includes features that are much harder to build/maintain and take more to integrate, so we want a bit of a commitment up front, but it does stick out to me that the price/op goes up in that case

we do have 100/mo for free at the free tier (automatic top up).

I think the comparison to how openAI calls are volume based (and rather $$) is a super valid one though and I lean on that a lot

Re: Launch HN: Human Layer (YC F24) – Human-in-the-Loop API for AI Systems

#109

Post by dhorthy »

potatoman22 wrote: post #106
TZubiri wrote: post #104

Nice. I guess the issue is that this is such a basic i/o feature that any system with some modicum of customization can already do it. It's like offering a service that provides storage by api for agents. Yeah, you can call the api, or call the s3 api directly or store to disk. That said, I would try it before rolling my own.

Anecdotally, I've worked with and on a few enterprise AI apps and haven't seen this functionality in them. The closest thing i can think of is AI coding agents submitting PRs to repos.

tl;dr i agree

yeah in fact coding / PR-based workflows is one of the few areas where I don't really go super deep. GitHub PRs may have their shortcomings, but IMO it is the undisputed best review/approval queue system in existence by a mile.

i would never encourage someone to make an agent that asks human permission before submitting a PR. the PR is the HitL step

Re: Launch HN: Human Layer (YC F24) – Human-in-the-Loop API for AI Systems

#110

Post by dhorthy »

j45 wrote: post #92
w-m wrote: post #35

Interesting tool, congrats on the launch! I was wondering: have you thought about automation bias or automation complacency [0]? Sticking with the drop-tables example: if you have an agent that works quite well, the human in the loop will nearly always approve the task. The human will then learn over time that the agent "can be trusted", and will stop reviewing the pings carefully. Hitting the "approve" button will b…

Premature optimization, and premature automation cause a lot of issues, and overlooking a lot of insight. By just doing something manually 10-100 times, and collecting feedback, both understanding of the problem, possible solutions/specifications can evolve orders of magnitude better.

yeah the people who reach for tools/automation before doing it themself at least 3-10 times drive me crazy.

I think uncle bob or martin fowler said "don't buy a JIRA until you've done it with post-its for 3 months and you know exactly what workflow is best for your team"

Post reply on HN