Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Airbyte Agents – context for agents across multiple data sources

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Show HN: Airbyte Agents – context for agents across multiple data sources

#52

Post by aaronsteers »

sails wrote: post #43
swyx wrote: post #8

(former employee here) congrats Michel! so glad to see you guys adapting to the AI age so well (and using the crap out of Devin!) hmm so airbyte agents could serve as a form of MCP gateway, or a key building block of an MCP gateway, which btw is how anthropic uses mcp themselves for all their internal apps https://www.youtube.com/watch?v=CD6R4Wf3jnY&t=1s&pp=0gcJCd4K... i think my most sad/interesting observation abou…

I think this is right ( a big gap ) but I don’t think data companies even now what the right shape is for AI. It’s definitely not old school ETL + dbt + BI tool, it might be something like this, but it’s very early

Agreed 100% - we're still super early in this journey, gathering data from our own usage and from our customers' feedback.

Re: Show HN: Airbyte Agents – context for agents across multiple data sources

#53

Post by aaronsteers »

ck_one wrote: post #36

More and more SaaS companies like ServiceNow or Hubspot are creating new tollgates for agent api calls. How do you think will this impact Airbyte Agents? I guess that replicating data locally will be harder since the platforms will try to protect it or charge for it.

It's a good question and I won't pretend to predict the future on this one. I will say, I think Airbyte Agents is in a good position because our core Data Replication product has always had to mitigate the impacts of rate limiting and cumbersome upstream APIs. The new Agents toolset gives you the ability to query the upstream APIs directly (read: as a passthrough) while also letting you bypass them entirely when your agent can answer its question via the Context Store directly. Time and feedback from our users will confirm, but I do think this gives our customers a good balance of control - when to query upstream directly and when to utilize the Context Store to work around API limitations - whether inherent or artificially enforced by the vendor.

Re: Show HN: Airbyte Agents – context for agents across multiple data sources

#54

Post by aaronsteers »

pjm331 wrote: post #48
aaronsteers wrote: post #23

Earlier quoted context omitted.

That's great to hear - great minds think alike! > give the agent access to the DB This is where Airbyte really can shine, I think, and the total can be more the sum of the parts. Because Airbyte excels at data replication already, we can populate your the Agent Context Store without users or agents ever needing to think about the words "ELT" or "ETL". We're listening carefully to feedback so we hope you will give it…

yeah this is one of the few AI-related products that I have seen that make sense to me but i also wonder to what extent this needs to be its own thing or if this is just something that it looks like we need but really people just need to shovel more stuff into their data warehouse / data lake that you never had reason to before, because now that's all fodder for agentic search

Great point. Many of Airbyte's customers are doing just that - adding new sources to their warehouses - like Google Drive, Gong, and a ton of sources that weren't as interesting previously for data analytics. But this creates a ton of work for the data engineering teams - to not only load all that extra data, but to deal with rate limits and then to conform the schemas into a usable format after loading.

For now, I think its 100% appropriate to think of the Context Store complementing the Warehouse and not replacing it per se. We're evaluating future integration options between the new Context Store and the traditional data warehouse, but nothing we have publicly announced as of now. I think both approaches have their strengths and killer use cases.

Re: Show HN: Airbyte Agents – context for agents across multiple data sources

#57

Post by juancs »

aaronsteers wrote: post #29
jessewmc wrote: post #17

Looks interesting! If I'm reading correctly, the indexing (Context Store) is neutral/unopinionated? How does it select fields for indexing? Have you done any testing on guided indexing, or metadata layers on top of the data? My experience so far on similar work is that getting data in front of an agent isn't enough context to get useful/reliable answers enough of the time. I.e. _what_ you index, and how you signpost…

Hi, @jessewmc. Thanks for your reply. Regarding your points: > If I'm reading correctly, the indexing (Context Store) is neutral/unopinionated? How does it select fields for indexing? While we haven't yet published details on the backend implementation, I can say that our implementation performs very well without needing to prioritize specific fields for indexing. We aim for large text fields to perform decently and…

Great launch btw! I have some questions if you don't mind

you mentioned that performance was never an issue, I am really intrigued how this is achieved.

I have 3 General questions:

1. How big (estimate in bytes) and complex were the test datasources? I couldn't find this in the benchmark repo.

2. how is the business context managed? In the blog "Airbyte Agents: A New Era for Airbyte" it was mentioned handling the business context but in the context layer docs it only talks about schema discovery (I got a bit confused)

3. When you said performance was never an issue, do you mean the user always got the answer it was looking for?

Post reply on HN