21–30 of 63 posts
Your billing support email forwards to a google group which rejects the email entirely. So i embedded my question inside the websites sales enquiry form and received multiple rounds of emails that couldn’t be further from human. It’s not why we started using posthog but it definitely sealed the deal when you see how simple and reliable that experience is
I feel like we've been working in parallel here :) We are using PyAirbyte (hi aaronsteers) for our users to connect their data sources to our agents. We originally wanted to use the airbyte white-label platform, but the team said that it was being deprecated. I think this really drives home just how crucial it is to have a clear model for accessing your data, and Airbyte has been great at that for quite a while.
I am happy to hear you are still getting value out of PyAirbyte! If you do try out Airbyte Agents, please let us know how it goes! We are always listening to feedback and would love to hear from you as you explore the new tools and capabilities.
sounds very familiar to what I ended up doing on my internal system - especially anything to do with search - much better to just sync everything to a DB and give the agent access to the DB
> give the agent access to the DB
This is where Airbyte really can shine, I think, and the total can be more the sum of the parts. Because Airbyte excels at data replication already, we can populate your the Agent Context Store without users or agents ever needing to think about the words "ELT" or "ETL".
We're listening carefully to feedback so we hope you will give it a try and let us know how it goes! Thanks!
This is such a great direction airbyte is taking and congrats to the lunch! I think you're very well-positioned for this opportunity than most people realize, given your reputable brand and your uncanny expertise in etl. It's honestly a natural progression of airbyte as far as the current AI landscape goes. Kudos to you and the team! (We use airbyte at my company, although we self-host it.)
Earlier quoted context omitted.
Your point about search being a bottleneck is spot on. IMO, search APIs should return guidance to agents to help them winnow down the results faster. For example, if your query returns 1000 results, then it should tell the agent, "too many results, we recommend you filter on column X because of Y to improve your search. Here are the possible values in column X: ..."
There are a lot of APIs like this that I really wish would expose downloading a parquet file instead of trying to implement server-side filtering and reporting query features.
Working with APIs is often frustrating and the worst ones are terribly ineficient and frustrating. Our Agent SDK and Agent Context Store insulates you and your agent from this headache, allowing you to query from those synced datasets directly.
The feedback about wanting to download a parquet file is super interesting...
Earlier quoted context omitted.
AJ here, from Airbyte. Yes, we've definitely found that some API data models are easier for models to navigate than others. The largest factors of Agent inefficiency we've identified so far are: 1. Many APIs lack robust-enough search, forcing agents to page through hundreds or thousands of paginated responses until they find the record they are looking for (our Context Store addresses this). 2. Many APIs have HUGE re…
Your point about search being a bottleneck is spot on. IMO, search APIs should return guidance to agents to help them winnow down the results faster. For example, if your query returns 1000 results, then it should tell the agent, "too many results, we recommend you filter on column X because of Y to improve your search. Here are the possible values in column X: ..."
(former employee here) congrats Michel! so glad to see you guys adapting to the AI age so well (and using the crap out of Devin!) hmm so airbyte agents could serve as a form of MCP gateway, or a key building block of an MCP gateway, which btw is how anthropic uses mcp themselves for all their internal apps https://www.youtube.com/watch?v=CD6R4Wf3jnY&t=1s&pp=0gcJCd4K... i think my most sad/interesting observation abou…
(former employee here) congrats Michel! so glad to see you guys adapting to the AI age so well (and using the crap out of Devin!) hmm so airbyte agents could serve as a form of MCP gateway, or a key building block of an MCP gateway, which btw is how anthropic uses mcp themselves for all their internal apps https://www.youtube.com/watch?v=CD6R4Wf3jnY&t=1s&pp=0gcJCd4K... i think my most sad/interesting observation abou…
> airbyte agents could serve as a form of MCP gateway
Exactly! And a single set of tools for agents to access both realtime (direct reads/writes) as well as cached (Context Store), bringing hopefully the best access path for each different use case.
> would love a "data engineering for ai engineers" type braindump ... at AIE
Great idea - we have a booth at AIE, and we'll submit there for a talk. Mario will reach out to you about this. :)
Looks interesting! If I'm reading correctly, the indexing (Context Store) is neutral/unopinionated? How does it select fields for indexing? Have you done any testing on guided indexing, or metadata layers on top of the data? My experience so far on similar work is that getting data in front of an agent isn't enough context to get useful/reliable answers enough of the time. I.e. _what_ you index, and how you signpost…
> If I'm reading correctly, the indexing (Context Store) is neutral/unopinionated? How does it select fields for indexing?
While we haven't yet published details on the backend implementation, I can say that our implementation performs very well without needing to prioritize specific fields for indexing. We aim for large text fields to perform decently and retrieval based on small/compressible fields like ints to be fast. (More to come on this in the coming months.)
> Have you done any testing on guided indexing, or metadata layers on top of the data?
We've been testing with different data scales and shapes. Nothing detailed to share yet, but performance has (so far) never itself become the bottleneck in our agent testing. (The LLM thinking itself is often the bottleneck.)
> My experience so far on similar work is that getting data in front of an agent isn't enough context to get useful/reliable answers enough of the time.
Airbyte has rich metadata on our upstream connector's data models, which I think helps us a lot to deliver helpful context to the agent. Another option, when optimizing for specific use cases, is to build your own agent tools on top of our Agent SDK. This allows you to make the calls organic and build the tools in a way that makes natural sense to the agent, regardless of source shape or which system(s) that data is coming from.
> This does look like a good foundation for that kind of tooling though!
We agree! Thanks again for sharing your thoughts here.
What actions does agents enable that weren't already available from Airbyte?
1. Programmatic Interfaces: Including a new REST API, SDK, and MCP Server. 2. New action verbs: Not just replication anymore. We have get/set/list/update/upload, and more! 3. New credentials passthrough: For all the above, you OAuth to Airbyte and we OAuth on your behalf to the systems your agent needs. No need to provide your agents dozens of different secrets in order to access the systems it needs. 4. Context Store. Like your agents' own data warehouse, but completely automatic and hands-free. For those use cases that just aren't possible when calling the REST API directly.
Again - thanks for your comment and sorry for the longwinded response. More info here: https://docs.airbyte.com/ai-agents/