Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airbyte: Simple and extensible open-source EL(T)

github.com

Post reply on HN
Print view View original

21–25 of 25 posts

Re: Airbyte: Simple and extensible open-source EL(T)

#21

Post by xupybd »

forgetfulness wrote: post #7
xupybd wrote: post #4

I've found small scripts do a great job with ETL. Run by cron and log to text seems to do the job. What's the advantages in using a large tool like this?

It's a bit unclear on what the scope of this tool is. But there are a couple of new classes of tools for ETL/ELT or data engineering as it's called now. There's the "Data Integration Tools" like Fivetran, Stitch, and this. They are collections of connectors that they have coded to ease ingesting data from lots of different database products and stores to another. That's valuable, I wouldn't start writing my own scrip…

Thanks that makes a lot of sense.

Re: Airbyte: Simple and extensible open-source EL(T)

#22

Post by dal »

sherifnada wrote: post #15
dal wrote: post #13

Just to clarify, this is not a fully open source project. More like freemium or just enough for the community to contribute and maintian free connectors which make it usefull. If you want security, you have to pay. There are non-free parts to it that you are not allowed to use. And contributing these features would probably not be allowed.

Hi, Airbyte engineer here. All connectors and the core functionality of Airbyte will _always_ be open source and free to use. You will always be able to download and operate an instance of Airbyte, containing all connectors and ELT features (whether for personal projects or business operations in your company) for free. However, there are some things that we might offer as a monetized product in the future, such as:…

No, this was exactly my point.

It is a company behind and not a foundation backed by multiple interests. So profit is the main driver. You want to have it open enough so you get community maintained connectors. This pattern is common.

If somebody were to implement those features you described there and commit to the repo under the BSD license would you merge it?

Re: Airbyte: Simple and extensible open-source EL(T)

#23

Post by jeanlaf »

dal wrote: post #22
sherifnada wrote: post #15

Earlier quoted context omitted.

Hi, Airbyte engineer here. All connectors and the core functionality of Airbyte will _always_ be open source and free to use. You will always be able to download and operate an instance of Airbyte, containing all connectors and ELT features (whether for personal projects or business operations in your company) for free. However, there are some things that we might offer as a monetized product in the future, such as:…

No, this was exactly my point. It is a company behind and not a foundation backed by multiple interests. So profit is the main driver. You want to have it open enough so you get community maintained connectors. This pattern is common. If somebody were to implement those features you described there and commit to the repo under the BSD license would you merge it?

In all honesty, in that particular case, we'd honestly think about merging it. If that happens, we're facing 2 options. Either we merge it, and at that point the real business model for our business is about hosting, management, support, SLAs, etc. Or we don't, and that contributor could very well publish it in another repo. And then, what's the point?

We call this type of business model open core. It is the same one as Confluent or DBT. Without those companies, there would have been nobody to create those open source projects and get them to a place of value for a maximum number of companies. Kafka's impact goes well beyond Confluent's customer base. Similarly, our goal is to solve the data integration problem for every company or project, even if we only convert a small % of them as customers for us. That's why we won't be focusing on any premium features for at least a year and possibly 2. Hope that makes sense to you.

Re: Airbyte: Simple and extensible open-source EL(T)

#24

Post by specialist »

cgardens wrote: post #16
specialist wrote: post #14

Earlier quoted context omitted.

Now imagine those self contained scripts are deployed to target servers. Add logging, metrics, remote control, atomic data pipelines. Or what I like to call 'data processing'. That's more or less what I built for healthcare IT. Before the term "serverless" was coined. Just peeked at Airbyte. I'd definitely look longer next time I have to do ETL. -- My only pause for concern is their phrase "optional normalized schema…

Different Airbyte engineer here! Wanted to help answer your question as to what "optional normalized schemas" means. When writing data into your data warehouse we provide 2 options: 1. write each record as a json blob. 2. infer the schema of the data and write each value in a record to its own column with an appropriate type. We are betting on EL(T), meaning we think that Transform should be considered separately fro…

Thank you for replying. Just scanned DBT. Cool. It's reliance on SQL is The Correct Answer™. (For SQL capable systems, of course.) The solutions based on schemas, where the SQL is then somehow code generated, are terrible.

I'm skeptical of DBT's inference of dependencies between queries, but I'll keep an open mind until I have direct experience.

Happy hunting.

Re: Airbyte: Simple and extensible open-source EL(T)

#25

Post by glogla »

sdepablos wrote: post #19
glogla wrote: post #11

Earlier quoted context omitted.

If you don't mind me asking, how does Stitch scale? We're currently using bespoke Spark-based ETL tool because we need to move like 2 TB/day from various place, especially legacy system that don't make incremental data movement practical. How does Stitch even run, all SaaS in Stich Cloud or something? Thanks!

Pure SaaS. Check their pricing [1], as is row based and depending on how much data you need to move it can easily go out of budget ;) [1] https://www.stitchdata.com/pricing/

Right. Not something we can use then, since we can't have our data moving through public internet for legal reasons.

Thanks for the answer.

Post reply on HN