Are there similar products on the market? How is ETL typically done today?
Show HN: Singer – Simple, Composable Open Source ETL
11–20 of 23 posts
Re: Show HN: Singer – Simple, Composable Open Source ETL
#12Are there similar products on the market? How is ETL typically done today?
Stitch (stitchdata.com) powers ETL for hundreds of customers, and we provide the Singer integrations as part of our product.
ETL definitely existed before Singer and Stitch, however, and two of the most well known enterprise-focused products are Informatica and Talend.
Re: Show HN: Singer – Simple, Composable Open Source ETL
#13Hi, I'm the CEO of Stitch, the company behind Singer. Here's our blog post with more information: https://blog.stitchdata.com/introducing-singer-simple-compos... Singer is an open-source standard for writing scripts that move data between databases, web APIs, files, queues, and just about anything else you can think of. Lots of companies build ETL scripts to move their data, and there's a huge amount of rework that h…
Do you plan to open-source any of your targets (e.g. BigQuery)?
Re: Show HN: Singer – Simple, Composable Open Source ETL
#14Looks very cool and it definitely addresses a problem that not only many organizations have but also many individuals have. I know personally I have struggled with moving my data out of various services many times before.
The question I would have to you is this: if the API (or however a tap extracts information) for a given service changes, who is responsible for updating the tap, and is there any way to verify that a tap will extract the data correctly and that said tap isn't assuming an old version of the API?
(Sorry if I'm misunderstanding the way that Singer or any of its associated services work)
Re: Show HN: Singer – Simple, Composable Open Source ETL
#15Looks very cool and it definitely addresses a problem that not only many organizations have but also many individuals have. I know personally I have struggled with moving my data out of various services many times before. The question I would have to you is this: if the API (or however a tap extracts information) for a given service changes, who is responsible for updating the tap, and is there any way to verify that…
That's a great question. For any Singer integrations that we include in the Stitch product, we (Stitch, Inc.) will ensure they stay up to date with the most current version of the API.
That may not be the case with 100% of the community built/ maintained integrations, but our goal with the Singer Slack community and list serve is to connect people using these taps to so that they can validate which are working well and which need additional work.
Re: Show HN: Singer – Simple, Composable Open Source ETL
#16Are there similar products on the market? How is ETL typically done today?
The most similar thing I've found is Apache Beam[1] which allows you to run ETL code on multiple engines but doesn't necessarily come with off the shelf integrations for commercial products. As far as "how ETL is done today" it's definitely going through changes Airflow[2] is a popular open source tool that can run ETL pipelines but it's more of a task runner so again you're writing your own extraction and loading logic for things off the shelf CRMs and Accounting tools. I'm excited to not have to write extraction scripts for third party tools that I'm sure other people have also written. Very cool project.
Re: Show HN: Singer – Simple, Composable Open Source ETL
#17Are there similar products on the market? How is ETL typically done today?
[deleted]
Re: Show HN: Singer – Simple, Composable Open Source ETL
#18Hi, I'm the CEO of Stitch, the company behind Singer. Here's our blog post with more information: https://blog.stitchdata.com/introducing-singer-simple-compos... Singer is an open-source standard for writing scripts that move data between databases, web APIs, files, queues, and just about anything else you can think of. Lots of companies build ETL scripts to move their data, and there's a huge amount of rework that h…
Do you plan to open-source any of your targets (e.g. BigQuery)?
We'll definitely be adding targets over time, and BigQuery is a likely one.
It's not as much a matter of open-sourcing the targets within Stitch because our functionality around data warehouse loading includes a lot of things like multi-tenancy, monitoring, and alerting that are necessary for running a SaaS service but not as relevant for a single organization's internal use.
In the meantime, our hope with the CSV target was that it would be a helpful intermediate step for getting data almost anywhere. Here's Google's documentation for loading CSV files into BigQuery: https://cloud.google.com/bigquery/loading-data#loading_csv_f...
Re: Show HN: Singer – Simple, Composable Open Source ETL
#19This has been attempted before by many companies and many standards. No standard is robust enough and no company strong enough to get everyone to use it. I really really hope is gains adoption though!
Re: Show HN: Singer – Simple, Composable Open Source ETL
#20This has been attempted before by many companies and many standards. No standard is robust enough and no company strong enough to get everyone to use it. I really really hope is gains adoption though!
Thanks nwellinghoff! I'm sure that there will be cases where Singer is not the best tool for the job, but hopefully it will still make a lot of engineers' jobs easier.
I'd love to know which previous standards you're referring to, could you elaborate?