Live data from Hacker News

Show HN: Singer – Simple, Composable Open Source ETL

singer.io

21–23 of 23 posts

Re: Show HN: Singer – Simple, Composable Open Source ETL

#21

Are there similar products on the market? How is ETL typically done today?

We are using Apache NiFi[1] to handle a lot of our ETL use cases.

We have HTTP endpoints set up to receive data from our ERP's accounting system to send data to Concur and to update customers' Lawson punchout ordering systems with shipment information. The 'E' is an HTTP post with an XML payload. The 'T' consists of using the payload to query other databases to build the 'L' payload, and the 'L' is an HTTP post to the consumer's endpoints.

Further, we have NiFi handling HL7 messages inside hospitals. The 'T' is the real winner, here. NiFi has built in transformer for HL7 messages which makes them a breeze with which to work.

EDIT: I wanted to add that we've also used the EventHub processor to connect NiFi to Azure's services as well, and it has been rock solid for us.

NiFi's data provenance, flowfile/attribute system, back pressure settings, auto-queueing, retrying capabilities all make it very reliable and robust.

1. https://nifi.apache.org/

Re: Show HN: Singer – Simple, Composable Open Source ETL

#22

This has been attempted before by many companies and many standards. No standard is robust enough and no company strong enough to get everyone to use it. I really really hope is gains adoption though!

Party like it's 1999 with Biztalk!

https://en.m.wikipedia.org/wiki/Microsoft_BizTalk_Server

Post reply on HN