51–60 of 77 posts
I've just been spinning up new C# console projects in VS and pulling down Dapper to do most nasty ad-hoc things between databases. I never bother to save any of these to source control because the boilerplate is negligible and its usually a one-time deal.
Is this an advertisement?
In an early startup, (< 4 engineers in my case) I can't imagine using anything but a skanky script at the outset. We went broke before we needing a framework. Needed more engineering time on other things.
Can any of these ETL frameworks kick off an ETL script without a rewrite? Something that would handle scheduling, retries, emit metrics around those actions, but let me use my own tools for the actual data manipulation.
We use Jenkins for essentially this right now, and it works fine (especially if you have someone who knows Jenkins already).
The only ETL framework you need is a general-purpose programming language. Cf. that article "You can't buy integration" from the other day.
ETL was actually a new acronym for me: Extract, Transform, Load. https://en.m.wikipedia.org/wiki/Extract,_transform,_load
This is the data lake principle: you firstly dump all data verbatim into one coherent data storage. (data ingestion phase)
The transform will eventually be done by the data scientists (data curation phase).
The data curation is really difficult as it requires in-depth knowledge of the data, removing of slack, creation and lookup of reference data, data normalization for the datawarehouse schema, etc.
The data ingestion is relatively easy task, because the dirty data is just collected from upstream databases without any processing/conversion.
I believe data lakes is kind of "cheating" by IT departments.
Look at how Prefect does that (I know they are well along the path), but sonething is missing.
I've always found ETL frameworks to have their own problems. They seem great on paper but usually they don't account for a specific source system, APIs, applications, data size, data distribution or scheduling situations. If your project is using it then developers end up hacking the frameworks instead of writing simple code that does the specific thing they need to do. Before you know it you have super long and supe…
I agree. I've used AWS Data Pipelines for some jobs but there is a steep learning curve. It is good for launching servers on demand to run your ETL jobs if you need that. The best solution I have found is writing ETL scripts in Laravel which I use for most projects anyway. The framework has built-in scheduling and error reporting.