81–90 of 142 posts
Incidentally does anyone have resources for SMALL data? EG a few MB of a time, but requires the same ETL, scheduling, traceability. I'd love some lite versions of big-data tools but needs to be simple, small and cheap.
Incidentally does anyone have resources for SMALL data? EG a few MB of a time, but requires the same ETL, scheduling, traceability. I'd love some lite versions of big-data tools but needs to be simple, small and cheap.
Take a look at our https://www.easydatatransform.com tool. It is a drag and drop data munging tool for datasets up to a few million rows. It runs locally on Windows or Mac. You should be able to install it and start transforming your data within a few minutes. It doesn't have a built in scheduler (yet), but you can run it from the command line. Excel Power Query is also quite lightweight. But is pretty klunky in my (…
I think learning Scala is a bit of a waste of time, but I don’t know everyone’s stack. Maybe it’s a west coast bubble, but serverless seems to be the most popular choice for new ETL stacks even if the rest the cloud tech stack isn’t serverless. AWS tools like kinesis, glue (pyspark), step functions, pipelines, lambdas, etc. If you are working in that domain, being able to use the CDK in TypeScript becomes way more im…
Glue is both more of a pain in the butt than regular old spark with pyspark and way more expensive, from my experience I would seriously question someone suggesting to use it. We could have been using it wrong, but porting our Glue scripts to standard EMR after our initial POC saved us over 10x the cost and it was substantially faster.
https://aws.amazon.com/blogs/aws/aws-glue-version-2-0-featur...
Incidentally does anyone have resources for SMALL data? EG a few MB of a time, but requires the same ETL, scheduling, traceability. I'd love some lite versions of big-data tools but needs to be simple, small and cheap.
In AWS, lambda’s and step functions. https://aws.amazon.com/step-functions/
Data Engineering really comes down to being a set of hacks and workarounds, because there is no data processing system which you could use in a standardized systematic way that data analysts, engineers, scientists and anyone else could use. It's kind of a blue-collar "dirty job" of the software world, which nobody really wants to do, but which pays the highest.
There are of course other parts to it, such as managing multiple data products in a systematic way, which engineering minds seem to be best suited for. But the core of data engineering in 2020, I believe, is still implementing hacks and gluing several systems together so as to have a standardized processing system.
Snowflake or Databricks Spark bring you closest to the ideal unified system despite all their shortcomings. But still, you sometimes need to process unstructured jsons, extract stuff from html and xml files, unzip a bunch of zip archives and put them into something that these systems recognize and only then you can run sql on it. It is much better than the ETL of the past, where you really had to hack and glue 50% of the system yourself, but it is still nowhere near the ideal system in which you'd simply tell your data analysts: you can do it all yourself, I'm going to show you how. And I won't have to run and maintain a preprocessing job to munge some data into something spark recognizable for you.
It is not that difficult to imagine a world where such a system exists and data engineering is not evem needed. But you can be damn sure, that before this happens, that this position will be here to stay, and will be paying high, when 90% of ML and data science is data engineering and cleaning and all these companies hired a shitton of data science and ML people who are now trying to justify their salaries by desperately trying to do data engineers' job.
I think it's missing the resources to one of the hardest sections: Data modelling, like Kimball and Data Vault. That, and maybe a section to modern data infrastructure. I'd put a link to [1] and [2] for a quick overview and probably [3] for more detail. [1] https://www.holistics.io/books/setup-analytics/ [2] https://a16z.com/2020/10/15/the-emerging-architectures-for-m... [3] https://awesomedataengineering.com/
For example we use Vertica and DBA told us that Vertica loves wide tables with many columns, which doesn't look very Kimball to me. This gives me some trouble as I'm not really show how to model data properly.
Earlier quoted context omitted.
What languages are worth learning?
SQL has taken over the space completely. 90% of data munging and transforms happen via SQL. I would learn python. Its the number one language outside sql.
I think it's missing the resources to one of the hardest sections: Data modelling, like Kimball and Data Vault. That, and maybe a section to modern data infrastructure. I'd put a link to [1] and [2] for a quick overview and probably [3] for more detail. [1] https://www.holistics.io/books/setup-analytics/ [2] https://a16z.com/2020/10/15/the-emerging-architectures-for-m... [3] https://awesomedataengineering.com/
I think it's missing the resources to one of the hardest sections: Data modelling, like Kimball and Data Vault. That, and maybe a section to modern data infrastructure. I'd put a link to [1] and [2] for a quick overview and probably [3] for more detail. [1] https://www.holistics.io/books/setup-analytics/ [2] https://a16z.com/2020/10/15/the-emerging-architectures-for-m... [3] https://awesomedataengineering.com/
This. I also think modern columnar databases and other techniques somehow makes Kimball to be obsolete or relaxed some how, but I could be very wrong. For example we use Vertica and DBA told us that Vertica loves wide tables with many columns, which doesn't look very Kimball to me. This gives me some trouble as I'm not really show how to model data properly.
I have heard advice like this from colleagues and frankly I don't buy it. It certainly isn't gospel. I think it's an oversimplification.
Columnar stores love star schemas. You can get away with a single table model too but you still need some kind of dimensional or at least domain-based thinking. Your single table is going to basically be a Kimball model but already joined together.
No database is going to be happy with joining orders and billing. The single table is still just going to be a single fact table, you just degenerate all the dimensions.
Personally I think you can gain a lot of benefit from doing proper stars because you get more sorting options but I'm a Redshift guy so maybe I'm stuck in that headspace.
I'm still waiting for someone to come along and propose something different but honestly Kimball's dimensional mental model still resonates with me. Are there compromises, can you relax the model more? Of course, but you're still going to realize huge benefits from starting with that approach. I don't think there is some "new" way of thinking that really changed the data space. All the innovation is on the compute side.
I have precisely zero Vertica experience so maybe I'm totally missing something. I'd be happy for someone to tell me I'm wrong.