21–26 of 26 posts
Does anyone have a list of high quality data engineering, that aren't free? I don't mind paying, but there is so much out there and most of it is not great.
or you can also join our full-time/part-time course https://www.dataengineering.academy/curriculum
Can anyone vouch for the quality on this site? I've been looking for data engineering resources for my team, but don't want to share anything shoddy.
Looking over the resources, it doesn't seem worthwhile. The difficult part is not learning how to code, or work with SQL. The hard part is learning the platform and tooling you need to operate at scale. The ecosystem is full of tools that are great for certain workloads, but terrible for others. Your best bet is to start by getting an overview of the tools available for your team. If you're using AWS, GCP, or Azure,…
We don't see any new projects starting out with Airflow, Beam, Storm or Hadoop, so that's not a good choice for anyone, only if your plan is to keep horrible legacy stacks alive as a freelancer.
Let’s say you are the Lead Data Engineer of a small data-driven company. You need to define a strategy, pick a stack of tools, decide how data is going to be stored and normalised, what the workflow will be from ad-hoc, exploratory studies to productionized inference. Are there any good resources out there that are useful to this person? I am this person right now and I need to find some good guidance.
Let’s say you are the Lead Data Engineer of a small data-driven company. You need to define a strategy, pick a stack of tools, decide how data is going to be stored and normalised, what the workflow will be from ad-hoc, exploratory studies to productionized inference. Are there any good resources out there that are useful to this person? I am this person right now and I need to find some good guidance.
I am that person too, only for a big (non-data) money driven company :-) So many resources that you can easily get lost. Martin Kleppmann's Designing Data Intensive applications can be your starting point. It helps you establish a basic to advanced understanding on quite a few of the concepts that will be coming up and some key principles to drive your strategy from a technical perspective (you'll need a few facets o…
Let’s say you are the Lead Data Engineer of a small data-driven company. You need to define a strategy, pick a stack of tools, decide how data is going to be stored and normalised, what the workflow will be from ad-hoc, exploratory studies to productionized inference. Are there any good resources out there that are useful to this person? I am this person right now and I need to find some good guidance.
You may want to look at the guidance from a16z. https://future.a16z.com/emerging-architectures-modern-data-i... They are basically recommending picking one of the big three: Snowflake, Bigquery, or Redshift. Using standard connectors. Once your data is in one of the big three, you can use any analysis tool. I have been involved in this space for some time and am heavily involved with Salesforce. In my corner of the w…
Earlier quoted context omitted.
Looking over the resources, it doesn't seem worthwhile. The difficult part is not learning how to code, or work with SQL. The hard part is learning the platform and tooling you need to operate at scale. The ecosystem is full of tools that are great for certain workloads, but terrible for others. Your best bet is to start by getting an overview of the tools available for your team. If you're using AWS, GCP, or Azure,…
This is a great reply! The greatest difficulty I face is convincing my data engineers to stop reinventing the wheel and leverage existing (and appropriate!) tools. Writing code to directly manage events and batch jobs should be a thing of the past by now. Pick a tool. Configure your jobs, retry policy, etc. and be done with it.