Data engineer sounds like a great second remote job to pick up. If you already have a remote software engineering job, picking up an entry level data engineer role on the side will allow you to build up that resume in parallel, without having to worry about the lower salary. It’s an investment and gives you a second career branch to eventually fall back on if you get tired of software engineer, or to eventually just have two different six figure jobs with plenty of skill crossover.
There is no data engineering roadmap
41–50 of 133 posts
Re: There is no data engineering roadmap
#42This person is essentially correct.
Re: There is no data engineering roadmap
#43> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…
There's plenty of support for testable SQL. There's just a long tradition of not using it.
Re: There is no data engineering roadmap
#44Can't forget about SQL. It's also worth noting that designing any central data system without a SQL interface is probably a mistake. The people making it may be more comfortable with a REST interface or Python API, but data users will show up and demand that data be SQL-accessible. So now you get to hack it in, or help them develop their own shadow database that is poorly replicated from the main system.
The author stated that the only logical starting point is to learn to solve problems in a dialect of SQL then approach problems of the same type in other dialects of SQL. You’re entirely right, and the author seems to agree to an extent.
Re: There is no data engineering roadmap
#45I think there's a fair caveat here: In a context free situation , sure, just learn SQL. It's the workhorse, and you'll definitely use it. If you have to start somewhere, start there. ...buuuut, obviously look at job ads. If all the junior jobs say "experience with SnowPanda" or "Working knowledge of FancyCloudPlatform" or whatever the name of the latest hotness is, you'd be daft not to invest some time and effort int…
This also works when reviewing resumes. If you see enough candidates list xyz as a skill, then it's probably worth looking into.
Re: There is no data engineering roadmap
#46Earlier quoted context omitted.
Interesting. I work in the space currently (an ELT company) and I wouldn’t characterize the skill sets you mentioned as critical to a data team or a data driven organization. They are helpful, sure, but a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform…
>a lot of the problems that data teams face in my experience are around how to expose data to the organization in a meaningful, regular, digestible, actionable way. This means thinking about how to ingest and transform data and present it to end users for analysis (BI) in such a way that it’s not painful or arduous to consume it, and ensuring that it is clean and accurate 100% this. I basically work in this role in m…
One really interesting aspect of this that I’ve had some revealing conversations with the data engineers in my org is how to best expose the firehose of data for people in BI tooling. We use dbt in my org, and Metabase as the BI tool, and a lot of thought is put into how to create a clearinghouse that serves the needs of the organization. The current pattern that has been of interest is to ELT into what the data engineers call OBT (one big table). The OBT is cleaned, denormalized, and able to be sliced on. An org might have several of these OBT consisting of various areas of interest. End users then import the OBT in Metabase to drive their filtering and build dashboards. The goal is to reduce reliance on custom SQL scripts and push all of that custom slicing and dicing of the data into Metabase’s front end logic where filtering can be applied dynamically rather than trying to maintain a bazillion sql variants.
Eventually I think we will move into a post ChatGPT world where you’ll give ChatGPT (or whatever equivalent) your schema and a question and it will output the dashboards for you. We aren’t quite there yet though
Re: There is no data engineering roadmap
#47> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…
> Data engineering can be unsatisfying if you thrive on writing reliable systems This is what I needed for me to stop considering data engineering. Not that I was thinking about it a lot. I feel very happy with my current job but the thought of the code I write being so brittle as to paralyze me from making any changes terrifies me.
Unless you get to work at the place that values your test driven perspective. Either because they are doing it or desperately want to (where want = they are already investing heavily in it)
Re: There is no data engineering roadmap
#48Earlier quoted context omitted.
You made great points. Data Engineering cannot claim to support test-driven development to half the extent of the rest of software engineering. Frequently DE projects have no concept of Dev, Test, Staging and then Prod. Quite often its dev and then straight to Prod. SQL of course is to partly or fully blame for this. My last job for an large insurance company, they happily set a best practices of 90% test coverage (w…
In my experience another big problem is that it's just expensive. For Spark jobs for instance, it's very common for it to run fine on the small test dataset on your laptop but then when you release it to run for 2 days on the prod dataset you end up having to do tuning there with pretty long turnaround times. And that then extends to why staging and even dev aren't very useful - the scale is part of the equation. Eve…
You seem to be touching on two disparate issues that I find very interesting to tackle.
One is testing performance of code (SQL or otherwise) and ensuring it achieves a particular level of performance or ensuring no regressions. This is a problem not just with automatically unit-testing SQL, but any other code. In general, for regular code, we simply go with special, "manual" tests of the execution time for critical pieces (the biggest issue is fragility or flakiness: due to changing conditions a test runs under, speed is not always stable). We could do similar for testing SQL performance. If we know the database we are targetting, we could also use some of the introspection tools it offers to get even better tests (eg. we could run an "EXPLAIN (FORMAT JSON)" query on a very large Postgres database matching production to ensure right indexes are being hit and no seqscans are being done and Postgres' estimation of the time is on target).
Basically, so far it's hard because "performance" means so many different things, but I don't think it's impossible.
As for the other point, those 1-in-a-million edge cases, we've got those with regular code too! If you really want to test something against production-like DB, it's not hard (make an anonymised replica of prod DB), it's just expensive (tests will be slow, getting this set up will be slow, etc).
I personally believe the right balance for cases like those is to catch them in production: automated tests should be quick and allow quick iteration if someone wants to do TDD on any part of the codebase. There are certainly product niches where this is not true (let's not have airbags in cars deploy accidentally every 1M rides, because there's a lot more than 1M rides daily :)), but for our regular applications, that's usually more than fine: quick tests will make it easy to fix the particular edge case once we hit it.
FWIW, I love the idea of pgTAP for those who haven't seen it too.
Re: There is no data engineering roadmap
#49Telling data engineers to learn sql because it's what's used in BoringCorp is a bit like telling front end developers in 2003 to learn VB because it's what most user interfaces are written in. Data engineering which pays well happens exactly where databases fail and you need bespoke solutions.
They’re speaking to entry-level people, who will not be designing bespoke solutions. SQL is exactly the right recommendation.
In the same way that VB was to front end developers 20 years ago.
Re: There is no data engineering roadmap
#50> All you need is l̶o̶v̶e̶ SQL I’ve ended up doing a lot of data engineering over the years, because I have a background in low-level search/databases systems coding and know text well. I have mixed feelings about the field precisely because it’s so SQL dominated. Data engineering can be unsatisfying if you thrive on writing reliable systems. There’s a whole lot of big-ball-of-inscrutable-SQL work out there which whe…
There's plenty of support for testable SQL. There's just a long tradition of not using it.
Just yesterday I installed a bunch of TSqlT tests on a test SQL Server database for our subsidiary, and watching unit testing written in SQL run inside of SSMS made me realize just how massively I miss this ability. There is absolutely no reason we should be able to test API calls and not the database calls underneath.