101–110 of 142 posts
Earlier quoted context omitted.
Agree 100% with this comment, Old stack: Hadoop, spark, hive, hdfs. New stack: kafka/kinesis, fivetran/stitch/singer, airflow/dagster, dbt/dataform, snowflake/redshift
Huh, what replaces Spark in those lists? For my money, its the best distributed ML system out there, so I'd be interested to know what new hotness I'm missing.
(source for everything following: I recently hired entry-level data engineers) The experience required differs dramatically between [semi]structured transactional data moving into data warehouses versus highly unstructured data that the data engineer has to do a lot of munging on. If you're working in an environment where the data is mostly structured, you will be primarily working in SQL. A LOT of SQL. You'll also n…
As many advantages as SQL has, in many cases it gets into the way. The closer you move to moving data (instead of doing analysis), the more it becomes annoying.
On the other hand, current languages (such as python) lack support when it comes to data transformations. Even Scala, which is one of the better languages for this, has severe drawbacks compared to SQL.
Hopefully better type-systems will help us out in the long term, in particular those with dependent types or similar power to describe data relations.
(source for everything following: I recently hired entry-level data engineers) The experience required differs dramatically between [semi]structured transactional data moving into data warehouses versus highly unstructured data that the data engineer has to do a lot of munging on. If you're working in an environment where the data is mostly structured, you will be primarily working in SQL. A LOT of SQL. You'll also n…
We yet have to wait for the proper sweet spot: a language that allows SQL-like handling without the restrictions of SQL. As many advantages as SQL has, in many cases it gets into the way. The closer you move to moving data (instead of doing analysis), the more it becomes annoying. On the other hand, current languages (such as python) lack support when it comes to data transformations. Even Scala, which is one of the…
Earlier quoted context omitted.
> Snowflake has support for custom Javascript UDFs and a lot of built in features (you can do absurd things with window functions). I also found it much faster than Spark. UDF support isn't really the same, to be honest. You're still prisoner of the select from pattern. Don't get me wrong, SQL is wonderful where it works, but it doesn't work for everything that I need. I completely agree that it's faster than Spark,…
>UDF support isn't really the same, to be honest. You're still prisoner of the select from pattern. Don't get me wrong, SQL is wonderful where it works, but it doesn't work for everything that I need. Not sure how it's different from what you can do in Spark in terms of data transformations. Taking a list of objects as an argument basically allows your UDF to do arbitrary computations on tabular data. > I forgot abou…
What? The Apache stack that's written in Scala recompiles all your code into JVM bytecode, regardless of what language you've written it in. Yes, that includes Scala. Spark isn't actually firing up a python interpreter and running your python code on the data.
Earlier quoted context omitted.
We yet have to wait for the proper sweet spot: a language that allows SQL-like handling without the restrictions of SQL. As many advantages as SQL has, in many cases it gets into the way. The closer you move to moving data (instead of doing analysis), the more it becomes annoying. On the other hand, current languages (such as python) lack support when it comes to data transformations. Even Scala, which is one of the…
What's your opinion of LINQ in C#? It's been a while since I've used it but to me it seems like one of the most powerful ways to manipulate data inside of a language.
Where it lacks is abstraction. To make that more concrete, let me ask: can you write LINQ that takes an arbitrary structure and selects every numeric field and calculate the sum over it? And if that is not possible, it should not compile.
I.e. can you define a function "numericsSum(...)" and call it with "List(User(salary: Number, deductions: Number, name: String))" and have it calcuate the sum (ignoring the name-field) but having it fail to compile when calling with "List(User(name: String, date_of_birth: Date))"?
Another example: is it possible to create your own special "join" function, that joins to data structures if they have exactly one common field (which you don't have to specify)?
In both examples, the LINQ compiler must be able to inspect the structure of a type (such as User) at compile time (not runtime) and do pretty arbitrary calculations. Most languages don't support that and I think even LINQ only works with concrete types in that sense. Which, by the way, is already better than what most languages offer - don't get me wrong here. But it is not as powerful as what SQL "compilers" offer - however those are then limited to SQL only, lacking the ability to do general purpose programming.
Amazon introduced Step Functions, which are very nice to dig into and a helpful skill for Data Engineering.