51–60 of 70 posts
The correct language for querying data is, as always, SQL. No one cares about the implementation details. “I have data and I know SQL. What is it about your database that makes retrieving it better?” Any other paradigm is going to be a niche at best, likely outright fail.
Earlier quoted context omitted.
20 years ago SQL lacked type safety, testability, and composability. Today the same is true. I doubt it will be different 2 decades from now. SQL is powerful. It is also very old and has very large warts.
All data query languages eventually reduce themselves into SQL, or something equivalent to it.
(2nd user & developer of spark here). It depends on what you ask. MapReduce the framework is proprietary to Google, and some pipelines are still running inside google. MapReduce as a concept is very much in use. Hadoop was inspired by MapReduce. Spark was originally built around the primitives of MapReduce, and you see still see that in the description of its operations (exchange, collect). However, spark and all the…
Are you confusing kafka with something else? Kafka is a persistent write append queue.
The "streaming systems" book answers your question and more: https://www.oreilly.com/library/view/streaming-systems/97814... . It gives you a history of how batch processing started with MapReduce, and how attempts at scaling by moving towards streaming systems gave us all the subsequent frameworks (Spark, Beam, etc.). As for the framework called MapReduce, it isn't used much, but its descendant https://beam.apache.o…
This book looks interesting, should I buy it or does anyone else have newer recommendations? I have Designing Data-Intensive Applications which is a fantastic overview and still holds up well.
It's a little biased towards Beam and away from Spark/Flink though. Which makes it less practical and more conceptual. So as long as it's your cup of tea go for it.
Earlier quoted context omitted.
Spark is really failing, all right. SQL lacks type safety, testability, and composability.
It’s crazy to think how old I am now. But give it 20 more years and you’ll come around.
Working with 1000+-line SQL scripts written by other people is no fun. Why wouldn't you want to decompose that into legible, testable functions using an expressive language like Scala?
Earlier quoted context omitted.
20 years ago SQL lacked type safety, testability, and composability. Today the same is true. I doubt it will be different 2 decades from now. SQL is powerful. It is also very old and has very large warts.
All data query languages eventually reduce themselves into SQL, or something equivalent to it.
Earlier quoted context omitted.
Being old doesn’t automatically make you more right. You don’t get wisdom as a birthday gift.
no, but you do get experience, as in, when last somebody “invented” this idea thirty years ago, it was a total crapshoot, so wonder what’s changed? oh right, new language. that’ll definitely fix it. :eyeroll:
Earlier quoted context omitted.
All data query languages eventually reduce themselves into SQL, or something equivalent to it.
Ok? Even if that were true — and I’m not entirely sure how you would even prove that — SQL still lacks type safety, testability, and composability.
Testability - you use a general purpose language to execute SQL. Again, I don't know what you mean.
Composability - I suppose, but remember SQL is a language to retrieve data. I reuse fragments everywhere in a general purpose language.
Earlier quoted context omitted.
20 years ago SQL lacked type safety, testability, and composability. Today the same is true. I doubt it will be different 2 decades from now. SQL is powerful. It is also very old and has very large warts.
I think one of the biggest missed opportunities in language design is the integration of powerful relational database and query models directly into a modern language. Not as a bolt-on or an ORM but as a first class part of the language in the same way as maps and arrays. Make a language that deals with data relationally and where relational queries are a core part of the language and relational query execution is ba…
Once you start getting fancier in your files, and the data grows large, you need special ways to read it. A Postgres database can be considered a single big file on disk. It is the Postgres server that is required to access the file in the most efficient way to store and randomly access enormous amounts of general data.
SQLite is interesting in that there is no server, it's just a special library that enables efficient random access of a single file, which can be thought of as a black box that only SQLite knows how to interpret.
Unless you mean, making something like SQL built directly into the language as a first class citizen. Mumps did something like this https://en.wikipedia.org/wiki/MUMPS
Earlier quoted context omitted.
All data query languages eventually reduce themselves into SQL, or something equivalent to it.
So use a better language, and let the compiler optimize it??