281–290 of 322 posts
So I'm not an expert, but for simplistic use cases I merely make use of https://github.com/oguimbal/pg-mem It's a lot faster and easier than dealing with containers and the like.
Earlier quoted context omitted.
Spark makes it easy to wrap SQL in functions that are easy to test. I am the author of the popular Scala Spark (spark-fast-tests) and PySpark (chispa) testing libraries. Some additional tips to speed up Spark tests (can speed up tests between 70-90%): * reuse the same Spark session throughout the test suite * Set shuffle partitions to 2 (instead of default which is 200) * Use dependency injection to avoid disk I/O in…
That's super useful, thanks. Could you expand on the 'Use dependency injection to avoid disk I/O in the test suite' point please - I'm not sure I understand what it means but it sounds interesting!
You can structure your code to read from paths when run in the production environment, but inject DataFrames you build in memory for your test suite. Spark is designed to read multiple files in parallel, so it's not optimized to read a single tiny file. That's why it's best to avoid I/O in Spark test suites whenever possible.
This way i don't waste time with unit tests that quickly get old and no one wants to maintain and run
Try and write any complex SQL as a series of semantically meaningful CTEs. Test each part of the CTE pipeline with an in.parquet and an expected_out.parquet (or in.csv and out.csv if you have simple datatypes, so it works better with git). And similarly test larger parts of the pipeline with 'in' and 'expected_out' files. If you use DuckDB to run the tests, you can reference those files as if they were tables (select…
Earlier quoted context omitted.
I’ve not encountered your definition of declarative before. SQL is often cited in cs texts as an example of a declarative language. That said, SQL does have a lot of imperative features, but those features are used to declare the result of the sql dml query.
SQL does not make sense as a declarative language to me. How you can say that SQL and HTML is the same by nature? Also I can't see why we can't argue about this against the literature. Saying "you are wrong because 20 years old textbook say you are wrong" is not an argument for me
Can you please expand on what you mean by DBT? DBT, specifically, DBT-2 is a suit of tests designed to benchmark a database system . These tests aren't interested in, eg. correctness of an application that is using the database. They are meant to be testing the system as a whole by modeling some sort of an "average business" and defining some sort of an "average business operation" and estimating how many of such ope…
Oh, someone else below suggested that DBT in OP may be this: https://docs.getdbt.com/ . Well... I don't know anything about this tool, but from cursory reading, this is the correctness kind of testing that is meant to examine whether the way you defined the schema actually leads to the results you expect in the data. Well... it's not an interesting kind of testing, at least not for me. So, I don't know much about it.…
Earlier quoted context omitted.
I agree about the limitations of ORMs. However I have had great luck with using an ORM to load up the database and data, and then having a unit test that calls the function which does raw SQL in the middle. And now the raw database tests are integrated with the unit tests for the rest of the environment in a way that keeps them synchronized with the application code that also interacts with the same database. And, of…
There’s ORMs and there’s ORMs - at one end you have the (reprehensible) Active Record anti-pattern, at the other end you have EF Core extended with one’s own build-time type generation - they’re both “ORMs” to everyone involved, but they’re totally incomparable. …not to say they that EF Core doesn’t have flaws (it does, and they’re legion) but the ORMs of today are nothing like the ORMs of the 1990s… or even like 201…