71–80 of 322 posts
This approach didn't use an ORM and run the tests concurrently against the same database. I follow those steps on my pipeline: Every time I commit changes the CI/CD pipeline follow those steps, on this order: - I use sqitch for the database migration (my DB is postgresql). - Run the migration script `sqitch deploy`. It runs only the items that hasn't been migrated yet. - Run the `revert all` feature of sqitch to chec…
We spin up a docker container running the DB technology we use, run our DB migration scripts on it, and then run integration tests against it. You get coverage of your migration scripts this way too.
would love to do this, but how does one spin up a redshift cluster inside of a docker container?
jokes aside, redshift is based on pg^1, you can try an older version to get some semblance of it running locally.
1. https://docs.aws.amazon.com/redshift/latest/dg/c_redshift-an...
This approach didn't use an ORM and run the tests concurrently against the same database. I follow those steps on my pipeline: Every time I commit changes the CI/CD pipeline follow those steps, on this order: - I use sqitch for the database migration (my DB is postgresql). - Run the migration script `sqitch deploy`. It runs only the items that hasn't been migrated yet. - Run the `revert all` feature of sqitch to chec…
Earlier quoted context omitted.
SQL is not a declarative language. It is a functional language, and structured language on the top as extensions. HTML is a declarative language.
You tell SQL what you want, not how to get it. That's declarative. SQL : CSV :: GraphQL : JSON :: React : HTML
SELECT CASE WHEN employee.type = 'contract' THEN salary CASE WHEN employee.type = 'full-time' THEN salary + benefit_costs END CASE FROM employee
Tell me how is this declarative?
-SQL is functional -CSV is not a language, it's a data format -GraphQL, im not familiar -JSON is literally executable javascript code, arguable -React is a javascript framework, binding is a declarative nature (if it has it, i dont know it too well), but it is not a language, it's a framework -HTML is absolutely, 100% declarative, yes
Earlier quoted context omitted.
Running sqlite in memory as a test db speeds up your test runner as crazy. You can do this if you use an sql query builder library, because it can translate your queries to the specific database.
problem is that its not always compatible with features you use on your production database
Or along similar lines you could divide it such that developers can test things locally on their machines with SQLite, but once it gets pushed into CI (and passes code review etc.) it's tested against the heavy db.
Try and write any complex SQL as a series of semantically meaningful CTEs. Test each part of the CTE pipeline with an in.parquet and an expected_out.parquet (or in.csv and out.csv if you have simple datatypes, so it works better with git). And similarly test larger parts of the pipeline with 'in' and 'expected_out' files. If you use DuckDB to run the tests, you can reference those files as if they were tables (select…
My ignorance of the topic (and experience with a mostly unrelated one) is showing, but all I could think of when you said CTE was "chronic traumatic encephalopathy". This made a lot more sense when you generalized answering the question as if it's a given that Python is necessary (I know that's not your intent, but that's how it comes off). Not much more to say, just observing, sorry if this is irrelevant commentary.
We spin up a docker container running the DB technology we use, run our DB migration scripts on it, and then run integration tests against it. You get coverage of your migration scripts this way too.
would love to do this, but how does one spin up a redshift cluster inside of a docker container?
I have seen teams give each developer a personal schema on a dev cluster, to ensure their Redshift SQL actually works. The downside is that now your tests are non-local, so it's a real tradeoff. In CI you probably connect to a real test cluster.
We spin up a docker container running the DB technology we use, run our DB migration scripts on it, and then run integration tests against it. You get coverage of your migration scripts this way too.
Running sqlite in memory as a test db speeds up your test runner as crazy. You can do this if you use an sql query builder library, because it can translate your queries to the specific database.