371–380 of 390 posts
Earlier quoted context omitted.
Every conversation I've ever had came back to using RedGate SQL Compare to diff databases and TeamCity for CI. You basically shouldn't allow anyone to modify anything without it being scripted (bonus points if it comes with a rollback and is repeatable for testing). Your scripts then all go into Git.
RedGate's SQL Change Automation (formerly ReadyRoll) is pretty slick. DbUp is a good, free alternative. I second the rollback and repeatability bonus. Every script should leave the database in either the new state or the previous good state no matter how many times it's run.
I wonder if there were somewhere a website to describe good idioms to achieve this?
Earlier quoted context omitted.
RedGate's SQL Change Automation (formerly ReadyRoll) is pretty slick. DbUp is a good, free alternative. I second the rollback and repeatability bonus. Every script should leave the database in either the new state or the previous good state no matter how many times it's run.
> Every script should leave the database in either the new state or the previous good state no matter how many times it's run. I wonder if there were somewhere a website to describe good idioms to achieve this?
Having a second person run a script is the best way to ensure no mistakes.
SQL is one the most amazing concepts I've ever experienced. It's nearly 5 decades old and there is no sign of a replacement. We've created countless other technologies to store and process data, and we always seem to try to re-create SQL in those technologies (e.g. Hive, Presto, KSQL, etc). I run a early stage company that builds analytics infrastructure for companies. We are betting very heavily on SQL, and Craigs p…
I disagree you can't use SQL for Google search. No engineer has ever made this possible. We've CommonCrawl data but you can't run SQL queries on that data this makes SQL useless. When smart people have figured out how to do that, come back claiming SQL is important.
> Learning SQL once will allow you to re-use it heavily across your career span without having to re-learn. Like all good abstractions, SQL is the practical expression of a mathematical theory. In the case of SQL, you use Zermelo-Fränckel (ZF) set theory to reason about data sets. While it is easy to come up with merely conjectural implementations for haphazardly doing things -- arbitrary trial and error, really -- i…
Introduction
Project:M36 implements a relational algebra engine as inspired by the writings of Chris Date.
Description Unlike most database management systems (DBMS), Project:M36 is opinionated software which adheres strictly to the mathematics of the relational algebra. The purpose of this adherence is to prove that software which implements mathematically-sound design principles reaps benefits in the form of code clarity, consistency, performance, and future-proofing.
Project:M36 can be used as an in-process or remote DBMS.
Earlier quoted context omitted.
This is a very real problem I bang my head against regularly. There just seems no way to achieve all three of readability, maintainability and performance in a large enough SQL codebase. You can piece together views right up until the moment the query planner forgets to push where clauses down. You can wrap a query in a function to guarantee the where clause is evaluated then and there, but now you have to maintain t…
>save the query plan forever is this even possible? Afaik, the query plan can depend on the data distribution and needs to be re-optimized when the data distribution changes.
SQL is a mind bender for me. I do a lot of work on Data, use python and pandas to do a lot of data magic, but the problem with me is my mind is too procedural in thinking. - Step 1 - Step 2 - Loop through results in Step 2 - Curate and finish output. I try very hard to transform the above steps into an SQL statement spanning multiple tables, but always fail and I usually fallback to python for manually extracting and…
Earlier quoted context omitted.
Your counter counter argument on not being as flexible as SQL is spot on. A DAL that sits in front of storage mechanisms is not a bad idea though. I've found moderate success in doing the reverse and using communication channels from SQL => services either via NOTIFY or queues. ie. Dispatching an event that signals a non SQL service to do an action. eg saving a file to S3, then updating the database when the action i…
> WRT JSON, please for the love of god no. I don't like it either, but for most scenarios that this sort of thing works for a RESTful API or microservice or equivalent (or whatever you want to call it these days since I think the expiration date on those terms has elapsed) and not all data can be meaningfully coded without a structure like JSON or XML. Or, rather, you can , but, you're reinventing the wheel just like…
Not saying microservices don't work, they are great for specific use cases, but 9/10 people just want to microservice everything just to say they use microservices.
> If you're USPS and you're providing an address and ZIP code resolver
Not sure I see your point with this example - isn't a resolver like this just a fixed database of entries?
Earlier quoted context omitted.
> WRT JSON, please for the love of god no. I don't like it either, but for most scenarios that this sort of thing works for a RESTful API or microservice or equivalent (or whatever you want to call it these days since I think the expiration date on those terms has elapsed) and not all data can be meaningfully coded without a structure like JSON or XML. Or, rather, you can , but, you're reinventing the wheel just like…
Having worked with three large JSON http microservice based projects in the past, I abhor the day they got popular. Relative productivity has definitely gone down due to the project overheads caused by them. Not saying microservices don't work, they are great for specific use cases, but 9/10 people just want to microservice everything just to say they use microservices. > If you're USPS and you're providing an addres…
Until you get to how well it handles misspellings and incomplete information. Addresses are also notoriously difficult to parse. There is some logic and ranking at work behind the scenes. The only times I've seen it consistently fail are when the city is incorrect, or it's a genuinely new address (new construction or address renumbering).
In any event, is a read-only service somehow less of a service? I'd wager read-only services see a lot higher demand than anything.
Earlier quoted context omitted.
I think what your are doing is fine. SQL is strongest at answering questions not processing data. I think metabase has the right approach: What question do you want to ask your data? If you want to process and transform your data, I think your tools you are using are great for that.
Not trying to play word games, but what is the difference between answering a question and processing data? Aren't they effectively the same? Using another tool for processing data often results in recreating SQL mechanics at application level. E.g. select this data, retrieve it, loop and if this, then set that, etc. SQL does it way better, guaranteed. Of course that's often required for technical reasons (scalabilit…