Live data from Hacker News

How I write SQL

craigkerstiens.com

81–84 of 84 posts

Re: How I write SQL

#81
post #80

Earlier quoted context omitted.

Sorry for the confusion. When my list of columns is long, I split it up into multiple lines, usually 7-8 per line (where I work, column names are capped at 30 chars, hence the 200 character estimate), unless it is a case statement. In case statements, each case/when clause get its own line, so you'll have: case when condition then result \ when condition_2 then result_2 \ ...\ when condition_n then result_n end as co…

My personal experience when I tried to clump things together is that I lost track of what columns were where. My eyes scan vertically really fast. And once I was used to it for large queries, well, what works for large queries works for small as well if you're used to it. Honestly I don't think about formatting that much anymore, as I've started storing most of my statements as "metrics" which can be easily repurpose…

It would seem we're each doing what works for us.

I work mainly in MySQL and Teradata (which has tmp tables which are called volatile tables), and I do exactly what you describe when creating complex queries. My metric system is just a way to build those temp tables more rapidly.

I use two main functions to manipulate metrics:

  create_metric(metric_name,{cols_added},join_src,{join_cols},{extra_sql},{indices});
This stores the metric for later use.

  add_metric(metric_name, my_other_table, {my_join_conditions});
This retrieves the metric, and returns a string of (Teradata) SQL in the form:

  CREATE VOLATILE MULTISET TABLE add_metric_name AS (

  SELECT a.*, b.cols_added_1, b.cols_added_2,...,b.cols_added_n
  
  FROM my_other_table a

  LEFT JOIN join_src b

  ON a.my_join_condition_1 = b.join_col_1

  AND a.my_join_condition_2 = b.join_col_2

  ...

  AND a.my_join_condition_n = b.join_col_n

  [If there is extra sql, like where a.condition = X, or group by's like group by 1, 2, it would show up here. SQL here can reference the join columns and table name in an add_metric stmt]

  ) WITH DATA PRIMARY INDEX(indice_1, indice_2) ON COMMIT PRESERVE ROWS;
I can also store entire create tmp table chains as metrics, with the last table appending all of the information from that chain to another source (I do this by storing the chain as preparatory sql, which is run before the create add_metric_name statement.

It also allows me to search all of my metrics on a number of different dimensions: the common name of the information I am adding (metric name), column names, tables, join conditions (particularly useful - It helps you map how you'll get from one metric to another), indices, or any combination of the above. For example I can find all metrics that have the word phone in the metric name and are joinable on user_id.

I'm aware of Oracle's lack of tmp tables. My fiancée has to use Oracle SQL at work, and I quickly discovered its lack of tmp tables when trying to help her solve a SQL issue.

Re: How I write SQL

#82
post #80

Earlier quoted context omitted.

My personal experience when I tried to clump things together is that I lost track of what columns were where. My eyes scan vertically really fast. And once I was used to it for large queries, well, what works for large queries works for small as well if you're used to it. Honestly I don't think about formatting that much anymore, as I've started storing most of my statements as "metrics" which can be easily repurpose…

It would seem we're each doing what works for us. I work mainly in MySQL and Teradata (which has tmp tables which are called volatile tables), and I do exactly what you describe when creating complex queries. My metric system is just a way to build those temp tables more rapidly. I use two main functions to manipulate metrics: create_metric(metric_name,{cols_added},join_src,{join_cols},{extra_sql},{indices}); This st…

Yup, they look like similar solutions to similar problems.

One of the things that I built into reports at that location was the ability to see all of the tmp tables that had been created, and the ability to stop the report on any particular one and display that. I built this as a debugging aid for myself, but was quite surprised when finance came to me one day and said, "Report X is going wrong on step Y - it looks like you're filtering out duplicate records."

I like having users that will debug my stuff. :-)

Re: How I write SQL

#83

Earlier quoted context omitted.

If you're using an ORM, and you almost certainly are, it's equal to all the other work you do to map your objects. If you're querying directly, array_agg on the way out is the same work as unnest is on the way in. You can have your cake and eat it to. Tagging has been discussed a lot recently as a good excuse to use arrays in Postgres, but since the whole point of tagging is to enhance searching I think it really mis…

> If you're using an ORM, and you almost certainly are I'm not a web developer - I spend more of my time in raw-sql-land, so this discussion is largely academic to me :-). That said, even if you're using an ORM, it's presumably not exactly free for the ORM to produce an object out of the thousands of distinct rows you can end up getting if you have multiple one to many relationships. In general, the availability of a…

Without some numbers I see no reason to believe that Postgres would be more efficient at transporting arrays than rows. One of those is a highly optimized use-case that literally everybody relies on every day. The other one is a corner case. I'm not saying it's impossible but I'm going to need more than a good feeling to buy it.

Re: How I write SQL

#84

Earlier quoted context omitted.

> If you're using an ORM, and you almost certainly are I'm not a web developer - I spend more of my time in raw-sql-land, so this discussion is largely academic to me :-). That said, even if you're using an ORM, it's presumably not exactly free for the ORM to produce an object out of the thousands of distinct rows you can end up getting if you have multiple one to many relationships. In general, the availability of a…

Without some numbers I see no reason to believe that Postgres would be more efficient at transporting arrays than rows. One of those is a highly optimized use-case that literally everybody relies on every day. The other one is a corner case. I'm not saying it's impossible but I'm going to need more than a good feeling to buy it.

Perhaps my wording was unclear - I'm not talking about postgres' efficiency, but efficiency of processing the data that postgres produces.

Consider the example of an ORM that's storing an object with three multi-valued properties on the object it's encoding, each with 100 values. Encoding one has a row with three arrays in it, encoding two is the result set from joining to separate tables. In the end, it's quite plainly obvious that it's going to be cheaper (and, frankly, easier) to construct the object from a row with three arrays of size 100 in it than it is from a million row result set. Yes, this is absolutely a corner case (particularly with many multi-valued props of such a cardinality), but I'm not really sure why it requires numbers to prove - unless there's some kind of JDBC/ODBC technique for dealing with this situation that I'm simply unaware of?

Post reply on HN