Live data from Hacker News

Show HN: Csql – Python lib for composeable SQL queries

github.com

31–40 of 41 posts

Re: Show HN: Csql – Python lib for composeable SQL queries

#31
Advice on the Parameters class: implement attribute access (__getattr__) in addition to __getitem__. Compare

  f"""... WHERE created_on > {p['created_on']}} ..."""
to

  f"""... WHERE created_on > {p.created_on} ..."""
The latter is a lot more readable and typeable, especially when it's meant to be used in f-strings which place additional restrictions on the usage of quotes.

Re: Show HN: Csql – Python lib for composeable SQL queries

#33

This project seems nice, but there's some detail I have trouble wrapping my head around: why is a template string represented as a lambda containing an f-string, instead of as a plain string? Namely, wouldn't the code below suffice? Q("""select 1 from {otherQuery}""") What does the following code enable that the code above cannot do? Q(lambda: f"""select 1 from {otherQuery}""")

It's impossible to hook into Python's string interpolation system to the degree required for the first to work. JS and Julia can do it, e.g. in JS (with typescript annotations) it'd just be a matter of defining function Q(stringBits: string[]): Query { for (bit in stringBits) { if (bit is string) { add to sql } else if (bit is Query) { add bit to dependencies add bit.name to sql } } } But this cannot be done currentl…

I'm pretty confident f-strings use str.format under the hood, so instead of AST mambo-jumbo, you can do just query.format(var1="something") or something.

Re: Show HN: Csql – Python lib for composeable SQL queries

#34

Earlier quoted context omitted.

Can you say more about that? I though SQLAlchemy core goal was exactly that. When does it fails, compared to Csql?

My desire was to be able write real SQL directly in the dialect of my database. SQLAlchemy can't give me that, instead I'd need to learn a special SQLAlchemy-specific query builder syntax that probably won't support all the analytical functions I want to use anyway. It's really a whole different beast to csql.

From what I understand, not only SQLA API does provide support for all functions you wish to use, but it provides also raw SQL escape hatch.

What kind of limitations did you encounter in the past?

Re: Show HN: Csql – Python lib for composeable SQL queries

#35

As an analyst, I often am faced with a choice between - write a giant unmaintainable SQL query, or - pull everything to my PC and use pandas, to take advantage of its ability to build up results piece by piece. I wrote this library to try and enable that piece-by-piece development approach with SQL queries, without resorting to the mental overhead of full on query builders like SQLAlchemy or Linq. Seeking feedback -…

Does the unit of composition have to be a complete query/CTE? Theoretically it seems like it could be any fragment of SQL, but the docs seem to imply it must be a complete query.

I created something similar recently: https://docs.racket-lang.org/plisqin/index.html. At its core, it is also a library for composing fragments of SQL, but it has some novel (as far as I know) ideas. The most notable is that joins are values (or "expressions", if you prefer) that can be returned from a procedure like any other value. I was hoping that the world would realize "that's obviously how query builders should work" and copy the approach when starting new projects, but that hasn't happened.

Re: Show HN: Csql – Python lib for composeable SQL queries

#36

PandaSQL allows you to do SQL on Pandas dataframes.[0] [0] https://github.com/yhat/pandasql

This library is badly abandoned. I've been looking for a better alternative, but ended up writing my own relatively simple and somewhat inefficient code.

There's some serious potential in combining Pandas and DuckDB[1], which has an ability to efficiently transfer query results into DataFrames.

[1] https://duckdb.org/docs/api/python

Re: Show HN: Csql – Python lib for composeable SQL queries

#37
post #9

For those of us who didn't know what a CTE was standing for. Here is a link to the PostgreSQL documentation on Common Table Expressions (CTEs)[1], and an article on Wikipedia on Hierarchical and recursive queries, that explains the concept [2]. [1] https://www.postgresql.org/docs/12/queries-with.html [2] https://en.wikipedia.org/wiki/Hierarchical_and_recursive_que... edit: formatting

Also for those who didn't know, and are just getting started with them:

They're incredibly handy for a variety of reasons, but can also have unexpected performance impacts. For example, here are functionally equivalent queries written with a CTE vs an inlined/derived table:

  WITH cte_foo AS 
  (SELECT * FROM bar LIMIT 1000000)
  SELECT * FROM cte_foo LIMIT 1
and

  SELECT * FROM 
  (SELECT * FROM bar LIMIT 1000000) 
  as inlined_foo LIMIT 1
Depending on the database you're using, those two could have wildly different performance due to a concept called an optimization fence[1]. In Postgres versions 11 and below, the CTE would have truly returned/materialized 1 million rows, then the outer query would execute and ultimately return 1 row for the resultset. Whereas the second version would have been optimized such that the outer LIMIT 1 would have been pushed into the subquery and not materialized those extraneous 999,999 rows to begin with.

As mentioned in [1], Postgres 12 (and 13) have started to tackle that optimization fence within Postgres. But it's still a concern/concept to be aware of, since many databases that support CTEs have varying levels of optimization fences, and you'll want to be sure you understand what optimization/performance impacts exist for your particular database before you go down the CTE path.

[1] https://auto1.tech/postgres12-a-precious-release/

Re: Show HN: Csql – Python lib for composeable SQL queries

#39
post #7

Earlier quoted context omitted.

Have you tried dbt? Even if you don't want to adapt their "all in SQL" idea, I definitely took inspiration from their Jinja templating idea and have similar libraries as you, except it uses Jinja templating to parameterize the SQL. This allows more logic to be put in the queries than just variable names!

Huh, no I haven't come across dbt. Looks interesting! Jinja'd sql scares me a bit, but to be fair the string interpolation in my lib here scares me as well :)

Practical experience is that while there are kinks, Jinja and SQL marry fairly well together. Combining this idea and snowflake we have been able to create some extremely versatile metrics reporting systems that run with zero intervention I wouldn't have imagined a year or two back!
Post reply on HN