Live data from Hacker News

Sq.io: jq for databases and more

sq.io

61–70 of 134 posts

Re: Sq.io: jq for databases and more

#61
post #53

Though I respect and applaud the effort that went into creating this and successfully releasing it, It has fewer features than duckdb supports at the moment. Duckdb supports both Postgres, Mysql, SQLite and many other extensions. Postgres: https://duckdb.org/docs/extensions/postgres MySQL: https://duckdb.org/docs/extensions/mysql SQLite: https://duckdb.org/docs/extensions/sqlite You can try this yourself. 1. Clone th…

not to mention the dozen+ other sources DuckDB supports (Iceberg, Parquet, CSV, Delata, JSON, etc.).

DuckDB extension support / dev experience is quite good now too. I've been working on some improvements (e.g. predicate pushdown) to the Iceberg extension and it's been pretty smooth.

Re: Sq.io: jq for databases and more

#62
post #37

I love JQ. But ... I'd never considered its query language to be particularly admirable. If I want to ask questions of some databases, I don't understand why I'd choose JQ's XPATH-like language to do it.

What I love about `jq` that I can edit my query (or "program") by appending tokens at the end. Similar to unix pipes. With plain SQL that is not easy.

KQL has some good things, pipes is one of them I think.

Re: Sq.io: jq for databases and more

#63
post #58
post #49

Earlier quoted context omitted.

HN is inundated with posts announcing paper thin abstractions on top of existing technology or utilities that just move the goalpost of what you knowledge you need to be effective. It's a weird trend that seems almost entirely motivated by people wanting open source projects in their resume, or seek funding if its a startup.

> It's a weird trend that seems almost entirely motivated by people wanting open source projects in their resume That’s really harsh and misguided. If people didn’t do “paper thin abstractions” projects on their own time for the simple pleasure of doing it we wouldn’t have 90% of the successful projects we have today. Let people have fun and don’t judge their motives when they’re making something Open Source. I can g…

Unless you're the GP your guarantee about the persons motivation is as meaningless as the post you're replying to.

Re: Sq.io: jq for databases and more

#64
post #56
post #54

Earlier quoted context omitted.

I think it'd be a moot point if SQL wasn't painful and awkward to work with in the first place. But database purists control it and won't let go, so we will have to live with everyone else inventing layers to make their lives easier.

For me this feels like the complaints about error handling in Go. People who work with it all the time, don't even think about it past the first week. If you are starting out it might bother people because they are not used to it. Personally I really like working with SQL and find it quite elegant, I always encourage people to use it as it really is a job-superpower if you can just dig up issues directly in the DB qu…

I agree with the GP’s point that people should probably just learn SQL, but there are things about SQL that are objectively bad, especially from the perspective of a software developer.

It’s not a very composable or consistent language, so I think it makes total sense that we see so many abstractions over it.

Re: Sq.io: jq for databases and more

#65
> sq is pronounced like seek. Its query language, SLQ, is pronounced like sleek

As a person who is apart from the tech scene, and lurks in the tech space out of interest, I appreciate this guidance. For the longest time I didn’t know nginx was pronounced Engine-X; I called it N-jinx.

Re: Sq.io: jq for databases and more

#66
post #56

Earlier quoted context omitted.

For me this feels like the complaints about error handling in Go. People who work with it all the time, don't even think about it past the first week. If you are starting out it might bother people because they are not used to it. Personally I really like working with SQL and find it quite elegant, I always encourage people to use it as it really is a job-superpower if you can just dig up issues directly in the DB qu…

I agree with the GP’s point that people should probably just learn SQL, but there are things about SQL that are objectively bad, especially from the perspective of a software developer. It’s not a very composable or consistent language, so I think it makes total sense that we see so many abstractions over it.

The issue with the abstractions over SQL is that while they fix some problems they always introduce a bunch of new problems so in the end SQL is still preferable. I have yet to see an example where that is not the case.

Re: Sq.io: jq for databases and more

#67
post #45

This is interesting. I wonder if there is anything that does the opposite - takes JSON input and allows you to query it with SQL syntax (which would be more appealing to an old-timer like me)

This tool can do that: $ cat example.json | sq sql 'SELECT * FROM data' actor_id first_name last_name last_update 1 PENELOPE GUINESS 2020-06-11T02:50:54Z 2 NICK WAHLBERG 2020-06-11T02:50:54Z One of the data source types is 'json' and the command 'sq sql ...' lets you execute SQL directly instead of using their default query language (SLQ).

wow, I missed that in the docs but this is exactly what I was thinking!

Re: Sq.io: jq for databases and more

#68
post #48

Sometimes I wonder if it wouldn't be more efficient for people to just learn SQL instead of trying to build tools or layers on top of it that introduce more complexities and are harder to search for.

SQL interop is where the pain is at. Using standard tooling of most database systems, best you can get is CSV with all the pains this shithole of a data transfer format brings.

What's wrong with CSV? I love me a CSV file.

Re: Sq.io: jq for databases and more

#69
post #56
post #54

Earlier quoted context omitted.

I think it'd be a moot point if SQL wasn't painful and awkward to work with in the first place. But database purists control it and won't let go, so we will have to live with everyone else inventing layers to make their lives easier.

For me this feels like the complaints about error handling in Go. People who work with it all the time, don't even think about it past the first week. If you are starting out it might bother people because they are not used to it. Personally I really like working with SQL and find it quite elegant, I always encourage people to use it as it really is a job-superpower if you can just dig up issues directly in the DB qu…

> People who work with it all the time, don't even think about it past the first week.

I think SQL is often quite awkward, at least when you look at people exploring alternatives, like https://prql-lang.org/

Not just “the standard” variety, but also all of the vendor specific varieties that you’ll use in practice (MySQL/MariaDB, PostgreSQL, SQLite, Oracle, SQL Server etc.) and how the features offered by each differ, how the support for custom types differs, how the procedural languages differ, the actual experience of using them, seemingly half baked error messages when compared to most imperative languages, varying support for getting the query optimiser to do what you want (e.g. hints), query plans that aren’t pleasant to stare at, often no support for automated analysis of how things are running and suggestions for indices (e.g. Oracle had that sort of feature, it is sometimes helpful, but the automatically generated indices are or at least last I checked were treated as their own special thing and you couldn’t easily delete them). Even things like varying support for working with geospatial data, JSON or time series data.

Not just that, but also the tooling (or the lack of it) - good luck debugging the execution of stored procedures in your DB or placing breakpoints in there, good luck hooking up observability/tracing solutions as easily as you would for your back end, good luck debugging why your database link calls take 100x longer when executed through JDBC in your app but not in the developer tooling.

Not that ORMs or NoSQL make everything much better, you’d just trade one set of annoyances for another, especially when you end up trying to generate dynamic SQL with the likes of myBatis XML mappers. Don’t get me started on people over fetching or ending up with N+1 problems with most ORMs, or executing bunches of queries against the DB just to retrieve some data that you might as well get in one go with a well crafted DB view. Not that you can’t make your DB schema a mess as well with a liberal application of EAV or OTLT (or just attempting to make polymorphic links).

I think SQL as a language makes sense, but PRQL often feels more sensible. I feel like the current RDBMSes out there are very powerful, but that there are also aspects about them (and the differences between the ones you’ll use) that absolutely suck. It feels like where back end languages are getting progressively better DX, databases instead gave us… CTEs? Some syntactic sugar for JSON in PostgreSQL? Feels like they move painfully slow.

For what it's worth, that's why I welcome every new tool or attempt at improving things, even if they won't replace the usual enterprise stacks of having some RDBMS, a JDBC/ODB/whatever driver and probably an ORM on the back end.

Re: Sq.io: jq for databases and more

#70

Earlier quoted context omitted.

SQL interop is where the pain is at. Using standard tooling of most database systems, best you can get is CSV with all the pains this shithole of a data transfer format brings.

What's wrong with CSV? I love me a CSV file.

Out of the random shit I had to deal with CSV file wrangling in the last two years:

- no defined encoding, so it may be anything from US-ASCII over ISO-8859-x to UTF-8 and it's always a guesswork what it actually is

- no definition of the separator sign, usually it can be a literal comma, a semicolon, or tabs

- escaping of the separator sign in column values is always fun

- escaping of newlines is even MORE fun

- line endings are not specified

Every piece of software there is has their own expectations on what specific kind of format it expects, and to make it worse, using "Microsoft Excel" as a target/source isn't foolproof either because it behaves differently between the Mac and the Windows version!

JSON in contrast is clearly defined in all of these issues and has battle-tested implementations in every programming language there is.

Post reply on HN