Live data from Hacker News

TextQL: Execute SQL Against CSV or TSV

github.com

71–80 of 95 posts

Re: TextQL: Execute SQL Against CSV or TSV

#71

If you are on Windows you may consider using Log Parser Studio [1]. It is a GUI over logparser.exe which understands a lot of various file formats and uses SQL to query them [2]. [1] https://gallery.technet.microsoft.com/office/Log-Parser-Stud... [2] https://en.m.wikipedia.org/wiki/Logparser

Came here to say this. Log parser is amazing. Check out Log Parser Lizard for a good IDE.

Re: TextQL: Execute SQL Against CSV or TSV

#72
post #69
post #40

Earlier quoted context omitted.

#!/usr/bin/env rc row_headers='' output_mode=column fn usage { echo $0 usage echo -o '#' which output mode to use echo -r '#' if present, output row headers echo -h '#' this message } while(~ $1 -*) { switch($1) { case -o output_mode=$2 shift case -r row_headers='.headers on' case -h usage exit 0 case * echo Bad args: $* usage exit 1 } shift } test $#* -ne 1 && echo Wrong number of arguments && usage && exit 1 sql_st…

Erm, people don't have rc installed though. I had completely forgotten it existed, to be honest.

OK, thanks for the heads up.

Re: TextQL: Execute SQL Against CSV or TSV

#73
post #70

Earlier quoted context omitted.

In this case, It’s what it does that matters, not how it does it. I don’t care if SQLLite is powering it. Maybe the author would want to swap out the SQLLite engine with something else in the future too. Then it’s even better that they didn’t advertise the link to SQLLite as a selling point.

It does actually matter what's running under, so you know what to expect from its behavior and if one doesn't exactly need the few features textQL offers exclusively, you would think to just use SQLite without random layers on top.

Sure, but it doesn’t need to be in the title

Re: TextQL: Execute SQL Against CSV or TSV

#74
post #11

Earlier quoted context omitted.

But it is SQLite under the covers? A better title then would be an SQLite wrapper for use in shell pipelines.

In this case, It’s what it does that matters, not how it does it. I don’t care if SQLLite is powering it. Maybe the author would want to swap out the SQLLite engine with something else in the future too. Then it’s even better that they didn’t advertise the link to SQLLite as a selling point.

SQL dialects have significant differences. You can't simply change the underlying engine without breaking backward compatibility. It's also good to redirect documentation to official sqllite docs instead of writing a 1000 page manual yourself.

Re: TextQL: Execute SQL Against CSV or TSV

#75
There is also BigBash [0] that converts an Sql statement to a bash one-liner (using sed, grep, awk,...) which can then run to execute the query. The advantage is that you can let it run on very large file(s) because of the streaming nature of linus bintools.

[0] http://bigbash.it or the corresponding Github repo.

Re: TextQL: Execute SQL Against CSV or TSV

#76
post #37

ClickHouse ships with a command line tool which does this (without the actual database server): ps aux | tail -n +2 | awk '{ printf("%s\t%s\n", $1, $4) }' | \ clickhouse-local -S "user String, mem Float64" \ -q "SELECT user, round(sum(mem), 2) as memTotal FROM table GROUP BY user ORDER BY memTotal DESC FORMAT Pretty" ┏━━━━━━━━━━┳━━━━━━━━━━┓ ┃ user ┃ memTotal ┃ ┡━━━━━━━━━━╇━━━━━━━━━━┩ │ clickho+ │ 0.7 │ ├──────────┼──…

the additional requirement to set up the schema is kind of onerous.

Shameless plug: Sqawk can do nearly the same without you defining a schema.

  $ ps aux | sqawk -output table \
                   'select user, round(sum("%mem"), 2) as memtotal
                    from a
                    group by user
                    order by memtotal desc' \
                   header=1
  ┌────────┬────┐
  │dbohdan │67.1│
  ├────────┼────┤
  │  root  │3.5 │
  ├────────┼────┤
  │ avahi  │0.0 │
  ├────────┼────┤
  │ daemon │0.0 │
  ├────────┼────┤
  │message+│0.0 │
  ├────────┼────┤
  │ nobody │0.0 │
  ├────────┼────┤
  │  ntp   │0.0 │
  ├────────┼────┤
  │ rtkit  │0.0 │
  ├────────┼────┤
  │ syslog │0.0 │
  ├────────┼────┤
  │ uuidd  │0.0 │
  ├────────┼────┤
  │whoopsie│0.0 │
  └────────┴────┘
Link: https://github.com/dbohdan/sqawk

Re: TextQL: Execute SQL Against CSV or TSV

#77
post #48

Ha. Shameless plug, but just about a week ago I started to work on a somewhat analogous thing. Executing SQL on XML [1]. I came up with this idea after trying to use stackoverflow data dump and finding out it was stored as XML. I wanted to run queries with LIKE operator on it. [1]: https://github.com/kamac/AskXML

This is a great idea. There are so many APIs and text dumps from websites and databases that are XML, and it would be nice to be able to query them with regular relational operations. I'll be following your project! I will say that I believe a streaming backend would be more suitable, since most of the XML dumps I'm thinking of are humongous and won't always fit in memory.

Re: TextQL: Execute SQL Against CSV or TSV

#79
post #37

ClickHouse ships with a command line tool which does this (without the actual database server): ps aux | tail -n +2 | awk '{ printf("%s\t%s\n", $1, $4) }' | \ clickhouse-local -S "user String, mem Float64" \ -q "SELECT user, round(sum(mem), 2) as memTotal FROM table GROUP BY user ORDER BY memTotal DESC FORMAT Pretty" ┏━━━━━━━━━━┳━━━━━━━━━━┓ ┃ user ┃ memTotal ┃ ┡━━━━━━━━━━╇━━━━━━━━━━┩ │ clickho+ │ 0.7 │ ├──────────┼──…

the additional requirement to set up the schema is kind of onerous.

[deleted]

Re: TextQL: Execute SQL Against CSV or TSV

#80
post #4

For anyone looking to do this with plain SQLite, one can import a CSV by running (in the REPL) .mode csv .headers on .import my.csv tablename This does look extremely convenient, though; being able to use UNIX pipes will be a huge improvement to my workflow.

To have an idea of what can be done with sqlite look at this solution to the Hashcode 2018 https://github.com/mingodad/sqlite3-hashcode-2018 using only sqlite3 command line.
Post reply on HN