Live data from Hacker News

Q – Run SQL Directly on CSV or TSV Files

harelba.github.io

61–62 of 62 posts

Re: Q – Run SQL Directly on CSV or TSV Files

#61
post #60
post #54

Earlier quoted context omitted.

Are you telling me that awk can correctly identify delimiters inside quoted strings? Escaped quotes inside quoted strings? Newlines inside quoted strings? I.e. that awk actually has a csv parser? Very cool if so.

Yeah, you can implement a basic FSM and use `next` to handle fake `RS` (e.g. newlines). I'm not necessarily recommending it, but it's certainly possible and could be portable and really fast to run with a low memory footprint.

Well, awk having a csv parser via the user implementing that parser is not quite what I have in mind when I turn to awk for some quick field splitting—and I don't think it's what others in the thread meant either, as evidenced by the linked site.

Personally I prefer using a readymade and tested library in any language that I might touch, so I can just do my own thing on top. Or, in command line, to use an util that employs such a library. Kind of hope that I'm never so constrained that only awk is available and I can't even spin up Lua.

Re: Q – Run SQL Directly on CSV or TSV Files

#62
post #57

Earlier quoted context omitted.

Hi, author of q here. Regarding the error you got, q currently does not autodetect headers, so you'd need to add -H as a flag in order to use the "country" column name. You're absolutely correct on failing-fast here - It's a bug which i'll fix. In general regarding speed - q supports automatic caching of the CSV files (through the "-C readwrite" flag). Once it's activated, it will write the data into another file (wi…

Ah, got it, thank you! Just one minor suggestions/feedback point, in case you find helpful, which is that I had to also add the `-d` flag with a comma value. Otherwise with just -H, I get the error "Bad header row" even though my header was simply "Country,City,AccentCity,Region,Population,Latitude,Longitude". This suggests to me that `q` is not assuming the input to be a CSV file, but that seems at odds with the fir…

Hi again, thanks a lot for the suggestions!

You're absolutely right about the auto-detection (and documentation) of both the header row and the delimiter, I was busy with the auto-caching ability in the last few months in order to provide generic sqlite3 querying, so never got around to it.

I will update the docs and also add the auto-detection capability soon.

Harel

Post reply on HN