Live data from Hacker News

Q: Run SQL Directly on CSV Files

harelba.github.io

21–30 of 102 posts

Re: Q: Run SQL Directly on CSV Files

#21
post #17
post #8

If you're on Windows, you've had the ability to do this for Quite Some Time®: https://support.microsoft.com/en-us/help/850320/creating-an-...

Ah, I thought you were going to link to the fascinating skunkworks Log Parser tool https://www.microsoft.com/en-us/download/details.aspx?id=246... Wonder if it still runs on Windows 10?

If you need a UI, don't forget the tool: Log Parser Lizard ^^

http://www.lizard-labs.com/log_parser_lizard.aspx

Re: Q: Run SQL Directly on CSV Files

#23
post #17
post #8

If you're on Windows, you've had the ability to do this for Quite Some Time®: https://support.microsoft.com/en-us/help/850320/creating-an-...

Ah, I thought you were going to link to the fascinating skunkworks Log Parser tool https://www.microsoft.com/en-us/download/details.aspx?id=246... Wonder if it still runs on Windows 10?

Yeah, still works fine.

Re: Q: Run SQL Directly on CSV Files

#24
post #12

Real-world CSV files generally contain some or all of the following horrors: - some strings enclosed in speechmarks, but some not - empty fields - speechmarks within strings - commas within strings - carriage returns within strings How does Q do up against a CSV file with those traits?

All of your “horrors” seem...correct? Its comma delimited, so anything that is between two commas should be parsed without issue; if it’s a string with a comma in it, and unquoted, you simply have a broken csv file. If its quoted, than anything until the next (unescaped) quote is fine, including commas Unless you’re trying to parse csv files with regexes, none of those should be difficult, or even unexpected, to hand…

Yes, it would be valid CSV. I suppose my point is that naive attempts to roll-your-own CSV parsers tend to fail on the points I listed. Hopefully Q does not do that.

Re: Q: Run SQL Directly on CSV Files

#25
Many of the applications at my workplace use flat text files in CSV format for logs and configuration. For new development I've been using Sqlite to replace some of these usages. Sometimes when I need to analyze legacy log files I import the CSV data into Sqlite tables. After doing this a few times I hit upon the idea of why not skip the import (which balloons my otherwise-small db files) and write an Sqlite plugin that allows treating the flat CSV file as a virtual table? It's neat to see there's some existing work here!

Re: Q: Run SQL Directly on CSV Files

#26

Real-world CSV files generally contain some or all of the following horrors: - some strings enclosed in speechmarks, but some not - empty fields - speechmarks within strings - commas within strings - carriage returns within strings How does Q do up against a CSV file with those traits?

PowerShell will clean up a gross CSV:

Import-CSV .\file.csv | Export-CSV .\file.csv -NoTypeInformation -Encoding UTF8

Re: Q: Run SQL Directly on CSV Files

#27
post #4

I've never heard of this before, but it reminded me of this other command line tool for wrangling csv files in a cli. This tool makes use of SQL syntax much more than xsv, so there isn't 100% overlap here. https://github.com/BurntSushi/xsv

Thanks! xsv is new to me. I love Andrew Gallant's ripgrep(rg) grepping cmd tool.

You should definitely consider using xsv under the hood. It’s dramatically faster than any other tool of its kind that I’ve tried, with low-level implementation in Rust.

Re: Q: Run SQL Directly on CSV Files

#29
“Any file is a database if you awk hard enough.” —Bryan Horstmann-Allen

https://twitter.com/neilkod/status/914217352564137984

Furthermore, no thread on CSV files can be complete without mentioning this infamous bit of fact-trolling: the ASCII standard has had unit- and record-delimiters baked into it from the beginning.

https://ronaldduncan.wordpress.com/2009/10/31/text-file-form...

Post reply on HN