Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Q: Run SQL Directly on CSV Files

harelba.github.io

Post reply on HN
Print view View original

101–102 of 102 posts

Re: Q: Run SQL Directly on CSV Files

#101

Post by fulafel »

marcodave wrote: post #72
stouset wrote: post #71

Earlier quoted context omitted.

They don’t really get used IMO because if you’re storing ASCII or Unicode text, you have to be prepared for those characters to be in your data. And if you’re storing binary data, delimiters don’t really cut it in the first place.

also they cannot be found in keyboards, if you need to write CSV by hand

You can type it at a terminal the same way you can write other low-ascii codes. For example record separator, 30, is ctrl-^ - you can type it in the terminal by the usual literal control char way, ctrl-v[1].

So for example at a Python prompt:

  >>> ord('^^') # typed ctrl-v ctrl-^ here
  30
(the low unprintable ascii codes correspond to certain ctrl combinations because ctrl-x is just the ascii code of x with the high bits masked off (i.e. & 0x1f) )

[1] custimizable in your terminal settings, see stty lnext ("literal next")

Re: Q: Run SQL Directly on CSV Files

#102

Post by barrkel »

harelba wrote: post #100
barrkel wrote: post #32

Earlier quoted context omitted.

More interesting: any kind of delimiter, including chars from utf8 and windows-1252, and you need to detect encoding too. And CSV embedded in CSV, a result of flattening an XML source. And fixed width files, not CSV but where you see CSV you may need to support. And let's not get into date parsing or other typed data, and type inference over sample files.

Hi, q's creator here, Any kind of input/output delimiter is supported (-d and -D ), and also multiple encodings (-e ). Also, q performs automatic type inference over the actual data. Encoding autodetection and fixed width files are not supported though.

The company I work for also does delimiter autodetection, quote character inference (from a limited set), and encoding inference (which is mostly limited to utf8 / windows-1252 / iso-8859-15, but it can't reliably differentiate latter two).
Post reply on HN