I'm rather fond of using [1] csvkit for doing my csv wrangling. That and google refine (now open refine).
CSV as a Data Source
31–40 of 53 posts
Re: CSV as a Data Source
#32Data isn't "locked up" in CSV, rather it is available in CSV. CSV is one of the most open things possible. As a programmer I do a lot of one-time makeshift data reports for other people, and I always use CSV (or precisely, tab separated) because that's what every program happily emits and consumes. If it does not, it's trivial to transform thanks to UNIX sort, awk and uniq.
If I were to build a MVP for any type of data charting or visualization or analysis the first data format that I would think of supporting would be CSV.
You'd think so but when the use case involves dynamically retrieving fresh data it becomes one of the last things to work on. Lots of people have CSV data and it makes sense to support it but since it's generally a manual process it doesn't deliver a good "flow" for the customer. If they have to repeatedly export/import their data manually to have the chart data update it'll seem like a chore.
Compare this to connecting to a customer's database directly. You can refresh your (cached) chart data on your own schedule and the customer doesn't need to be involved at all.
Oh and if anyone is thinking, "Yeah but I can schedule a cron job to extract the CSV file and publish it every X minutes", yes you could do that but again that's work for the customer. If the pitch is "sign up, plug in to your database, and boom charts!" there really should be a "schedule automated cron job" component.
All that aside, it is useful to be able to manually add data like this. Particularly for static data where the manual work is infrequent.
Re: CSV as a Data Source
#33Data isn't "locked up" in CSV, rather it is available in CSV. CSV is one of the most open things possible. As a programmer I do a lot of one-time makeshift data reports for other people, and I always use CSV (or precisely, tab separated) because that's what every program happily emits and consumes. If it does not, it's trivial to transform thanks to UNIX sort, awk and uniq.
Once your CSV (or TSV) files start having quoted fields, they become very tricky to parse using standard multi-purpose tools like sort, awk, & uniq. It's hard enough when you have delimiters in quoted fields, but dealing with quoted newlines starts to become unreasonable, especially for line-based tools. CSV files, as you say, are absolutely wonderful to create. Problems come up when you try to parse files other peop…
With tab separated values this is not a big problem in practice. On the other hand, you can sort tsv, but you can't sort quoted csv.
Re: CSV as a Data Source
#34CSV is the biggest pile of nuisance you'd never expect from a seemingly simple data format.
- Header line or none?
- "\n" or "\r\n"?
- Is there a newline at the last line? How about two?
- Escape quotes with doubling or backslash? How about both in the same file? How about both, inconsistently, in different fields? How about quotes including a newline and commas?
- Strings always quoted? Only if necessary? Is ,, a null or an empty string, or an error?
- How about mixed line lengths? Are missing trailing entries nulls? How about multiple data types in a file, with the first field being type, and line length only fixed per type?
I have generally found "TSV with a rule that data cannot represent tabs or newlines, period" as vastly superior.
Re: CSV as a Data Source
#35CSV format does not define encoding. Also customers usually think about CSV as export/import format from/to Microsoft Excel. Unfortunately each Office localization uses different output encoding and delimiters (and date format). And you cannot suppose that the encoding is UTF-8. The most complete importer of CSV file which I have seen is in Open Office. It is also worth mentioning that Excel can import CSV with other than system locale and different delimiters - but it just not work through open file command.
Re: CSV as a Data Source
#36CSV is the biggest pile of nuisance you'd never expect from a seemingly simple data format. - Header line or none? - "\n" or "\r\n"? - Is there a newline at the last line? How about two? - Escape quotes with doubling or backslash? How about both in the same file? How about both, inconsistently, in different fields? How about quotes including a newline and commas? - Strings always quoted? Only if necessary? Is ,, a nu…
TSV has all those same issues. However, you just have less frequent need of a tab, so most of the escaping edge cases never come up.
Those aspects are defined in RFC 4180 - just a lot of systems don't bother. How would you define a simpler data format?
Re: CSV as a Data Source
#37Data isn't "locked up" in CSV, rather it is available in CSV. CSV is one of the most open things possible. As a programmer I do a lot of one-time makeshift data reports for other people, and I always use CSV (or precisely, tab separated) because that's what every program happily emits and consumes. If it does not, it's trivial to transform thanks to UNIX sort, awk and uniq.
Once your CSV (or TSV) files start having quoted fields, they become very tricky to parse using standard multi-purpose tools like sort, awk, & uniq. It's hard enough when you have delimiters in quoted fields, but dealing with quoted newlines starts to become unreasonable, especially for line-based tools. CSV files, as you say, are absolutely wonderful to create. Problems come up when you try to parse files other peop…
Problems come up when you try to parse files other people write. Not everyone follows RFC 4180.
Plus you've got encodings. If you're accepting CSVs from users, they'll generally come from Excel, which will produce different encoding in different circumstances.
Re: CSV as a Data Source
#38Earlier quoted context omitted.
From what I understand, you're basically right, but you're basically right only in a sense that that's what most people do. There's no "CSV Data Format" spec. It's all just what most people agree on, most of the time. Unless someone has another idea.
text/csv is defined in RFC 4180: https://tools.ietf.org/html/rfc4180
Strictly, the definition in RFC 4180 mandates ASCII which makes it unusable for many purposes. I guess there's nothing practical stopping you from using another encoding though.
Re: CSV as a Data Source
#39CSV is the biggest pile of nuisance you'd never expect from a seemingly simple data format. - Header line or none? - "\n" or "\r\n"? - Is there a newline at the last line? How about two? - Escape quotes with doubling or backslash? How about both in the same file? How about both, inconsistently, in different fields? How about quotes including a newline and commas? - Strings always quoted? Only if necessary? Is ,, a nu…
TSV has all those same issues. However, you just have less frequent need of a tab, so most of the escaping edge cases never come up. Those aspects are defined in RFC 4180 - just a lot of systems don't bother. How would you define a simpler data format?
It's a lot easier to administratively ban tabs/newlines in your data than it is to ban commas, and TSV doesn't have escape mechanisms or quoting. So you actually can parse it with line.chomp().split("\t"), and that doesn't break horrendously.
TSV is streamable and minimally wasteful, I rather approve of it. Netstrings are better though if having sized data and nested data is needed. They are proof against all the ills of quoting and escapes.
Re: CSV as a Data Source
#40CSV is the biggest pile of nuisance you'd never expect from a seemingly simple data format. - Header line or none? - "\n" or "\r\n"? - Is there a newline at the last line? How about two? - Escape quotes with doubling or backslash? How about both in the same file? How about both, inconsistently, in different fields? How about quotes including a newline and commas? - Strings always quoted? Only if necessary? Is ,, a nu…
TSV has all those same issues. However, you just have less frequent need of a tab, so most of the escaping edge cases never come up. Those aspects are defined in RFC 4180 - just a lot of systems don't bother. How would you define a simpler data format?
Also, I should have known there was a RFC for CSV. Too bad that the main call for CSV data is from cranky old versions of Excel in the hands of non technical customers.