Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CSVs Are Kinda Bad. DSVs Are Kinda Good

matthodges.com

Post reply on HN
Print view View original

51–60 of 133 posts

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#51

Post by SammyStacks »

throw0101d wrote: post #35
atrus wrote: post #29

Earlier quoted context omitted.

Don't use broken tools. The key phrase is "in control of producing data". If you're forced to use Excel, then it's not really you in control, is it?

>> […] Excel […] > Don't use broken tools. Tell that to your accounting and finance department and let us know how the message is received. > If you're forced to use Excel, then it's not really you in control, is it? In which case the up-thread's advice to "just produce strict RFC 4180-compliant CSV data" is worthless. "Just." We're stuck with whatever CSVs we get, so 'just' doing X is not an option.

From a pragmatic viewpoint, the CSVs that I get from finance (usually saved as .xlsx) have the same issues for parsing the data as a CSV. But since the issues are consistent, I can automate conversion from .xlsx to CSV, then process the CSV using awk to eliminate errors in further parsing the CSV (for import, analysis, etc.). Sure, I'm essentially parsing the CSV twice but, because the parsing issues are consistent, I can automate to make the process efficient.

Obviously that wouldn't work for CSVs with different structures, but can be effective in the workplace in certain scenarios.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#52

Post by bvrmn »

[Grumpy mode start]

Some nitpicks, maybe someone finds it useful. Could we talk about a code design a little bit?

    class DSV:
        @property
        def delimiter(cls) -> bytes:
            return b'\x1F'

        @property
        def record_separator(cls) -> bytes:
            return b'\x1E'

        @property
        def encoding(cls) -> str:
            return 'utf-8'
It's Python, do not make a premature properties for static values.

    class DSV:
        delimiter = b'\x1F'
        record_separator = b'\x1E'
        encoding = 'utf-8'
Also it's a false inheritance relationship. Writer is not related to configuration. You can't make any other useful subclasses for DSV (ok maybe DSVReader, but that's it). At least it should be in the opposite way: an abstract Writer operating on instance configuration attributes and DSVWriter defining these attributes.

Also `self._buffer += chunk` is O(N^2). It starts to bite even for buffers small as 100 bytes. It's ok for an example, but it's an issue for real code. Example at least buffers incomplete record not a whole read chunk (good!). But does only one split at a time (bad).

[Grumpy mode end]

Nevertheless article is very valuable and interesting to read. CSV gotchas are well described.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#53

Post by Kon-Peki »

throw0101d wrote: post #24
teddyh wrote: post #6

The article talks about reading and parsing CSV data of unknown variants, but then skips to the solution being using a different format altogether. But you can only switch to a different format if you are producing data, not if you are reading it! And if you are in control of producing data, just produce strict RFC 4180-compliant CSV data and everybody will be able to read it just fine. There is no need to make your…

[flagged]

> Excel

If you’re deep in the Excel world, chances are extremely high that you also have access to SSMS, which has a really, really good data import tool that makes short work of nasty CSV files. The output of this tool doesn’t even have to be SQL Server, it will use any ODBC driver you’ve got installed; you can send the data to Excel or even a new, properly formatted CSV.

And if you want a repeatable package, there is always SSIS.

Look, I would never recommend anyone jump into the Microsoft ecosystem. But when in Rome, do as the Romans do.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#54

Post by dghf »

usrbinbash wrote: post #17

> CSVs are kinda bad. Not really. What's bad is when people keep insisting on coming up with new and amazing CSV dialects. https://www.ietf.org/rfc/rfc4180.txt is very clear about what CSV files are supposed to look like, and the fact that people keep ignoring this for whatever reason, is not the formats problem. And no, "using another format" is not a solution to this. Because: I can just invent a new DSV dialect. O…

> https://www.ietf.org/rfc/rfc4180.txt is very clear about what CSV files are supposed to look like

Mm, not really. By its own admission, it is descriptive, not prescriptive:

> This section documents the format that seems to be followed by most implementations

And it came out in 2005, by which date CSVs had already been in use for some twenty or thirty years.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#55

Post by snapcaster »

usrbinbash wrote: post #17

> CSVs are kinda bad. Not really. What's bad is when people keep insisting on coming up with new and amazing CSV dialects. https://www.ietf.org/rfc/rfc4180.txt is very clear about what CSV files are supposed to look like, and the fact that people keep ignoring this for whatever reason, is not the formats problem. And no, "using another format" is not a solution to this. Because: I can just invent a new DSV dialect. O…

No it isn't in the real world. It's very much your problem if you're the team consuming these files. Try to go tell the head of accounting they need to make all their data rfc4180 compliant see how that goes

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#56

Post by bathtub365 »

throw0101d wrote: post #35
atrus wrote: post #29

Earlier quoted context omitted.

Don't use broken tools. The key phrase is "in control of producing data". If you're forced to use Excel, then it's not really you in control, is it?

>> […] Excel […] > Don't use broken tools. Tell that to your accounting and finance department and let us know how the message is received. > If you're forced to use Excel, then it's not really you in control, is it? In which case the up-thread's advice to "just produce strict RFC 4180-compliant CSV data" is worthless. "Just." We're stuck with whatever CSVs we get, so 'just' doing X is not an option.

The up-thread’s comment (emphasis mine):

> And *if you are in control of producing data*, just produce strict RFC 4180-compliant CSV data

The point of the comment was that you likely aren’t in control of producing the data, so the article’s recommendation of using an entirely different format is likely also invalid. I’m not sure what you are arguing against as you seem to actually agree with them.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#57

Post by teddyh »

throw0101d wrote: post #24
teddyh wrote: post #6

The article talks about reading and parsing CSV data of unknown variants, but then skips to the solution being using a different format altogether. But you can only switch to a different format if you are producing data, not if you are reading it! And if you are in control of producing data, just produce strict RFC 4180-compliant CSV data and everybody will be able to read it just fine. There is no need to make your…

[flagged]

If you have to use Excel, you have to produce whatever data Excel produces. It all depends on what (explicit or implicit) contracts you have in place with whoever is to consume the data which you produce.

Re: CSVs Are Kinda Bad. DSVs Are Kinda Good

#58

Post by vaylian »

lazide wrote: post #39
throw0101d wrote: post #28

Earlier quoted context omitted.

> Aka a completely different use case than CSV. How many CSVs are generated, edited, or viewed by Notepad.exe and how many by Excel (or Google Sheets)? I would posit the vast majority of CSVs are generated through some kind of program where you go to File > Export or File > Save As…. In which case doing selecting a drop down with the option for File Format to be TSV or DSV (with the corresponding file extension) woul…

How many get edited or inspected in notepad at some point in their life? Nearly all of them (for any given workflow).

It is nice that text editors are abundantly available and that they can be used for the task. But once the CSV columns get too wide and irregular, then you probably want to reach for a dedicated spreadsheet program, because it is otherwise too hard to figure out which column you are currently reading.

There is still room between a text editor and a full-blown spreadsheet program. New DSV editors could emerge when the DSV format gains popularity.

Post reply on HN