Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time to retire the CSV?

bitsondisk.com

Post reply on HN
Print view View original

481–490 of 594 posts

Re: Time to retire the CSV?

#481

Post by mr_toad »

pdonis wrote: post #62
grenoire wrote: post #8

Earlier quoted context omitted.

I think this issue can be bypassed if a better format such as Arrow or Parquet can be used in Excel.

CSV isn't supported in Excel because Microsoft prefers it. MS prefers its own Excel format. Excel supports CSV because a large enough majority of Excel users want it, which means MS includes support for it. For MS to remove CSV support in favor of Arrow or Parquet support would not benefit MS at all, and would make many of its customers push back, so I don't see it happening.

Even if they did support Parquet they’d use the same embrace and extend mechanisms they do with CSV files.

Re: Time to retire the CSV?

#482

Post by soheil »

Literally there are two characters required to create a csv. Anything that replaces it will necessarily be more complex. So let’s keep csv and create something better at the same time.

Re: Time to retire the CSV?

#483

Post by ArsenArsen »

selfhoster11 wrote: post #473
ArsenArsen wrote: post #472

AFAICT, XML fulfils all of the requirements of this. It is self describing, rigorously defined for the machine, highly flexible, highly extensible, human readable, text-based and an open format, and it is only mentioned as a storage format in the post, not even nearly doing it justice! XML is complex, but there's already very established libraries for handling it, so that shouldn't be an issue, and that's the only dr…

Most XML is not human-readable. There's just too much line noise, and most of what's emitted has a weird schema that's difficult to parse using Human Brain 1.0.

I don't think it's much worse than CSV, with it's inflexible structure, not allowing for any formatting to be inserted. As for weird schemas, that really is an implementation specific issue. This example was sketched up in minutes and already reads better than most spreadsheets I've seen over the years!

  
      
          100
          237.87

Re: Time to retire the CSV?

#485

Post by icelancer »

postalrat wrote: post #406
munk-a wrote: post #367

Earlier quoted context omitted.

There is a definite demand for some import format that you can trivially edit - CSV excels at this right now and JSON isn't that bad. Binary dump formats definitely do have a time and a place but there is also a separate need for trivially human readable formats.

I guess my point is that CSVs or any text files aren't really "human readable". You still need some application top view them.

I can - and do - read a CSV by opening it in Notepad++ to view basic data. This isn't possible with a SQLite file.

Re: Time to retire the CSV?

#486

Post by ArsenArsen »

mr_toad wrote: post #477
ArsenArsen wrote: post #472

AFAICT, XML fulfils all of the requirements of this. It is self describing, rigorously defined for the machine, highly flexible, highly extensible, human readable, text-based and an open format, and it is only mentioned as a storage format in the post, not even nearly doing it justice! XML is complex, but there's already very established libraries for handling it, so that shouldn't be an issue, and that's the only dr…

Too easy to produce bloated files. People do dumb things like put field attributes and metadata on every row of the output, so you end up with files that can be several times the size of the raw data. Parsing times are often horrible. There’s no standard for tabular data. You invariably need some overly complicated XML map, because people can’t resist the temptation to over-engineer.

I had anticipated the overhead of XML being brought up, hence the other suggestions there, which is anything + jsonschema. jsonschema was used for illustrative purposes, it's a lot more powerful than most usecases call for, and it pays forward for that by making the syntax very verbose and longwinded. It'd certainly be an alternative with less overhead, though.

I've not benchmarked XML parsing times in a very long time, I'd be interested in seeing the numbers now.

Tabular data is barely data, honestly, hence my PS, I don't think spreadsheets in general are a very good way to store anything.

And yes, absolutely, overengineering is bound to happen, which is unfortunate, but I'm not sure if it really can be avoided while still keeping many of those upsides (especially the rigorous definitions)

Re: Time to retire the CSV?

#488

Post by shadowwolf007 »

breck wrote: post #355
shadowwolf007 wrote: post #241

Earlier quoted context omitted.

Yeah - I used to lead a department that would process somewhere around 10TB of CSV formatted data per day. The edge cases are a hassle but they don't become less of a hassle from a business perspective by switching to json or really any other format. We tried an experiment of using more json and eventually gave it up because it wasn't saving any time at a holistic level because the "data schema" conversations massive…

> "data schema" conversations massively dominated the entirety of the development and testing time. Agreed. JSON let's me know something is a number. That's great, but I still have to check for min/max,zero etc. A string? That's great, but I got to check it against a set of enums, and so forth. Basically, the "types" JSON gives you is about 20% of the work, and you're going to have to parse things into your own types…

Honestly with the schema thing I'd probably be fine with either/or!

A checksum would be crude and user-hostile, only being able to say "you did it wrong" but not really good at tell you what it means to do it right.

If I understand the concepts correctly then it seems like a shared ontology could potentially solve the problem in a non-hostile way.

Plus, it makes me happy because I feel like types are a real-world problem, so it is always nice if the type system could enforce that real-world-ness and all the messiness that comes along for the ride.

Re: Time to retire the CSV?

#489

Post by mr_toad »

CobaltFire wrote: post #165
qwerty456127 wrote: post #112

> It's Time to Retire the CSV > This column obviously contains dates, but which dates? Most of the world It's time to retire local formats and always write YYYY-MM-DD (which is both the international and the Swedish standard, and the most convenient for parsing and sorting). > A third major piece of metadata missing from CSVs is information about the file’s character encoding. It's bloody the time to retire all the c…

Honestly I constantly see dates argued about and people state various formats that are still confusing. 4-2-2 of any variety can be confused. Why not 2-3-4 or 4-3-2 (DD-MMM-YYYY or YYYY-MMM-DD)? I’ve never understood why that isn’t more widely used.

If someone gives you a date like 10–04-2021 you need to know whether they’re American or not. Because it could be October 4th, or April 10th.

(In theory people could write yyyy-dd-mm, but I’ve never seen anyone actually do that).

Re: Time to retire the CSV?

#490

Post by shadowwolf007 »

radus wrote: post #382
shadowwolf007 wrote: post #241

Earlier quoted context omitted.

Yeah - I used to lead a department that would process somewhere around 10TB of CSV formatted data per day. The edge cases are a hassle but they don't become less of a hassle from a business perspective by switching to json or really any other format. We tried an experiment of using more json and eventually gave it up because it wasn't saving any time at a holistic level because the "data schema" conversations massive…

Would DuckDB ( https://duckdb.org/ ) work as your file format with enforced column types?

We looked at it and there were a few problems we had with where it would force us to put VMs that we just weren't super comfortable with due to the in-process-ness.

More a byproduct of decisions made 5 - 7 years ago when the company was in raw startup mode versus a more mature roadmap.

Post reply on HN