TOML will solve all these problems. More structured than JSON, easier to consume than XML. https://github.com/mojombo/toml
Is it really more structured than JSON?
first class dates says yes it is!
How to share your data effectively
91–100 of 127 posts
Re: How to share your data effectively
#92Why not just provide multiple formats?
Re: How to share your data effectively
#93RDF, RDF, RDF, RDF, RDF, RDF, RDF. Yes. Please use RDF. And please use a common vocabulary. If you want to know how to do this? Have a look at the UK governments open data portal (which uses RDF): data.gov.uk And their SPARQL (query) endpoint: http://data.gov.uk/sparql
Deep in the clearance bin of web technologies (looking at you xhtml) lies like RDF and SPARQL. The only reason governments use it is for job security and because only they have the endless budgets to bother with it.
Re: How to share your data effectively
#94Earlier quoted context omitted.
I don't have the full picture here but by moving away from an Excel format you are also potentially taking away a lot of control they have themselves over managing the data and working the data which Excel is extremely good at with little to no programming.
Your point is true generally, but I have yet to use a version of Excel that can handle multiple gigabytes of data without serious performance degradation and probably a lot of crashes.
Which would matter, if businesses cared. I spend 4 days every month compiling a 1.5GB, Excel-based report. 80% of that time is spent with Excel frozen up and me playing on my phone, hoping that when Excel unfreezes it doesn't crash.
I suggested porting the process to Access with an Excel-based front end, which would cut the compilation time down to 4 hours at the most. I was denied due to the fact that whomever supports it after me will more than likely not have Access experience, and told to continue with the 4 days of hours-long freeze ups.
Corporate life for you, I guess.
Re: How to share your data effectively
#95My preference is .csv, because it can be read by almost any program. Except the one most used for it: Excel. Excel sucks at csv, especially if the recipients have various internationalized versions. It can't deal properly with multi-line strings and spaces, but especially the internationalization is hell (semi-colons instead of commas, decimal separators, date formats). Many times have I dealt with CSV-exports that c…
From someone currently dealing with Excel's CSV writing for a business application, I heartily concur. There is a decade old bug (at least) where the CSV save as format does not properly handle Unicode, and there's no simple way to handle it down the pipeline. I guess the Office team would rather work on the metro ribbon.
I guess the Office team would rather work on the metro ribbon.
Of course they would! They've already achieved their goal with Excel: make it a lock-in product. In this case, they did it by breaking CSV compatibility, thereby forcing everyone to use XLSX.
Re: How to share your data effectively
#96Earlier quoted context omitted.
You left out the absolute worst bit: Excel tries to "helpfully" guess the correct data format for each cell in a CSV file. This can lead to silent data loss. Long strings of numbers are converted to a Numeric type that only stores a certain number of significant figures. The string "1-5" is converted to a Date field with the value "5-Jan"
You can specify the field types when opening the file.
Any numbers with leading zeroes, i.e. 0000123, excel will interpret as the integer 123 even if you change the column format to "text". It's infuriating.
The only way I found to get around this was to open a new workbook, change the column type to "text" and then paste the data in. I believe this was excel 2010 on windows.
Re: How to share your data effectively
#97Use a flexible file format. My preference is .csv, because it can be read by almost any program. I'll tolerate .xls, but I'm not pleased with .xlsx (not everyone uses Excel!). And please, please, please do not use pdf. Why is .xls tolerated but not .xlsx? XLS and XLSX are both Excel formats, but XLS is a binary blob while XLSX is just a zip containing xml files. I'd much rather parse xml files (XLSX) than the binary…
My experience is that open-source implementations like LibreOffice have much better support for the older file formats. docx/xlsx/etc may be a "zip of XML files" but it doesn't really matter if the original data is stored as XML or a blob when it comes to the end user's rendering experience -- it's about how the system can interpret that data. And the interpretations of the new file formats are mostly severely lackin…
Very much this. Sometimes, I don't want (or can't) open OO.org, I just want to be able to get a quick glimpse in Emacs (http://www.emacswiki.org/emacs/UnXls) or mutt of a simple table of data someone sent me, but so far there are no quick CLI tools (that I've found) to do this. It almost feels as if Microsoft said "hmm, all these free software products can open our files, but they're still complaining about open file formats; how can we abuse our monopoly even more and guarantee that some truly open standard doesn't get mandated?" and thus was born docx, xlsx, pptx, etc.
Re: How to share your data effectively
#98Earlier quoted context omitted.
I agree with you, I actually had a client who demanded that the data in their project be done in excel (with multiple files ranging from 1-3 gb in size!). Both myself and my partner tried exhaustively to use a database or even JSON/XML solutions but the client kept repeating "in the real world business is done in excel!" Needless to say it was a nightmare.
Maybe your client was wanting ubiquitous Excel-compatible export/import - not for the back end to actually be Excel? Making any table in your app exportable to at least CSV is pretty straightforward.
I think that is perfectly logical, the project was basically a series of command line tools. I tried to swap out excel for CSV or at least TSV, the client demanded .xls in .xls out. Believe me we tried every suggestion and that was as far as the client would negotiate. We got it to work and I died a little inside.
Re: How to share your data effectively
#99Earlier quoted context omitted.
If you are generating the CSV yourself, save yourself some agony and just wrap the text in ="" $ cat test.csv ="12.34567890124312341234123412341234",="1-5" $ open -a Microsoft\ Excel test.csv
Sure, it's a nice trick, but now it's not really a CSV file any more. If I have to create a file that can only be read by Excel it might as well be XLSX.
There's a slight difference between having to strip "=" from the column values for other applications to handle it, and require those other applications to be able to handle XLSX..
Re: How to share your data effectively
#100Earlier quoted context omitted.
You left out the absolute worst bit: Excel tries to "helpfully" guess the correct data format for each cell in a CSV file. This can lead to silent data loss. Long strings of numbers are converted to a Numeric type that only stores a certain number of significant figures. The string "1-5" is converted to a Date field with the value "5-Jan"
You can specify the field types when opening the file.
… and precisely how many users do you think know to do this? They might if it's happened before, been correctly diagnosed and explained before and it's obvious as soon as you open the file. It's more common, however, that the Excel bug only causes data loss or invalid results in a percentage of records and, when people notice it, it's corrected by hand.
If you need to do this a lot, generate xlsx - it's not that hard and there are decent libraries for most languages. Relying on users correctly setting column types every time is like making your daily commute depend on having airbags.