For most companies, "big data" is when they can't fit the data in a single Excel sheet anymore.
Devops Borat: "big data is any data which is crash excel"
Missing Covid-19 test data was caused by the ill-thought-out use of Excel
131–140 of 834 posts
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#132Earlier quoted context omitted.
It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical and important data, I know! Excel!'
Have you every worked in a large enterprise? I have personally tested an Excel based credit rating tool, to be rolled-out globally by a major financial institution. The fact that one should not do this, is by no means a reason not to do it.
Would be interested in hearing more about that Excel tool.
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#133Good lord. Old XLS files. I'd love to have seen how this was concocted.
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#134I've been bitten by this exact issue (albeit on non critical data), save a file as XLS it will silently drop rows beyond 65k - save it as XLSX and it's all good..
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#135Earlier quoted context omitted.
The way I heard it, it was an Excel problem: They had a CSV file that had more than 1 million lines, and Excel dropped the lines beyond 1 million. I can imagine an automated process that appended to the CSV and then Excel saving back to that same file with the lines stripped. I hope we can all agree that this is not the way Excel or any other software should behave, whether the user is incompetent or not.
If you read the article (perhaps its been updated?), they were using the old XLS format and they hit the limit at 64,000 lines PHE had set up an automatic process to pull this data together into Excel templates so that it could then be uploaded to a central system and made available to the NHS Test and Trace team as well as other government computer dashboards. The problem is that the PHE developers picked an old fil…
Like I said, that's the way I heard it, the way it was narrated to me.
The Guardian reported my version:
https://www.theguardian.com/politics/2020/oct/05/how-excel-m...
CSV would to me make more sense than an outdated binary format like XLS, but either story is plausible.
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#136Excel is NOT a database. I don't understand the people who use it as such (and I've come across many who do). Then they're surprised when it all goes tits-up.
Excel may not be a database solution, but it is the most commonly used database solution.
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#137This is indeed user incompetence, but so far every news source quoted the official PR and said it was an "Excel Problem". Microsoft PR was caught unprepared - I wonder how they'll re-spin it in the next few days (and for the first time that I can recall, a Microsoft product was wrongly blamed...) Pay attention, how every time there's a Windows virus or worm, it's a "Computer Virus", but in the (extremely rare) occasi…
It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical and important data, I know! Excel!'
The common case for stories like this is the answer is "nobody."
Time point 1: "We need a way to map the long-form epidemiology state. It's 2D data... Let's use Excel." "Will we hit limits?" "Pfft, no. Not unless we end up needing to track tens of thousands of patients, and that's not practical; we don't have enough people to do that tracking in the whole NHS."
Time point 2: "This pandemic is an URGENT problem. We need to hire more people than ever before to do tracing. And be sure to keep the epidemeology state mapper updated so it can drive the summary dashboards!"
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#138Earlier quoted context omitted.
It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical and important data, I know! Excel!'
I think you'd be shocked by how many of our financial and social institutions rely on emailing Excel spreadsheets to one another.
Don't forget healthcare! Excel and CSV for tons of data exchange.
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#139Friends don't let friends use Excel.
Unfortunately Excel is the great swiss army knife of software and it's hard to avoid.
Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel
#140We did a bit of work for laboratories this year and csv is not an uncommon exchange format between labs. In general almost all exchange formats are text based, with labs saying they will upgrade to „modern xml formats“ at some point in the future. So seen in this context a csv or an excel file doesn‘t really surprise me and should probably also be seen in this context.
> exchange formats are text based, with labs saying they will upgrade to „modern xml formats“ Worth noting that XML is also a text format. SGML even can treat CSVs as markup. There's nothing wrong with CSVs/TSVs anyway - it's a concise tabular format using only minimal special coding for a record and a field separator, as envisioned by ASCII and EDIFACT. The problem seems more like that there was no error checking in…
..this right here.
No error checking, inadequate testing, no reconciliation process.
Excel is used extensively in many industries. Any file could be cut off in processing by any number of reasons, one off errors for e.g.
So the solution is to "fix" the process by using the existing broken process and smaller files....