Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

171–180 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#171

Post by WrtCdEvrydy »

zarq wrote: post #154
WrtCdEvrydy wrote: post #122

Earlier quoted context omitted.

Alternatively... "We need to store some data for a virus" "Where does the data come from?" "Every hospital sends us an excel sheet" "Well, let's merge all of it into a bigger sheet and display it or export it"

"We should maybe start tracking some of this" "Where does the data come from?" "Dunno, a handful of hospitals... maybe?" "Right. Let's just use Excel, not gonna waste time on that now" ... six months later ... "Hey um..."

From the species that brought you "The Game of Life" comes "The Game of Technical Debt"

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#172

Post by shadowgovt »

gridlockd wrote: post #101
sails wrote: post #60

Earlier quoted context omitted.

A question, was it possibly a csv? ie a csv is equivalent to Excel at the Daily Mail, so they printed Excel as the issue due to it being more familiar to their readership. (Can't think of what the technical issue would be in the case of a csv, also doubt the credibility of the source!)

The story I heard was that it was indeed a CSV and that Excel (silently) drops anything beyond 1 million rows.

More specifically: they were detouring through the old XLS file format, which has a 65k row limit.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#173

Post by bleepblorp »

thu2111 wrote: post #88
chii wrote: post #9

Earlier quoted context omitted.

I'm surprised that MS Access didn't take this kind of niche. I know it's a pretty shitty program to use - excel is much more user friendly. If MS had made excel, but have all the features of access in it...

For a very long time, it did. Weaknesses of Access in no particular order: very poor multi-user/multi-device-access features. No real concept of servers. Buggy, had a nasty habit of corrupting its custom file format causing data loss (Excel never does this). Does expect you to understand SQL, data normalisation, etc. VBA just about serviceable for a spreadsheet but not good enough for a 'real' database-backed app. Pr…

LibreOffice Base is something of an attempt at a newer Access, but unfortunately it's far from ready for production use.

Base has very solid data storage foundations (including the ability to connect to real database servers) but the UI is very buggy and is painful to fight with. Note I said 'fight with' rather than 'use.'

Base seems to have a fair bit of potential, however, if someone where to pour some money and/or time into developing it further.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#174

Post by codeulike »

gridlockd wrote: post #144
sct202 wrote: post #128

Earlier quoted context omitted.

Did they pick XLS because it's easier to parse or something? I can't imagine why someone would pick a format that was largely replaced 10 years ago.

Compatibility. It's reasonable to predict that some of the participating entities are running software that is at least a decade old.

Indeed

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#175

Post by TeMPOraL »

bitlevel wrote: post #91

Excel is NOT a database. I don't understand the people who use it as such (and I've come across many who do). Then they're surprised when it all goes tits-up.

If you're thinking regular office workers, they know of no alternative.

Excel works enough for small data, particularly when you don't do complex queries on it. The more you know of it, the better it works. The only other thing that offers similar benefits to Excel but works as a database is MS Access, but the mental model behind it is too complex for your average office worker who wasn't trained in it, and like most database systems, requires a lot of up-front work with figuring out the schema, and doesn't particularly like the schema being modified later on.

As far as I can tell, there's literally nothing else out there. No, random SaaS webapps du jour don't count, because they're universally slow, and also store the data in the cloud, instead of the local drive.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#176

Post by SV_BubbleTime »

>As a consequence, each template could handle only about 65,000 rows of data

Oh? Like 65,535 or so? That seemed weird a first that even as old as xls is that only 16bits were allocated to max rows.

But then maybe not. Each row might get an ID, so that's 16bits * rows you have. I wonder if when XLS was designed they considered it very unlikely many people would have 500MB of db of empty rows and then would needed more data added to each one?

Simplified of course, I bet even original format had some explicit row identifier that could be shortened with optimizations and increment tags.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#177

Post by brnt »

jjkaczor wrote: post #156
codeulike wrote: post #142

Earlier quoted context omitted.

Perhaps some old integration code from 10+ years ago that was written to spit out XLS rather than XLSX e.g. some old versions of Crystal Reports can only do XLS. There must be loads old systems around that can only do XLS

XLSX files have been around for ... nearly 13-years... Ugh - I still deal with clients who are creating brand new XLS and DOC files and expecting to have all the latest features (co-authoring when hosted in 365/SharePoint Online/OneDrive/Teams)...

Worked on a world class radiotherapy piece of software. If you like 20 year old Delphi projects and pre C89 ... C, then come over! Xls export is probably a recent addition, xlsx support will most likely come no sooner than when Excel stops supporting xls.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#178

Post by colinmhayes »

indy wrote: post #121
callamdelaney wrote: post #105

Earlier quoted context omitted.

It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical and important data, I know! Excel!'

I think you'd be shocked by how many of our financial and social institutions rely on emailing Excel spreadsheets to one another.

I am absolutely not shocked by that. Most people do not know how to insert into a db yet alone spin one up and add tables. Everyone can excel.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#179

Post by acallaghan »

ivanstame wrote: post #147

My mind is blown how can you do any real change in the world without good data? We need exact classification of the data of dead people. If they had COVID, was it the main source of death, real cause of death, was it diagnosed(tested) or not...ect. Until then, this stuff about COVID is total bullshit and you are being deceived and you are being to scared. SNAP OUT OF IT!

You're talking about US death certificates, I assume. It's the norm to list more than one thing that lead to death: e.g.

Heart failure - Lung failure - Trauma - Automobile crash

So for covid-19

Respiratory failure - diabetes - COVID-19

You're saying that Respiratory failure or diabetes is the 'real cause of death', but COVID-19 is actually the cause of death. This has been common practice for years, stop believing what Mc. Nixon tweets out.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#180

Post by labster »

jjkaczor wrote: post #156
codeulike wrote: post #142

Earlier quoted context omitted.

Perhaps some old integration code from 10+ years ago that was written to spit out XLS rather than XLSX e.g. some old versions of Crystal Reports can only do XLS. There must be loads old systems around that can only do XLS

XLSX files have been around for ... nearly 13-years... Ugh - I still deal with clients who are creating brand new XLS and DOC files and expecting to have all the latest features (co-authoring when hosted in 365/SharePoint Online/OneDrive/Teams)...

Ten years ago people were still complaining about OOXML not being an open standard. Don't worry guys, I'm sure they'll make a reference implementation of the standard soon! I'm sure there were a few people waiting around for a "real" standard to come out before implementing XLSX support.
Post reply on HN