Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

431–440 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#431

Post by konjin »

TeMPOraL wrote: post #423
konjin wrote: post #418

Earlier quoted context omitted.

Peering into my crystal ball: You were between 16 and 22, you were 'great with computers', you had never used an OS other than Windows. There is a difference between a calculator and a type writer and there is a difference between an office os and one for programming. Using Windows and blaming Octave for Windows sucking is a rite of passage for everyone in a BSc program. I hope you got better and switched to a Unix.

Holy shit, how much does it cost to rent out your crystal ball for a few hours? Because you got it 80% right. I was 19 or 20 back then, and Windows was my main OS - though I did have some experience with Linux as well (running several distributions for desktop use, as well as working with Cygwin and SFU on Windows), and I was a relatively proficient C++ programmer, having spent ~6 years of pretty much all my after-sc…

>Holy shit, how much does it cost to rent out your crystal ball for a few hours?

$300 per hour, I bring my own crystal ball, runes, chicken bones or voodoo doll based on customer requirements.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#432

Post by _AzMoo »

TeMPOraL wrote: post #391
lordnacho wrote: post #385

Earlier quoted context omitted.

> Not to mention, the entire financial industry That IS the dumpster fire. Trust me, the reason I know is that I used to be that guy who thought Excel was a great tool. I built derivatives spreadsheets, backoffice spreadsheets, trading systems with realtime data, all sorts of crap in Excel. Really, it's Stockholm Syndrome. People who previously had zero computing power at their disposal think they've found the hammer…

You bring up some very good points. But to those, I have a question: what is the alternative? Both now (for UK healthcare), and in general? The "people who previously had zero computing power at their disposal" may be wrong in thinking "they've found the hammer that solves all problems" - but there's literally no other hammer available for them. They're not programmers, they won't write their own software (nor would…

The alternative is to pay professionals to do it for them.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#433

Post by monocasa »

srtjstjsj wrote: post #364
retbull wrote: post #336

Earlier quoted context omitted.

I don't really agree with much of google any more but I was at one point fairly involved in a project there. They do have some more secure systems for government use that isn't the crazy leaky ad cesspool gmail and docs is.

Where are the ads in Docs?

The docs are used for data into the advertisement systems.

I've heard on the grapevine that several fortune 500 companies are on O365 because Google wouldn't disassociate docs from data gathering, even at the level of high touch high paying customers and they rightfully consider their internal documents to be part of their IP. Microsoft says "sure" and flips a secret "don't even gather stack traces on the server" flag for these customers.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#434

Post by bigbubba »

skykooler wrote: post #306
bigbubba wrote: post #209

Earlier quoted context omitted.

Mimicking a spreadsheet with an RDBMS would be a neat hack, but I think the better solution is to simply use the database like a database. If an amateur carpenter is trying to hammer screws into wood, you hand him a screw driver, not try to invent some sort of hammer with a twisting mechanism that turns screws when you wack them. Use the right tool for the job, and use it like it's meant to be used. If the workers in…

> not try to invent some sort of hammer with a twisting mechanism that turns screws when you wack them. That's a thing, it's a manual impact driver. And sometimes it is the right tool for the job (for example, when you have stuck screws and a regular screwdriver would just strip the head).

Sure, kind of. You hit the impact driver with a hammer, but I wouldn't call the driver a hammer.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#435

Post by nobody9999 »

tqi wrote: post #319
lordnacho wrote: post #247

The problem is that anyone who isn't a decent coder think excel is a sensible choice for this, because they've seen it in their school or work. So when the proposals come in, you're going to see one guy who says it's all common sense and we use familiar old excel for everything, and another lunatic who says something called "pigsqueal" is actually the standard, connected to a "frontend" which for some reason is now s…

Your scenario makes it seem like these things are the result of incompetence or stupidity by people who aren't "decent coders". The reality is that for most use cases excel+manual processes are perfectly adequate/have little startup costs and what you're suggesting is grossly over engineered/require immense initial investment. The real problem is if/when scope begins to increase these processes are sufficient until t…

>The real problem is if/when scope begins to increase these processes are sufficient until they aren't, and the tools aren't flexible enough to gracefully manage the transition.

That's a good point.

I'd also add that, at least in the US, we don't have any sort of national standard for such things.

Each public health department (and there are >1,000 of them in the US) has their own set of processes and procedures.

I'd expect that the CDC has a proper database system and that data from those 1000+ entities is collated into that system.

However, state and local laws/rules require specific handling of such data and can differ significantly from state to state, county to county or even town to town.

Attempting to stand up an integrated nationwide system isn't a bad idea. But doing so while trying to manage a pandemic isn't reasonable.

That many places used Excel spreadsheets for storing testing data is neither surprising or necessarily a bad thing. Until March, no one needed to have huge databases of test results -- now we do.

Complaining that places which had , at most, a few dozen cases of reportable infectious diseases should have implemented a database that can support thousands (tens of thousands?) of test results seems rather silly to me.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#436

Post by segfaultbuserr »

theptip wrote: post #238
tannhaeuser wrote: post #39

Earlier quoted context omitted.

> exchange formats are text based, with labs saying they will upgrade to „modern xml formats“ Worth noting that XML is also a text format. SGML even can treat CSVs as markup. There's nothing wrong with CSVs/TSVs anyway - it's a concise tabular format using only minimal special coding for a record and a field separator, as envisioned by ASCII and EDIFACT. The problem seems more like that there was no error checking in…

> There's nothing wrong with CSVs/TSVs I can see the theoretical purity of this statement, but based on my experience working with CSV files generated by actual non-technical users I have to disagree here. There are a number of footguns here that are really subtle and the average non-technical user has no hope of spotting them. Problems that I've seen in the wild, off the top of my head: * Windows vs. Linux line term…

Almost none [0] of the problem, such as line terminators, encodings, or escape characters, is an inherent flaw in the idea of using a text-based file to represent data. They can be fixed by just a little bit of standardization. But unfortunately, I don't think it's possible, we're stuck at here.

[0] Except type conversion, which is a real problem.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#437

Post by konjin »

Aeolun wrote: post #376
srtjstjsj wrote: post #354

Earlier quoted context omitted.

> Goes through the same process of QA and line-by-line checking of requirements. That's simply not possible unless you have a whole system around excel running a year harness. You can maybe do it if you are super dedicated with locked cells and formal double-blind QA passes which maybe exist somewhere but not in the vast majority of operations.

> formal double-blind QA passes I don’t know what you think I have in my unit tests, but it’s almost certainly not whatever you mentioned. Using excel as an alternative to CSV is fine. It’s not like anyone was doing anhthing complicated here. Just storing data.

>Using excel as an alternative to CSV is fine.

Clippy the Paperclip

Hi, it looks like this alphanumeric constant is the date format used by the Democratic Peoples Republic of Arstotzka for the year 127 BC. I have changed the encoding for the file to Arstotzka standard and overwritten the original.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#438

Post by CamperBob2 »

ImaCake wrote: post #68
CamperBob2 wrote: post #21

Earlier quoted context omitted.

This. We need to drop calculus, even geometry if we have to, in order to make room in the curriculum for statistics. Statistical illiteracy kills.

I mean, statistics relies heavily on calculus. It might be best to try and fit both in. Something at least a little bit fun and educational could be achieved with some dice throwing or simple computer simulations. High school students place dice games, they will immediately appreciate how useful a bit of probability would be in that context.

I mean, statistics relies heavily on calculus

No, it doesn't. Not the kind you'd teach in high school to help young people grow up and maintain a functioning democracy.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#439

Post by CamperBob2 »

OJFord wrote: post #110
CamperBob2 wrote: post #21

Earlier quoted context omitted.

This. We need to drop calculus, even geometry if we have to, in order to make room in the curriculum for statistics. Statistical illiteracy kills.

Where did you go to school that you didn't do statistics?

Just about anywhere in the US?

Remember that HN comments are an exercise in selection bias.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#440

Post by syshum »

konjin wrote: post #407
TeMPOraL wrote: post #369

Earlier quoted context omitted.

My 4th order Runge-Kutta ODE solver I did in an afternoon in Excel the other day, because I didn't want to pirate Matlab to do a homework assignment, begs to differ :). Not to mention, the entire financial industry and many (most?) small businesses. Lots of people use Excel for various types of math, and it works fine, if you know what you're doing . Excel not being idiot-proof doesn't mean it's not usable. And that'…

Having worked in the financial industry, my first job was writing a system to cover for a $10m fuck up in excel where yahoo exchange rates stopped updating and the same spreadsheet was used for a few weeks until someone noticed "Shit, we're losing a lot of money here.". Excel is the right solution in the same way playdoh is a valid building material.

>>Excel is the right solution in the same way playdoh is a valid building material.

I sooo badly want that on a T-Shirt.....

Post reply on HN