Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

821–830 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#821

Post by 8bitsrule »

topkai22 wrote: post #579
8bitsrule wrote: post #573

A DB system that doesn't verify that each submitted record has been included in the DB (and scream if it hasn't) ... is a DB cruising for a bruising.

It sounds like a data processing pipeline rather than a database itself. A systems A...E at hospital push csv files to a shared system F. System G pulls the data from F, passes it through an excel workbook (failure point!) for data augmentation and cleanup tasks built by a public health SME, the drops the data in location H. System I loads the data then finally loads the data to a “real” database for additional repor…

Wow, what a mess. At the very least (which is about all that could be done) A could assign a new-record-ID to each record; I could tell F where to find it; F could verify its presence (or scream) and tell A.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#822

Post by nullsense »

pintxo wrote: post #812
nullsense wrote: post #300

Earlier quoted context omitted.

Fun fact: Toshiba has software that generates entire CRUD app skeletons driven 100% by excel spreadsheets. You fill out object definitions in excel spreadsheets with property names and data types etc. It's like an extremely uncool long lost cousin of rails -g Source: Got one of their engineers to show me after I heard about it and had to know if it was real. Anyone who's ever worked with or in Japan knows just how fa…

I've (miss)used excel as a web-service testing platform once. The test data (input/expected output) was anyway in the Excel sheet, so why not. We did not have other software available at the time, so Excel+VBA it is. It was one of my nicest testing gigs ever. A test session would take only minutes, all results documented to the t. And fiddling around with the test data was so easy. Would be interesting to know if thi…

>I've (miss)used excel as a web-service testing platform once. The test data (input/expected output) was anyway in the Excel sheet, so why not. We did not have other software available at the time, so Excel+VBA it is.

That's an upgrade on how Toshiba did testing. They did basically the same thing except it was Excel+Humans. No joke. Never saw that one with my own eyes but a coworker did. He also said they had doctors there and each doctor was responsible for X number of staff and every so often made them fill out a questionnaire of which the final question was "have you thought about killing yourself lately?" and if you answered yes you got a day off. Apparently people would ask for transfers to different offices where the doctor responsible for them wasn't physically present, so they could avoid even being asked the question for fear of being made to take time off work. One of the more senior guys on the project kinda just disappeared too. Fun times.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#823

Post by vertere »

stjohnswarts wrote: post #820
vertere wrote: post #599

Earlier quoted context omitted.

Perhaps by "correct way" you meant "dodgy hack to make Excel happy and risk breaking more sensible implementations"? Excel may predate the RFC but AFAIK MS didn't invent or coin the term CSV, so you can't just say whatever Excel does is correct. The RFC is loose because of nonsense like this, it doesn't mean it was ever a good idea.

How in the world is Excel supposed to know which fields you want to be numbers and which to be strings? CSV doesn't have that info built in unless you surround the number with quotes and you select the right process. Excel isn't just a CSV importer, it reads all sorts of files, and it needs some help if you expect it to work. What ever happened to process and a sense of responsibility and craft in your work?

Choosing based on whether it's in quotes seems like a better solution than using an equals sign. Not everything follows that convention admittedly, but not everything understands the '=' either. Or it could just treat everything as text until a user tells it otherwise.

But none of that is really the point. Because CSV files aren't just for importing into Excel. One of their main benefits is their portability. In other situations column types might be specified out of band, but even if not, putting equals signs before values is unconventional, so more likely to hurt than help. And in the cases it might help, i.e. when you only care about loading into Excel, then you have options other than CSV, rather than contorting CSV files for Excel's sake.

> What ever happened to process and a sense of responsibility and craft in your work?

I actually have no idea what you are on about. I'm talking about the "responsibility and craft" of not producing screwed up CSV files. Why do some people find that so offensive? Yes, it is not inconceivable that there could be some situation working with legacy systems where putting `="..."` in CSVs is, unfortunately, your best option. Sometimes you do have to put in a hack to get something done. But don't go around telling people (or yourself) that it is "the correct way".

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#824

Post by benhurmarcel »

fetbaffe wrote: post #780
benhurmarcel wrote: post #669

Earlier quoted context omitted.

I work in an office that relies way too much on Excel. To consider your solution, first showstopper, it needs a server. We don't have a server, nor anyone who knows how to manage one. We'd need to ask IT, that will take months and they'll require a budget transfer, so we'd need to request it to management (which will need a business case to convince) and involve the finance guys. We can't just plug a RaspberryPi into…

How do you share the spreadsheet? Shared network drive? By email? With Excel?

Shared network drive usually

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#826

Post by reportgunner »

iso1631 wrote: post #799
reportgunner wrote: post #728

Earlier quoted context omitted.

20,000 is not a lot of lines though, I used to scroll 300k sometimes. Anything below 500k rows is a 'small table' still.

The point is that's a manual, tedious, process that will be performed with error or completely skipped. We have computers to do that sort of work.

It is a manual and tedious process if you make it manual and tedious for yourself.

After all, it's people that make computers do this work in the first place.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#827

Post by slowhand09 »

jugg1es wrote: post #538

I work in healthcare and by far the biggest production issues we've ever run into are people who run Excel spreadsheets through our file processing where the leading zero's have been removed from patient identifiers because that's Excel's default behavior and you CANNOT TURN IT OFF! EDIT: I have no idea who downvoted my post because what I said is 100% true. We have to tell customers to stop opening CSVs in Excel and…

The issue I see is you have people who don't know the difference between a string data type and a number. I keep see "Listen to the scientists" and "Listen to the doctors". There are many facets to solving problems. Maybe many disciplines may be better at solving it. Had you/"healthcare" contacted folks with software expertise rather than being your own experts, this would have been a non-problem.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#828

Post by cabalamat »

Nightshaxx wrote: post #456
cabalamat wrote: post #450

Earlier quoted context omitted.

> It's just that you get a lot of "coding is a thing I have to do in order to get to some target", and so people think that once they've finally bashed out their spreadsheet, they've figured it all out. I have a friend who wrote a 641-line long bash script to automate a web site. It doesn't use subroutines anywhere, the body for the program is a 550 line long loop with multiple if statements and loops inside it. He t…

Wha...... What?? What school or book did he read that made him think that was a good idea?

He's entirely self taught. He's not stupid or anything, it's just that he hasn't had the experience of maintaining large codebases, attempting to modify his unmaintainable code or attempting to modify other people's unmaintainable code.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#829

Post by cabalamat »

TeMPOraL wrote: post #622
Nightshaxx wrote: post #456

Earlier quoted context omitted.

Wha...... What?? What school or book did he read that made him think that was a good idea?

He attended the school of self-reliance. He should be applauded for tackling the challenge and doing the job himself, where otherwise it would likely have not been done at all. Obviously, he's not a programmer. The script, for us, is bad. Perhaps if something important for him depends on this script working, he should pay a software developer to spend some time cleaning it up.

> He should be applauded for tackling the challenge and doing the job himself

Indeed. it was a hobby project of his.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#830

Post by manigandham »

callamdelaney wrote: post #105
beagle3 wrote: post #85

This is indeed user incompetence, but so far every news source quoted the official PR and said it was an "Excel Problem". Microsoft PR was caught unprepared - I wonder how they'll re-spin it in the next few days (and for the first time that I can recall, a Microsoft product was wrongly blamed...) Pay attention, how every time there's a Windows virus or worm, it's a "Computer Virus", but in the (extremely rare) occasi…

It's not a user issue, it's a government incompetence issue. Who was the technical genius who thought 'right, we need to store thousands of lines of critical and important data, I know! Excel!'

Excel is perfectly fit for that and is nowhere near the actual issue.

It's lightweight, fast, portable, compatible, and just about everyone can use it regardless of (technical) skill.

Post reply on HN