Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Missing Covid-19 test data was caused by the ill-thought-out use of Excel

bbc.com

Post reply on HN
Print view View original

631–640 of 834 posts

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#631

Post by TeMPOraL »

konjin wrote: post #431
TeMPOraL wrote: post #423

Earlier quoted context omitted.

Holy shit, how much does it cost to rent out your crystal ball for a few hours? Because you got it 80% right. I was 19 or 20 back then, and Windows was my main OS - though I did have some experience with Linux as well (running several distributions for desktop use, as well as working with Cygwin and SFU on Windows), and I was a relatively proficient C++ programmer, having spent ~6 years of pretty much all my after-sc…

>Holy shit, how much does it cost to rent out your crystal ball for a few hours? $300 per hour, I bring my own crystal ball, runes, chicken bones or voodoo doll based on customer requirements.

Does your divination work over video link too? I might take you up on this offer.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#632

Post by [deleted] »

thedudeabides5 wrote: post #519
totaldude87 wrote: post #491

All previous major spreadsheet screwups were costing money (largest being $6 billion in losses due to a excel formula mishap) , but this one is playing with lives :( 5 – London Olympics Oversells Swimming Event by 10,000 Tickets 4- Banking powerhouse Barclay’s accidentally bought 179 more contracts than they intended in their purchase of Lehman Brothers assets in 2008. Someone hid cells containing the unwanted contra…

Well, not exactly like non-Excel systems are bulletproof either. Entire Japanese stock market went down last week, and it's not like there was a flood of people on HN bemoaning that. At least with Reinhart and Rogoff et al you have a responsible party. As opposed to 'nameless machine failed, and nameless backup machine also failed, and now it's in JIRA so don't worry about it'. https://www.nytimes.com/2020/09/30/busi…

[deleted]

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#633

Post by collyw »

edanm wrote: post #589

As a data engineer, I think blaming Excel for this is the wrong framing. The real problem here is with the entire process, and specifically, the lack of testing. Excel is great for many use cases, especially if you need people to enter data somewhere. Its UI is unparalleled in terms of quickly giving something to users that they can understand, mess around with, and verify. It's a very common use case to then need to…

The main problem I have seen is that data in Excel is essentially not validated.

Sometimes you get an number, sometimes you get a string containing a number. Sometimes the string contains white space that you don't want. And then there are dates, which can have similar problems multiplied ten times.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#634

Post by tiernano »

collyw wrote: post #619
koliber wrote: post #613

Earlier quoted context omitted.

They are costing money. No doubt. At the same time, the business world runs on Excel. How much money is Excel making? I've done my share of cursing at Excel at various jobs. At the same time, I am grateful for the quick and easy way it allows me and many others to manipulate data. It's unfair to just cite the costs of using Excel without acknowledging the benefits it brings.

The real question should be would it be cheaper to replace it with a proper solution (probably something custom built application written by a software engineer). Excel's ease of use is it's downfall. It is the worlds most popular database, despite not actually being a database. I have wasted countless hours dealing with Excel where something else should have been used. I built a database for a friend recently, I thi…

it will only be cheaper after the company loses money... no business person ever looked at something and said "it works, but it might not, some day... better spend $XXX,XXX on an engineer to fix the problem".

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#635

Post by Closi »

collyw wrote: post #619
koliber wrote: post #613

Earlier quoted context omitted.

They are costing money. No doubt. At the same time, the business world runs on Excel. How much money is Excel making? I've done my share of cursing at Excel at various jobs. At the same time, I am grateful for the quick and easy way it allows me and many others to manipulate data. It's unfair to just cite the costs of using Excel without acknowledging the benefits it brings.

The real question should be would it be cheaper to replace it with a proper solution (probably something custom built application written by a software engineer). Excel's ease of use is it's downfall. It is the worlds most popular database, despite not actually being a database. I have wasted countless hours dealing with Excel where something else should have been used. I built a database for a friend recently, I thi…

The problem with a database is that business users struggle to build them, they take longer, and sometimes can’t hold logic in the same way.

Plus in a database you can’t do the same sort of real-time analysis and also pass the document around for other non-technical folk to add to and modify.

In the real world in big companies, people often don’t want to talk to IT because they over-spec and quote what are perceived to be giant sums of money for something that can be created in an hour in excel.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#636

Post by petre »

jugg1es wrote: post #538

I work in healthcare and by far the biggest production issues we've ever run into are people who run Excel spreadsheets through our file processing where the leading zero's have been removed from patient identifiers because that's Excel's default behavior and you CANNOT TURN IT OFF! EDIT: I have no idea who downvoted my post because what I said is 100% true. We have to tell customers to stop opening CSVs in Excel and…

Just format the identifier cells as text. I've also had this problem, this is how I solved it.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#637

Post by collyw »

throwaway_pdp09 wrote: post #595
edanm wrote: post #589

As a data engineer, I think blaming Excel for this is the wrong framing. The real problem here is with the entire process, and specifically, the lack of testing. Excel is great for many use cases, especially if you need people to enter data somewhere. Its UI is unparalleled in terms of quickly giving something to users that they can understand, mess around with, and verify. It's a very common use case to then need to…

I don't think data engineering is hard or needs much resources. It does need people who care and who will think about things. Anyone who comes at a problem with the mindset 'this is going to he hard' probably lacks experience and will throw big-data frameworks at it, really screwing things up. The most significant, and valuable, resource needed is thought first, and knowledge+experience second. All IMO anyway.

I always think that database design is pretty simple, but having worked on numerous databases designed by other, some people get it very badly wrong. "Wrong" maybe isn't the correct word, because you can use the database for what it is needed but it just requires extra joins and hacks at the application level to get around poor design.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#638

Post by janekm »

edanm wrote: post #589

As a data engineer, I think blaming Excel for this is the wrong framing. The real problem here is with the entire process, and specifically, the lack of testing. Excel is great for many use cases, especially if you need people to enter data somewhere. Its UI is unparalleled in terms of quickly giving something to users that they can understand, mess around with, and verify. It's a very common use case to then need to…

Collecting and distributing the results of life-or-death diagnostic tests is specifically one case where you don't want end users to "mess around with" the data though... (making sure the right test results get associated with the right record is surprisingly difficult and rigid software is part of that).

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#639

Post by theshrike79 »

corty wrote: post #600
edanm wrote: post #589

As a data engineer, I think blaming Excel for this is the wrong framing. The real problem here is with the entire process, and specifically, the lack of testing. Excel is great for many use cases, especially if you need people to enter data somewhere. Its UI is unparalleled in terms of quickly giving something to users that they can understand, mess around with, and verify. It's a very common use case to then need to…

Disagree on the "users can verify" part. Excel is very hard to verify, because of corner cases like date conversion. Formulas and macros, maybe spread out all over multiple sheets in hidden away white-on-white cells, make things very opaque and impossible to verify for most users. After a long gone colleague has passed the usage of some excel file, it is treated as an arcane and unchangeable wisdom from the past, nev…

I vaguely remember a discussion about this and the problem boils down to Excel sheets being impossible to unit-test.

There might be, for example, a formula going through 20k lines in column F. But the one on line 1138 has a typo and the formula references an incorrect cell. No human will ever go through all the lines to check. Excel itself doesn't check stuff like that. And there are no tools for it either.

Re: Missing Covid-19 test data was caused by the ill-thought-out use of Excel

#640

Post by eCa »

jugg1es wrote: post #538

I work in healthcare and by far the biggest production issues we've ever run into are people who run Excel spreadsheets through our file processing where the leading zero's have been removed from patient identifiers because that's Excel's default behavior and you CANNOT TURN IT OFF! EDIT: I have no idea who downvoted my post because what I said is 100% true. We have to tell customers to stop opening CSVs in Excel and…

I used to work with books, which obviously includes ISBNs. They are 13 digit numbers, helpfully turned into scientific notation by Excel.
Post reply on HN