Live data from Hacker News

Autocorrect errors in Excel still creating genomics headache

nature.com

51–60 of 108 posts

Re: Autocorrect errors in Excel still creating genomics headache

#51

I generally am very critical of my own gang ( programmers ) for being so into their own bubble they completely lose perspective of the real World, and with that ignorance comes the usual arrogance. But this time I feel we should give the business to the geneticists. The Excel "problem" is known and has been known for a long time, finance guys and policy makers were maybe the first big ones to "discover" this for slig…

The underlying assumption here is that when transferred into R or Julia, with an equivalent level of development effort and testing, that there will be no/fewer bugs. I'm not entirely sure that's true.

I think we are comparing something that is knocked out in an hour in excel to someone spending 4-10x of the time doing it in Python / Julia, but I don't think that's a like-for-like comparison.

Re: Autocorrect errors in Excel still creating genomics headache

#52

So we blame the software instead of blaming the downright lazy people who can't be fucked learning an appropriate tool even though that's their entire career. Honestly these kind of things really wind me up.

> So we blame the software instead of blaming the downright lazy people who can't be fucked learning an appropriate tool even though that's their entire career.

Further, the software would be fine if you use it properly for this sort of use case (i.e. bring the data in via PowerQuery which has enforced types).

Re: Autocorrect errors in Excel still creating genomics headache

#53

We need a clippy type feature "It looks like you are working on genomics data, should I turn off all the data destroying auotcorrect features?"

"It looks like you are working on genomics data, should I direct you to a search engine to find appropriate tooling instead?"

Re: Autocorrect errors in Excel still creating genomics headache

#54

That’s the price we pay. A reasonable alternative would be for example statically typing each column or row in the sheet. So in one sheet column A contains only floats, col B contains only text, col C contains only dates etc. And in another sheet col A may be date type, col B date type also, col C and D text, and col E monetary amounts. But this would come at the cost of not being able to mix types in column, and a l…

> A reasonable alternative would be for example statically typing each column or row in the sheet.

You can actually statically type the columns now for imported data via the PowerQuery editor (which is built into Excel), although not a lot of people know how to use it.

Re: Autocorrect errors in Excel still creating genomics headache

#55

Earlier quoted context omitted.

If a problem is re-occurring and between lots of people, it's probably the fault of the system that powers it. The system in this case is the software, so blaming the software is appropriate.

No it isn't. This isn't a gotcha. This is probably one of the most well known software behaviors on the planet. You KNOW the problem exists, yet you STILL choose to use the software, THEN expect it to change its behaviors because its inconvenient for your very specific use case. Microsoft aren't going to fix this behavior because there are likely millions if not billions of spreadsheets that rely on it. That is not a…

People don't choose to use Microsoft products, the software is chosen for them by their organization. For a problem like this, the existence of the article may very well be the first step to switching to another solution. The first step in that process is raising awareness that there is a problem.

Re: Autocorrect errors in Excel still creating genomics headache

#56
post #24

Earlier quoted context omitted.

Yeah. At some point though, people who want to do something that requires python or R, but refuse to learn it, and then use Excel (and make mistakes like the article mentions) should reconsider their choices. This is really out of character for me, but in this instance, relying on excel to do everything is a mistake by the user, not Microsoft's ridiculous data destroying import function. Use the right tool.

> Use the right tool. Step one should be learn your tool, no matter the tool. There's several ways to avoid the issue in Excel. If someone working with Excel isn't able to learn that (heck, a simple template would suffice), I have no hope the same demographic would have any success with R or Python.

> There's several ways to avoid the issue in Excel.

No, not really. There are several ways to help reduce the issue, but none of them eliminate what it does.

Example that comes to mind is the data file for College Board's SAT test. The data formats for student reports for schools come in two formats: PDF (one page per student), CSV, and fixed-width. That is the comprehensive list of your options. College Board doesn't care about you as a customer. They're too big. Any request you submit will be black holed.

Some of the columns in the file indicate a range, usually in the format "X-Y". Excel will try to coerce that into a date, if it's valid.

Other columns indicate a ratio, expressed as "X/Y". Excel will coerce that into a date, if it's valid.

Other columns indicate an ID number, expressed as a large, fixed-digit number, zero-padded. Excel will coerce that into an integer, or, if it's too long, into scientific notation discarding digits.

It doesn't matter how you format the CSV. Excel will do the above.

Here's an example CSV:

  ID,Range,Ratio
  12345678901234567890,8-9,"7/15"
  "00000000000000012345","9-10",21/35
I open that with Excel and immediately save it as a CSV. I look at the file in a text editor and I see:

  ID,Range,Ratio
  1.23457E+19,9-Aug,15-Jul
  12345,10-Sep,21/35
Do you have any idea how fun it is to explain to teachers and school administrators what happened here?

The correct way to work with this data file is: Do not, under any circumstances, open it with Excel if you expect to use it for anything else.

The problem is, there are very few applications that work well with CSV files. I know of CsvEd and Delimit. There are several text editors with a CSV column mode that makes the file look like a table (with varying degrees of success). All of these vary between "godawful" and "a complete nightmare" in terms of performance and usability compared to Excel.

Re: Autocorrect errors in Excel still creating genomics headache

#57
post #4

Earlier quoted context omitted.

Because an absolutely enormous number of people involved in all ends of genomics and biomedical science have absolutely no inkling whatsoever about programming. Furthermore, Microsoft office is installed everywhere everywhere everywhere… like it leave it love it or hate it, it doesn’t matter… it’s true (at least for huge swaths of the demographic). So the first thing most people learn is excel and the last thing most…

Yeah. At some point though, people who want to do something that requires python or R, but refuse to learn it, and then use Excel (and make mistakes like the article mentions) should reconsider their choices. This is really out of character for me, but in this instance, relying on excel to do everything is a mistake by the user, not Microsoft's ridiculous data destroying import function. Use the right tool.

Not specifically this topic but I ran into an issue that would have been trivial in any real programming language this week, but I’m not able to USE anything that’s not on my work computer due to data restrictions.

So I get to cobble together some really ugly spreadsheets in excel when I’d MUCH rather use something more appropriate. Sometimes it’s not the person, it’s the org. In this case, government.

Re: Autocorrect errors in Excel still creating genomics headache

#58
post #2

Can anyone with first hand experience on this tell shed some light on why stick with Excel versus using python, R or what have you to treat the data?

The same line of questioning can easily land in "why use csv?"

It is a terrible format that has very few redeeming qualities. It happens to also be the most widely used one.

Re: Autocorrect errors in Excel still creating genomics headache

#59
post #24

Earlier quoted context omitted.

> Use the right tool. Step one should be learn your tool, no matter the tool. There's several ways to avoid the issue in Excel. If someone working with Excel isn't able to learn that (heck, a simple template would suffice), I have no hope the same demographic would have any success with R or Python.

> There's several ways to avoid the issue in Excel. No, not really. There are several ways to help reduce the issue, but none of them eliminate what it does. Example that comes to mind is the data file for College Board's SAT test. The data formats for student reports for schools come in two formats: PDF (one page per student), CSV, and fixed-width. That is the comprehensive list of your options. College Board doesn'…

If you open the csv file with Excel you get those errors. If instead one uses Get&Transform (aka Power Query) then it is possible to specify the type of each column.

Re: Autocorrect errors in Excel still creating genomics headache

#60
post #24

Earlier quoted context omitted.

> Use the right tool. Step one should be learn your tool, no matter the tool. There's several ways to avoid the issue in Excel. If someone working with Excel isn't able to learn that (heck, a simple template would suffice), I have no hope the same demographic would have any success with R or Python.

> There's several ways to avoid the issue in Excel. No, not really. There are several ways to help reduce the issue, but none of them eliminate what it does. Example that comes to mind is the data file for College Board's SAT test. The data formats for student reports for schools come in two formats: PDF (one page per student), CSV, and fixed-width. That is the comprehensive list of your options. College Board doesn'…

> It doesn't matter how you format the CSV. Excel will do the above.

Only if you open the file, so that Exel assumes defaults for formatting. If you import the CSV (Data -> From Text in the current UI) you can specify the format with the Import Text Wizard as described on the College Board instructions for using the file. Except...

Unfortunately, the College Board has outdated instructions on their website; Excel used to offer the Import Text Wizard on opening a csv or text file, rather than making default assumptions, and the College Board instructions page, while it does tell you to use that Import Text Wizard and provide all the details of what to plug into that wizard, tells you to open the file to get it, which bypasses the wizard.

Things like this is why clerical and other low-level positions involving excel test candidates on proficiency with specific versions of excel. Higher-level workers are expected to be able to figure out these kind of changes themselves, though (or consume lower-level staff time, if they are in management.

Post reply on HN