Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Autocorrect errors in Excel still creating genomics headache

nature.com

Post reply on HN
Print view View original

71–80 of 108 posts

Re: Autocorrect errors in Excel still creating genomics headache

#71

Post by dec0dedab0de »

nnmg wrote: post #63

Excel is used as a database/storage/interchange format, especially after the initial analysis by someone who uses python or R. Bioinformatician does the analysis, then the PI wants to see it so they can Ctrl-F for genes they are interested in, so out comes an excel document. And really, even if you know python or R, are you really going to fire up a jupyter notebook, load the data, and run pandas queries every time s…

And really, even if you know python or R, are you really going to fire up a jupyter notebook, load the data, and run pandas queries every time someone in lab meeting or after a talk asks you about this gene or that gene in your data?

I don't do any scientific research, but I have been using jupyter as a replacement for excel since it was called the ipython notebook. I don't really use pandas all that often, I just find it easier to read and edit data in python. Though I first learned ipython added the notebook from a talk Wes McKinney gave about Pandas.

Re: Autocorrect errors in Excel still creating genomics headache

#72

Post by mceoin »

planet-and-halo wrote: post #11

Serious question, is this a viable area for a startup product? Obviously nothing will easily displace Excel for general spreadsheet work, but I could see room for a domain-specific spreadsheet that has core features and supports just a basic set of functions relevant to genomics, + a "we won't fuck up your data" feature.

Hi - I'm building a networked spreadsheet product and had heard of this auto-correct problem in genomics research through word of mouth a couple of times, so have asked this question myself. (I was astonished to learn just how much of genomics work involves sending Excel files back and forth!) Here's the conclusion that I have come to:

- It's definitely possible to build a custom spreadsheet product with a small team, even one targeted at such a "niche" user group. So it's an idea worth testing.

- Product can be "backwards compatible" - you can export to xlxs and import from xlxs - so you don't have to change behaviour of the entire industry on day one to get this to work, only a single genomics researcher or lab.

- Pricing, unit economics, etc are unknown to me (I have no background in genomics or scientific research). But presumably you could leverage standard SaaS models and build a viable model up from there using a few case studies. There's definitely schleppy behaviour going on here that can be solved.

- Even a "lifestyle business" has significant upside beyond the financial: improving genomics research improves genomic research!

- The product advantage over time presumably involve building more custom tooling into the genomics / data ecosystem. "Not creating typos" is just the beachhead.

I've never actually interviewed genomics people about their need here, but if anyone knows people with this problem I would love to talk to them: @mceoin on twitter. (DMs open)

Re: Autocorrect errors in Excel still creating genomics headache

#73

Post by soco »

gpapilion wrote: post #6

Autocorrect(and predictive text) has reached an interesting milestone, good enough that you mainly rely on it, but occasionally it distorts the data you intended to send. Disengaging the technology isn't easy, or even understanding field types it shouldn't correct(everyday my work VPN auth window tries to autocorrect my username). There is so much context that is unavailable to the computer, and there is no good way…

I get bitten by Excel's auto-correct all the time and I work nothing related to genomics. It's in my opinion just a dumb decision of Microsoft to not let you disable all those smarty feats (like, thinking you have a date if your string kinda looks like one)

Re: Autocorrect errors in Excel still creating genomics headache

#74

Post by SonicScrub »

nnmg wrote: post #63

Excel is used as a database/storage/interchange format, especially after the initial analysis by someone who uses python or R. Bioinformatician does the analysis, then the PI wants to see it so they can Ctrl-F for genes they are interested in, so out comes an excel document. And really, even if you know python or R, are you really going to fire up a jupyter notebook, load the data, and run pandas queries every time s…

I don't work in bioinformatics, but what you are describing is a completely accurate description of what I experienced working in manufacturing quality control. Raw data came in from suppliers in the form of spreadsheets, and management wanted to see results in spreadsheets. Meaning all our quality data was subjected to these issues. The date formatting issue was a particularly annoying "gotcha", particularly when features were defined with a XX-XX numeric code. The number of times I had to deal with someone in a meeting saying "hey, why is this feature called October-13?!" Super frustrating.

If I could choose the tools used by the whole process involving multiple different companies and departments, hey I would! It would be python all the way down. But I was but a cog in a massive organization.

Re: Autocorrect errors in Excel still creating genomics headache

#75

Post by IAmEveryone »

breakfastduck wrote: post #45

So we blame the software instead of blaming the downright lazy people who can't be fucked learning an appropriate tool even though that's their entire career. Honestly these kind of things really wind me up.

There is one problem people have for their use case, so it’s inappropriate? Have you ever created an issue on some GitHub project? I guess you were using the wrong tool?

People choose their tools for some reason(s), and there is no law of man or nature that says that using a common tool for some esoteric purpose is somehow wrong, just because you consider it too easy.

Re: Autocorrect errors in Excel still creating genomics headache

#76

Post by dragonwriter »

SonicScrub wrote: post #74
nnmg wrote: post #63

Excel is used as a database/storage/interchange format, especially after the initial analysis by someone who uses python or R. Bioinformatician does the analysis, then the PI wants to see it so they can Ctrl-F for genes they are interested in, so out comes an excel document. And really, even if you know python or R, are you really going to fire up a jupyter notebook, load the data, and run pandas queries every time s…

I don't work in bioinformatics, but what you are describing is a completely accurate description of what I experienced working in manufacturing quality control. Raw data came in from suppliers in the form of spreadsheets, and management wanted to see results in spreadsheets. Meaning all our quality data was subjected to these issues. The date formatting issue was a particularly annoying "gotcha", particularly when fe…

> A lot of "safety culture" is composed of things like checklists and hazard warnings which are more geared towards shifting the blame for accidents onto somebody else than actually preventing those accidents,

If you stay in spreadsheets these problems mostly don’t occur (that is, once data entry is squared away so that the initial spreadsheet has what you want it doesn't tend to get lost), its when you move in and out of spreadsheets via text and take the path of least resistance [0] to do the transition that the problem occurs.

[0] and to be fair, there is a lot of resistance off that path.

Re: Autocorrect errors in Excel still creating genomics headache

#77

Post by breakfastduck »

ironmagma wrote: post #65
breakfastduck wrote: post #61

Earlier quoted context omitted.

A convenient cop out there. "I didn't install it, I only chose to use it for something it isnt suited for instead of looking for alternatives, not my fault"

People working in large organizations are frequently not even allowed to install software that isn’t pre-approved by N layers of management. The bottom line though is that Excel is made for data entry. If it’s not suitable for that use case, that’s a big problem with the product-market fit.

I cannot imagine a single piece of software that is more widespread and used without issue by huge swathes of people than excel.

But one specific use case for genetics means its totally unsuitable & and big problem with the product. Got it.

Re: Autocorrect errors in Excel still creating genomics headache

#78

Post by BugsJustFindMe »

nnmg wrote: post #63

Excel is used as a database/storage/interchange format, especially after the initial analysis by someone who uses python or R. Bioinformatician does the analysis, then the PI wants to see it so they can Ctrl-F for genes they are interested in, so out comes an excel document. And really, even if you know python or R, are you really going to fire up a jupyter notebook, load the data, and run pandas queries every time s…

> Excel is used as a database/storage/interchange format, especially after the initial analysis by someone who uses python or R. Bioinformatician does the analysis

Sometimes, but the situation is in reality worse than that. Excel is also used as the gold standard database/storage/interchange format of record for random shit that clinical researchers have typed in by hand whether directly or transcribed from other notes, often when that data isn't actually fundamentally tabular in nature because people really like working in grids. Even when grids hurt more than help.

A big secret in genetic research is that the MDs, grad students, project managers, and coordinators running the research programs are often not super focused on what well-structured data looks like and don't know what things like "key-value store" or "nested tree-like structure" mean, and even if they did there aren't good GUI tools for entering them anyway, and it leads to countless errors that maybe (here I speculate) they just assume will wash out as noise.

> I think the important question is why is date conversion a default?

Yes, why any kind of conversion is ever the default is a real money question.

Re: Autocorrect errors in Excel still creating genomics headache

#79

Post by SonicScrub »

dragonwriter wrote: post #76
SonicScrub wrote: post #74

Earlier quoted context omitted.

I don't work in bioinformatics, but what you are describing is a completely accurate description of what I experienced working in manufacturing quality control. Raw data came in from suppliers in the form of spreadsheets, and management wanted to see results in spreadsheets. Meaning all our quality data was subjected to these issues. The date formatting issue was a particularly annoying "gotcha", particularly when fe…

> A lot of "safety culture" is composed of things like checklists and hazard warnings which are more geared towards shifting the blame for accidents onto somebody else than actually preventing those accidents, If you stay in spreadsheets these problems mostly don’t occur (that is, once data entry is squared away so that the initial spreadsheet has what you want it doesn't tend to get lost), its when you move in and o…

The process I had to deal with was filling out spreadsheets with data from a python-driven 3D inspection program that exported out data files in CSV format. Needless to say, these errors were inevitable for exactly the reasons you've stated. Why we didn't bypass the large, poorly formatted cumbersome spreadsheets and just directly export data via pandas? All the inspection was done by Python anyways. You tell me! Also, it did not help that the spreadsheets were not created by me, or any colleagues in my department.

God I hated working in old-school engineering/manufacturing. "That's not how we do things" is the answer to everything. I

Re: Autocorrect errors in Excel still creating genomics headache

#80

Post by bserge »

ironmagma wrote: post #55
breakfastduck wrote: post #50

Earlier quoted context omitted.

No it isn't. This isn't a gotcha. This is probably one of the most well known software behaviors on the planet. You KNOW the problem exists, yet you STILL choose to use the software, THEN expect it to change its behaviors because its inconvenient for your very specific use case. Microsoft aren't going to fix this behavior because there are likely millions if not billions of spreadsheets that rely on it. That is not a…

People don't choose to use Microsoft products, the software is chosen for them by their organization. For a problem like this, the existence of the article may very well be the first step to switching to another solution. The first step in that process is raising awareness that there is a problem.

If only the organization was run by people.
Post reply on HN