My time to shine. I work in the healthcare industry as an analyst and deal with all types of EMRs, Databases, and Molecular profiles. Before working in the healthcare industry, I would have assumed that the data quality in healthcare would be higher because you know, data quality errors can negatively effect patient outcomes. This is not the case.
A lot of these issue lie (lay?) with the UI of these systems. They are just not intuitive. This causes data to be entered incorrectly as honest mistakes. There are also fields which are free text, which should not be. I can't count how many times, I received data from a free text field for drugs, for example, and instead of information about drugs it said "The patient's pet passed away and they were sad today".
Furthermore, a lot of the data comes in separated value files, mostly CSVS (though I try to have them use TSVs if they can modify it). For such a ubiquitous format, at the volumes of data we receive, we get issues with the CSVs all the time. Lots of fields have strings that are improperly quoted, newlines in the strings, variable amount of fields per line, the usual.
That's on top of encoding issues. Randomly we will get files that aren't utf-8 and our parsers will blow up, and then we have to go back to the vendor and ask them to regenerate the data. Also trying to harmonize data across a bunch of proprietary file formats, and software is terrible
Also physicians and nurses are generally pretty skeptical when it comes to new technology even if you have a good product, and for good reason. They have been BURNED many times but companies coming in and promising the world, and delivering a half baked product. Anyone can get an MVP together quickly that kind of does what they need done, but to really support end users requires a lot more investment. I'm talking on-site trainings, software engineers actually watching the faculty USING THE TOOL, and taking notes when they struggle, being proactive about the wants and needs of the staff instead of adding features that seem useful. This almost never happens
Then there are factors outside of just EMRS and the user experience that just make the patient experience much more difficult, even when everyone involved is following standard practices. Data is king in this space, and companies are reticent to share it without lots of money and an books worth of clauses and agreements. You can win in this space just by having more data, or rare data. Lots of patient populations for obscure diseases require large sample sizes before you will have enough patients to perform a study. Pharma companies will approach you with prospective studies and the more data you have the more feasibility work you do that end up leading to studies where you get paid, and your name on paper. This HURTS patients and does nothing but help to keep competitive advantages.
On the bright side, since the U.S healthcare system is so capitalistic, we can usually find what we need coded in the billing data, because patients are nickle and dimed for everything that happens at a medical facility.
I have about twenty more pages of things I could talk about. This whole system is terrible and it doesn't have to be. On a more positive note, I love being able to track patient outcomes and see them improving, and I really enjoy working with huge datasets.