Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientific datasets are riddled with copy-paste errors

sciencedetective.org

Post reply on HN
Print view View original

41–50 of 68 posts

Re: Scientific datasets are riddled with copy-paste errors

#41

Post by kqr »

coppsilgold wrote: post #21

What should give people pause is how not complicated (I'd hesitate to say easy) it would be to create a python script that would generate fake data such that it would be all but impossible to determine whether it's real or not. You just need to model the measuring device and hypothesis you want to support, then sample away. The people who get caught red handed like this are lazy, incompetent and stupid. Makes you won…

The conspiratory reason would be that copy-paste errors give plausible deniability of ill intent.

Re: Scientific datasets are riddled with copy-paste errors

#42

Post by mcdonje »

IanCal wrote: post #34
jcattle wrote: post #29

Earlier quoted context omitted.

Yes, immediately thought the same. CSV alone is a footgun and a half on any computer which doesn't have . as the decimal separator. Let alone column sorting and joining of data.

CSV occupies, even years after moving away from more raw data work, way too much of my brain is still dedicated to "ways of dealing with CSV from random places". I can already hear people who like CSV coming in now, so to get some of my bottled up anger about CSV out and to forestall the responses I've seen before * It's not standardised * Yes I know you found an RFC from long after many generators and parsers were w…

You're blaming a lot of normal ETL problems on DSVs.

Like, specifying date as a type for a field in JSON isn't going to ensure that people format it correctly and uniformly. You still have parsing issues, except now you're duplicating the ignored schema for every data point. The benefit you get for all of that overhead is more useful for network issues than ensuring a file is well formed before sending it. The people who send garbage will be more likely to send garbage when the format isn't tabular.

There are types and there is a spec WHEN YOU DEFINE IT.

You define a spec. You deal with garbage that doesn't match the spec. You adjust your tools if the garbage-sending account is big. You warn or fire them if they're small. You shit-talk the garbage senders after hours to blow off steam. That's what ETL is.

DSVs aren't the problem. Or maybe they are for you because you're unable to address problems in your process, so you need a heavy unreadable format that enforces things that could be handled elsewhere.

Re: Scientific datasets are riddled with copy-paste errors

#43

Post by kriro »

Innocent mistakes and frustrating back and forth are also very common, especially for interdisciplinary teams. The mismatch in tooling and workflows and manual copy & paste conversion is a thing to behold. Add multiple countries and Excel to the mix (dot vs comma, formulas being language specific), maybe have a couple of Chinese, Japanese, Russian or Arabic speking researchers in the group for some extra UTF-8 magic. Line endings on Linux vs. OSX vs. Windows.

Re: Scientific datasets are riddled with copy-paste errors

#44

Post by jcattle »

mcdonje wrote: post #42
IanCal wrote: post #34

Earlier quoted context omitted.

CSV occupies, even years after moving away from more raw data work, way too much of my brain is still dedicated to "ways of dealing with CSV from random places". I can already hear people who like CSV coming in now, so to get some of my bottled up anger about CSV out and to forestall the responses I've seen before * It's not standardised * Yes I know you found an RFC from long after many generators and parsers were w…

You're blaming a lot of normal ETL problems on DSVs. Like, specifying date as a type for a field in JSON isn't going to ensure that people format it correctly and uniformly. You still have parsing issues, except now you're duplicating the ignored schema for every data point. The benefit you get for all of that overhead is more useful for network issues than ensuring a file is well formed before sending it. The people…

I would kind of disagree.

We are talking here in the context of scientific datasets. Of course ETL plays a part here. However here it is really more the interplay of Excel with CSV which is often outputted by scientific instruments or scientific assistants.

You get your raw sensor data as a csv, just want to take a look in excel, it understandably mangles the data in attempt to infer column types, because of course it does, its's CSV! Then you mistakenly hit save and boom, all your data on disk is now an unrecoverable mangled mess.

Of course this is also the fault of not having good clean data practices, but with CSV and Excel it is just so, so easy to hold it wrong, simply because there is no right.

> so you need a heavy unreadable format

I prefer human unreadable if it means I get machine readable without any guesswork.

Re: Scientific datasets are riddled with copy-paste errors

#45

Post by tempaccount5050 »

tremon wrote: post #39
tempaccount5050 wrote: post #35

Earlier quoted context omitted.

I think that's a little unfair. It really comes down to parsing text and you'll have similar issues even if you use a database or whatever you think the "real" solution is. I have a project I'm working on right now that stores dates, phone numbers, and website links. Cleaning/parsing has been 90% of the work and I still have edge cases that aren't fully solved. Every time I think I'm done, I find something else I hav…

that's a little unfair. It really comes down to parsing text and you'll have similar issues even if you use a database or whatever Feel free to show a real-world example of a database or whatever that takes the input string "IGF1 SEPT2 PRX3 MARCH1" and writes that into storage as ["IGF1", "2026-09-02", "PRX3", "2026-03-01"]. Also with Excel, an inadvertent click+drag can move data between cells, and since the cells a…

You still have to push and pull from the db. Meaning transforms still need to happen in either direction. I get what you're saying but it's just as easy to screw up a regex in either direction. Or making assumptions about how your language of choice will handle dates etc.

Re: Scientific datasets are riddled with copy-paste errors

#46

Post by adrianN »

IanCal wrote: post #34
jcattle wrote: post #29

Earlier quoted context omitted.

Yes, immediately thought the same. CSV alone is a footgun and a half on any computer which doesn't have . as the decimal separator. Let alone column sorting and joining of data.

CSV occupies, even years after moving away from more raw data work, way too much of my brain is still dedicated to "ways of dealing with CSV from random places". I can already hear people who like CSV coming in now, so to get some of my bottled up anger about CSV out and to forestall the responses I've seen before * It's not standardised * Yes I know you found an RFC from long after many generators and parsers were w…

This is an excellent rant, thanks for sharing. I didn’t have to work with csv s mich as you, but what experience I had I share your sentiment.

Re: Scientific datasets are riddled with copy-paste errors

#47

Post by xorcist »

IanCal wrote: post #34
jcattle wrote: post #29

Earlier quoted context omitted.

Yes, immediately thought the same. CSV alone is a footgun and a half on any computer which doesn't have . as the decimal separator. Let alone column sorting and joining of data.

CSV occupies, even years after moving away from more raw data work, way too much of my brain is still dedicated to "ways of dealing with CSV from random places". I can already hear people who like CSV coming in now, so to get some of my bottled up anger about CSV out and to forestall the responses I've seen before * It's not standardised * Yes I know you found an RFC from long after many generators and parsers were w…

While I appreciate a good rant just as much as the next person, most of these points have nothing to do with CSV. They are a general problem with underspecifying data, which is exactly what happens when you move data between systems.

The amount of hours I have wasted on unifying character sets across single database tables is horrifying to even think about. And the months it took before an important national dataset that supposedly many people use across several types of businesses was staggering. That fact that that XML came with a DTD was apparently not a hindrance to doing unspeakable horrors with both attributes and cdata constructs.

Sure, you can specify MM/DD/YY in a table, but it people put DD/MM/YY in there, what are you going to do about it? And that's exactly what happens in the real world when people move data across systems. That's why mojibake is still a thing in 2026.

Re: Scientific datasets are riddled with copy-paste errors

#48

Post by Starman_Jones »

adampunk wrote: post #8
dataviz1000 wrote: post #7

Earlier quoted context omitted.

> their workflows aren't great Sounds like a startup idea.

You'll want to sit down when I tell you the budget these folks have for workflow solutions. Ain't gonna take long but might be shocking if you've got big startup hopes. ;)

This was almost two decades ago, but I worked in a lab running particle detection experiments from an “internet-capable”computer that started life with “Windows 98 already installed- no upgrade needed.” Any “workflow solutions” talk started and ended with “Can we get undergrads to do it for class credit?”

Re: Scientific datasets are riddled with copy-paste errors

#49

Post by CWwdcdk7h »

l5870uoo9y wrote: post #15

> It could be either a fat-finger mistake when editing the Excel file or deliberate tampering to cover up real data that didn't tell the right story. I can easily imagine after spending years or decades devoted to discovering a scientific breakthrough that some could be tempted to slightly alter the data. I believe there was some scandal about this a few years back with climate data. Fixing this is however something…

> some could be tempted to slightly alter the data

We even suspect the G. Mendel of altering his pea data, it is unlikely he got results so close to predicted ratios. So it is not "some", everyone is tempted to "clean up" data, be it by removing outliers or by duplicating "good" rows.

Re: Scientific datasets are riddled with copy-paste errors

#50

Post by cowartc »

The real rate is certainly higher because this only catches the laziest form of error. The harder problem is the same one we see in production ML. Your pipeline can produce confident results on garbage data and nothing in the system tells you. The first step isn't better models or better tools, its profiling the input before you trust anything downstream of it.
Post reply on HN