so.. where can i get this data? XD
UK Biobank health data keeps ending up on GitHub
31–40 of 61 posts
Re: UK Biobank health data keeps ending up on GitHub
#32Earlier quoted context omitted.
In my experience with health data, the dates are usually offset by a random but constant amount for each person (e.g. id 12345 will have all their dates shifted by +5 weeks) to avoid identification by dates. Unfortunately the sequence of treatments and locations are usually enough to identify someone, especially if it's a rarer condition.
Location data is very readily available, so you can easily correlate visits to a health facility with a treatment, and even with an offset, you can probably uniquely identify someone with 4 visits depending on the size of the medical facility.
I had access to several health datasets for my research in the past. Date of birth was rarely given, especially for the bigger projects where there were more resources to allocate to privacy protection. Neither was date of death, location, or visits to a health facility with a treatment. Typically the relevant variables are age (in years), treatment type and possibly number of cycles. Probably insufficient to identify someone without access to hospital records. But if you have that, you have all these data anyways.
Most researchers likely would want to summarize these data in a similar way anyway, so this works out nicely.
Re: UK Biobank health data keeps ending up on GitHub
#33the issue is with jupyter notebooks because they keep some of the data in the output (typically a few rows, but still). They should strongly recommend to use regular python scripts, and keep the jupyter books just for verification, which is a very sane thing to do also from a SW engineering perspective.
I cant really understand why Jupyter Notebooks do this in the first place. It makes it (a) really hard to version control, as there will always be some random blob of non-textual data in the notebook that pops up in a diff and makes it basically unreadable and (b) I can't really see the benefit, as it only stores some part of the data, and not the full table, as far as I am aware.
Enforcing Jupytext is a good adaption, and gives you all the, arguably really nice, comfort from a notebook, and the proper code practice from SW engineering.
Re: UK Biobank health data keeps ending up on GitHub
#34What are the pros/cons of just open-sourcing everything for future bio bank projects?
They need to sell the data to fund the project
Re: UK Biobank health data keeps ending up on GitHub
#35What are the pros/cons of just open-sourcing everything for future bio bank projects?
It's exceptionally difficult to avoid the data being de-anonymised. If an 'anonymised' medical record says the person was born 6th September 1969, received treatment for a broken arm on 1 April 2004, and received a course of treatment in 2009 after catching the clap on holiday in Thailand - that's enough bits of information to uniquely identify me. And medical researchers are usually very big on 'fully informed conse…
... received a course of treatment in 2009 after catching the clap on holiday in Thailand
Yeah, sorry about that
Re: UK Biobank health data keeps ending up on GitHub
#36Thank you for sharing.
I work in a central RSE team and have raised this topic to the team, with a view of bringing attention to this issue and better educating our researchers (as part of our training offerings and documentation).
Re: UK Biobank health data keeps ending up on GitHub
#37> It has given 20,000 researchers around the world access under strict agreements that prohibit sharing data further. To me it seems rather naive to have done that. After all, you can't un-leak medical data. So even if the "strict agreement" included huge punishments, there's no getting the toothpaste back in the tube. If you want to ensure compliance before a leak happens you have to (ugh) audit their compliance. An…
Not giving the data to researchers means not getting the scientific benefits from that data. Which was the point of collecting that data in the first place.
Reckless harm prevention is the root of many evils.
Re: UK Biobank health data keeps ending up on GitHub
#38I've opted in to Australia's version of the biobank knowing that it's inevitable that it will be leaked some day, I think the data is so valuable in perpetuity that it's worth it. I remember Ben Goldacre has been working on how to make data more accessible in a safer way to (in part) avoid this very thing, but I haven't heard much of it since [0] [0] https://www.bennett.ox.ac.uk/blog/2025/02/opensafely-in-brie...
This is the right mindset. Securing huge piles of heterogeneous data while giving PhD students the freedom to "play" with it are quite conflicting goals.
Re: UK Biobank health data keeps ending up on GitHub
#39> It has given 20,000 researchers around the world access under strict agreements that prohibit sharing data further. To me it seems rather naive to have done that. After all, you can't un-leak medical data. So even if the "strict agreement" included huge punishments, there's no getting the toothpaste back in the tube. If you want to ensure compliance before a leak happens you have to (ugh) audit their compliance. An…
One of the favorite lessons I learned is that anything at scale has to be designed for idiots. I am pretty sure every person reading this has had days where they have done absolutely stupid things without realizing. Now assume there are thousands of users, and you could be providing tools to the smartest people in the world and still have people do stupid stuff all the time. This doesn't just apply to UX.
Then there's the question of trust. You probably have friends you know not to tell certain secrets to, because they believe they get to delegate your secrets onwards to people they trust. The further away someone is from you, the less respect they will show. Researchers have been loaning the dataset in good faith to people who they trust, but who probably didn't take the whole secrecy thing as seriously.
With 20k researchers this was inevitable. The kind of factors above need to be factored in when designing on what grounds such a dataset is to be released.
Re: UK Biobank health data keeps ending up on GitHub
#40> It has given 20,000 researchers around the world access under strict agreements that prohibit sharing data further. To me it seems rather naive to have done that. After all, you can't un-leak medical data. So even if the "strict agreement" included huge punishments, there's no getting the toothpaste back in the tube. If you want to ensure compliance before a leak happens you have to (ugh) audit their compliance. An…
Not giving the data to researchers means not getting the scientific benefits from that data. Which was the point of collecting that data in the first place. Reckless harm prevention is the root of many evils.
As a biostatistician who's touched epidemiological studies, I'd argue losing the trust of participants and the public is one of the biggest threats to the viability of the whole research enterprise. It's reckless to jeopardize that as well. Conversely, this dataset will be mined for at least 30-50 years - there are an infinite number of questions that can be asked of this dat. Given that timescale, I think a little delay here is acceptable.