Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What can I do to accelerate scientific research?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 179 posts

Re: Ask HN: What can I do to accelerate scientific research?

#61

Post by massung »

ims wrote: post #59
massung wrote: post #51

> What ideas do you have on how one can accelerate scientific research? I work in genetics (as a software engineer). If there was a major flaw in current scientific research (that involves software), it's that most labs care more about getting published than they do about the reproduce-ability and validation of their work. This means most of the software written in research is ad-hoc, write once, and often never look…

Is anyone using Athena on AWS for tasks like this? It goes without saying that the documentation is hit or miss, but SQL-ish queries of flat files on S3 (even gzipped) can be a nice way to get the same result without managing Spark instances.

Obviously, I can't speak for "anyone", but all the groups I work with do not. There are a myriad of reasons for this, and reasonable people could effectively argue to the merits of those reasons.

It's also important to remember that some of the reasons AWS, Google, and other cloud services are often NOT used are legal in nature. For example, some EU laws prohibit any personally identifiable (genetic) data from studies being put in the cloud. So, even if summary statistics - or data with PII data removed - can be put in the cloud, work has to be done on that data to remove it.

Re: Ask HN: What can I do to accelerate scientific research?

#62

Post by mariushn »

Glench wrote: post #55

I would go talk to scientists and try to understand deeply what they’re working on, how they do it, and what systems (social/technical/political) allow them to do it. I’ve done this a few times and it’s always illuminating and inspiring.

Correct. How do you connect with scientists when not working in an university like you do? Getting an email from a nobody offering help for free does sound like a scam.

One answer would be "Enroll into an university". Others?

Re: Ask HN: What can I do to accelerate scientific research?

#63

Post by massung »

mariushn wrote: post #57
massung wrote: post #51

> What ideas do you have on how one can accelerate scientific research? I work in genetics (as a software engineer). If there was a major flaw in current scientific research (that involves software), it's that most labs care more about getting published than they do about the reproduce-ability and validation of their work. This means most of the software written in research is ad-hoc, write once, and often never look…

Thanks so much, mussung, for your excellent practical feedback! Both alternatives that you listed are very tempting. May I please ask you some followup questions? There's no email in your profile. My email is marius.andreiana@gmail.com > A command line tool that allowed someone to run SQL (or SQL-like) commands on a gzipped CSV file FAST would be invaluable. And if it could JOIN across CSV files ... wow! What prevent…

mariushn, I've sent you an email.

> What prevents one importing each CSV in a postgres db as tables, creating indexes and then start running queries?

There are many reasons:

* Experience/knowledge. Many labs don't have anyone experienced with databases.

* Security. Without proper dev ops a local DB is often out of the question. And when dealing with PII genetic data, [cloud] security can be a major concern.

* Funding. Machines cost money. AWS RDS instance cost money. Maintaining them costs money. Dev ops costs money. etc.

* Often times the queries being done are simple. For example, you may have a giant CSV with cross-ethnic trait data, but only need samples of African decent with a beta value > 0.1. Sure, you could spin up a database, load the entire thing into a table (O(N) + disk space + time), then index it (now it's O(2N) + more disk space + more time), and then finally run your query. Or you can just O(N) run over the CSV once and output the results with no extra disk space or "wasted" (perception of the researcher) time.

Finally, don't fool yourself about the capabilities of researchers. Many are code-savvy, but lack experience. Writing a SQL query is easy. Loading multiple TB of data into a relational database, indexed properly, and done in a manner that won't take days of time is a level higher.

Re: Ask HN: What can I do to accelerate scientific research?

#64

Post by ken »

biophysboy wrote: post #37

This is a pipe dream of mine, but I would love a Wikipedia of null results: nullpedia! Nobody publishes null results bc they’re not exciting, but I think a lot of NSF money would be saved if “failed” experiments were aggregated somewhere in a searchable way. There’s lot of questions. How to organize it? How to encourage participation? How to maximize usefulness while at the same time minimizing volunteer effort? How…

Snopes and Mythbusters showed how you can actually make entertainment out of "This thing we thought might be true, actually isn't true / doesn't work at all".

As for the fairness issue, my first thought is to have it moderated by scientists but from different fields. It's like we do in other areas, like law. You need an expert on the general process, but if they have any personal connection to the topic at hand, they must recuse themselves.

In a sense, it's just a specific type of scientific journal, right? An online journal of only null results.

Re: Ask HN: What can I do to accelerate scientific research?

#65

Post by ken »

6d6b73 wrote: post #40
maxaf wrote: post #36

Forget science: as an “army of one” you’re unlikely to make a contribution of sufficient magnitude that some needle might be moved in the right direction. Take advantage of the skills you do have: make money by doing what startup entrepreneurs do, then donate all proceeds to scientific research. There may be the next Zuckerberg hiding in you. Imagine what would be possible if you could reach that level of wealth and…

A lot, if not most of the important scientific discoveries were result of work of "an army of one". From biosciences to physics you can do amazing stuff. We don't need any new zuckebergs that siphone all of the great minds that are out there into showing more ads.

The individual scientific discovery was common in the early history of science, but has grown less common over time. Looking at this list [1], for example:

- Prior to roughly WWII, it was mostly individuals, and so we named things after them (Kelvin, Doppler, Joule, Ohm, etc).

- From WWII until the late 20th century, it was a lot of small teams (transistor, 3; DNA, 4; pulsars, 2; etc).

- Since then, team sizes have grown so that individuals aren't even named (cloning, an Institute; Top Quark, a Lab; Tau neutrino, a Collaboration; etc).

In fact, in the past 35 years, the only individually named contributors on that list were mathematicians who constructed proofs of long-standing unsolved problems in mathematics.

[1] https://en.wikipedia.org/wiki/Timeline_of_scientific_discove...

Re: Ask HN: What can I do to accelerate scientific research?

#66

Post by Glench »

mariushn wrote: post #62
Glench wrote: post #55

I would go talk to scientists and try to understand deeply what they’re working on, how they do it, and what systems (social/technical/political) allow them to do it. I’ve done this a few times and it’s always illuminating and inspiring.

Correct. How do you connect with scientists when not working in an university like you do? Getting an email from a nobody offering help for free does sound like a scam. One answer would be "Enroll into an university". Others?

I just emailed them and told them I was doing research. Lots of people, especially students, love talking about their work.

Re: Ask HN: What can I do to accelerate scientific research?

#67

Post by amirathi »

Make programming accessible to scientists.

Today, running any compute/memory intensive experiment requires working through so many tools and terminologies (e.g. AWS, SSH, python, dependencies, virtualenvs). Scientists really need just Jupyter notebook like experience that's collaborative, reproducible, and powerful (i.e. run on any hardware with 1 click).

Re: Ask HN: What can I do to accelerate scientific research?

#68

Post by biophysboy »

mariushn wrote: post #52
biophysboy wrote: post #37

This is a pipe dream of mine, but I would love a Wikipedia of null results: nullpedia! Nobody publishes null results bc they’re not exciting, but I think a lot of NSF money would be saved if “failed” experiments were aggregated somewhere in a searchable way. There’s lot of questions. How to organize it? How to encourage participation? How to maximize usefulness while at the same time minimizing volunteer effort? How…

Good idea, and great challenges indeed. Taking this further, I'd call it experimenpedia, with experiments published as they are thought out. Be open for comments/review and showing related experiments before the actual experiment being done. This might prevent potential failures and let owners tweak the planned experiment before being actually done. Then do the experiment and publish the results, whatever they are. I…

eLife is sorta like this. The experiment isn’t live, but the publishing process is.

https://elifesciences.org/about/peer-review

It’s pretty interesting

Re: Ask HN: What can I do to accelerate scientific research?

#69

Post by badpun »

amirathi wrote: post #67

Make programming accessible to scientists. Today, running any compute/memory intensive experiment requires working through so many tools and terminologies (e.g. AWS, SSH, python, dependencies, virtualenvs). Scientists really need just Jupyter notebook like experience that's collaborative, reproducible, and powerful (i.e. run on any hardware with 1 click).

Check out https://www.dominodatalab.com/ - you've pretty much described their product.

Re: Ask HN: What can I do to accelerate scientific research?

#70

Post by badpun »

TallGuyShort wrote: post #50
keldaris wrote: post #29

Earlier quoted context omitted.

As a computational physicist (meaning I do science, but most of my time is spent programming), I agree with everything the parent said, but perhaps I can add some more specifics. The matplotlib example is a very good one in the sense that it's a piece of vital infrastructure almost everyone has used at some point. It works well enough for performance insensitive (meaning non-realtime and small-ish datasets) 2D visual…

>> Other scientists, depending on their interests, will readily give you similar examples of obvious general purpose libraries that are lacking or non-existent I'd love to hear some of these, if folks on the thread can share more. Added 3d visualization to my list...

Symbolic computation libraries for Python are lacking - there's SymPy, but it can throw a "NotImplementedException" at you if you try processing some more hairy formulas... Would be great if SymPy was improved.
Post reply on HN