Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel C++ on AWS Lambda for CRISPR

benchling.engineering

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Parallel C++ on AWS Lambda for CRISPR

#11

Post by cobookman »

zbjornson wrote: post #4

We have a similar application (parallelized C++ code operating on large files, for bioinformatics even) and ended up down the reverse path: started on Lambda, moved to our own RPC system. Lambda got super expensive (in part because there was no way to reuse a worker while it was doing async work like an S3 download), couldn't parallelize nearly enough without hitting AWS hard-limit quotas, had significantly lower CPU…

I feel like serverless is only benefitial for low volume / low traffic workloads.

The only other perk I've found is that the serverless billing model makes it easier to estimate costs.

Re: Parallel C++ on AWS Lambda for CRISPR

#12

Post by reconbot »

cobookman wrote: post #11
zbjornson wrote: post #4

We have a similar application (parallelized C++ code operating on large files, for bioinformatics even) and ended up down the reverse path: started on Lambda, moved to our own RPC system. Lambda got super expensive (in part because there was no way to reuse a worker while it was doing async work like an S3 download), couldn't parallelize nearly enough without hitting AWS hard-limit quotas, had significantly lower CPU…

I feel like serverless is only benefitial for low volume / low traffic workloads. The only other perk I've found is that the serverless billing model makes it easier to estimate costs.

I found it lowered our server cost incredibly for a high volume read heavy site. It allowed us to scale in response to increased traffic (not instantly that's a lie, spikes in the thousands of requests a second are not handled well, but over a few minutes it catches up without issue) and not have to provision and spin up servers. We were constantly over provisioned before and now it's a much lower but moving margin.

Re: Parallel C++ on AWS Lambda for CRISPR

#14

Post by ComputerGuru »

niklasrde wrote: post #8
ComputerGuru wrote: post #3

Earlier quoted context omitted.

I read that article before and just reread it now - thanks for writing it up, but I still have the same question I did the first time around. You posit: > Because we run 200 invocations or so in parallel we’ll only need to download the model once and save it there From my own reading of the Lambda docs, it seems that a simultaneous request for the same Lambda may or may not spin up a new container, ie while serial re…

The CloudWatch "keep warm" events are just one invocation, yes. I do not remember having had issues with it, but honestly, I don't think I actually have stats on that anymore. I've just checked in S3, but it doesn't look like we have request or data transfer metrics enabled on the model bucket. I may enable those next week to monitor the effectiveness of our strategy better.

I imagine if the code were properly written to deal with race conditions you wouldn’t notice any issues either way besides an increased latency for some requests.

Good luck!

Re: Parallel C++ on AWS Lambda for CRISPR

#15

Post by inciampati »

vineetg wrote: post #9
andrewon wrote: post #6

What's the difference between this CRISPR search problem and DNA sequence alignment? There were extensive development in the latter and is highly optimized. The author seems to be coming up with solution from scratch. https://en.wikipedia.org/wiki/Sequence_alignment

Original author here. You're right - conceptually the CRISPR search problem and DNA sequence alignment are related. In both, you're looking for place where two (or more) sequences are very similar. I would say there are two major differences. The first is in the goal of the search. Typically, alignment tools try to find the best positional alignment for two (or more) sequences. The CRISPR search problem is to find ev…

Honestly you are taking a big risk designing a DNA search algorithm something from scratch. It's akin to the risk people take when they roll their own crypto. There are aspects of this that you may not be considering, and it tends to be best to rest on the extensive work in the field than assume it is a trivial problem.

How do you deal with natural variation in the genome? Can you be sure your gRNA doesn't target an essential locus in some percentage people who carry a particular allele? The data to solve this is out there (1000 Genomes for instance).

Edit: excuse me, I appreciate that you are using a collection of whole genomes as your target. Will this reliably scale to thousands or millions of genomes and likely recombinations between them?

Re: Parallel C++ on AWS Lambda for CRISPR

#16

Post by friendly_chap »

cobookman wrote: post #11
zbjornson wrote: post #4

We have a similar application (parallelized C++ code operating on large files, for bioinformatics even) and ended up down the reverse path: started on Lambda, moved to our own RPC system. Lambda got super expensive (in part because there was no way to reuse a worker while it was doing async work like an S3 download), couldn't parallelize nearly enough without hitting AWS hard-limit quotas, had significantly lower CPU…

I feel like serverless is only benefitial for low volume / low traffic workloads. The only other perk I've found is that the serverless billing model makes it easier to estimate costs.

Well, yes. This is why we are building a solution that you can install anywhere https://github.com/1backend/1backend

You might jokingly say we are reintroducing the servers into the serverless concept :D

But I think we are just giving freedom back to the users.

Re: Parallel C++ on AWS Lambda for CRISPR

#18

Post by cobookman »

reconbot wrote: post #12
cobookman wrote: post #11

Earlier quoted context omitted.

I feel like serverless is only benefitial for low volume / low traffic workloads. The only other perk I've found is that the serverless billing model makes it easier to estimate costs.

I found it lowered our server cost incredibly for a high volume read heavy site. It allowed us to scale in response to increased traffic (not instantly that's a lie, spikes in the thousands of requests a second are not handled well, but over a few minutes it catches up without issue) and not have to provision and spin up servers. We were constantly over provisioned before and now it's a much lower but moving margin.

You can do that today though with K8s and auto-scaling. Not sure how well autoscaling works on AWS, but on GCP its a breeze.
Post reply on HN