Parallel C++ on AWS Lambda for CRISPR
benchling.engineering
Parallel C++ on AWS Lambda for CRISPR
1–10 of 18 posts
Re: Parallel C++ on AWS Lambda for CRISPR
#2Lambda is an exciting platform, but it does require some bending to get it to work for certain use cases.
This reminds me of some problems we solved last year [0], which should not have been problems for a fairly straightforward application.
[0]: https://iplayer.engineering/evaluating-tensorflow-models-in-...
Re: Parallel C++ on AWS Lambda for CRISPR
#3Lambda is an exciting platform, but it does require some bending to get it to work for certain use cases. This reminds me of some problems we solved last year [0], which should not have been problems for a fairly straightforward application. [0]: https://iplayer.engineering/evaluating-tensorflow-models-in-...
I read that article before and just reread it now - thanks for writing it up, but I still have the same question I did the first time around. You posit:
> Because we run 200 invocations or so in parallel we’ll only need to download the model once and save it there
From my own reading of the Lambda docs, it seems that a simultaneous request for the same Lambda may or may not spin up a new container, ie while serial requests within the 15 minute timeout will likely reuse the same instance with the same frozen/cached tmp data, parallel requests do not have that assumption.
Was this your finding in practice? If so, were your CloudWatch “keep warm” events a series of just 1 Lambda invocation ~10/15min apart of 200 simultaneous requests serially spaced to keep the instances spun up?
Re: Parallel C++ on AWS Lambda for CRISPR
#4We have a similar application (parallelized C++ code operating on large files, for bioinformatics even) and ended up down the reverse path: started on Lambda, moved to our own RPC system. Lambda got super expensive (in part because there was no way to reuse a worker while it was doing async work like an S3 download), couldn't parallelize nearly enough without hitting AWS hard-limit quotas, had significantly lower CPU perf and didn't provide a way to cache files (like a genome in this case). Spot/preemptible instances keep our costs down while letting us keep a few hundred servers up at a time.
Re: Parallel C++ on AWS Lambda for CRISPR
#5Lambda is an exciting platform, but it does require some bending to get it to work for certain use cases. This reminds me of some problems we solved last year [0], which should not have been problems for a fairly straightforward application. [0]: https://iplayer.engineering/evaluating-tensorflow-models-in-...
I read that article before and just reread it now - thanks for writing it up, but I still have the same question I did the first time around. You posit: > Because we run 200 invocations or so in parallel we’ll only need to download the model once and save it there From my own reading of the Lambda docs, it seems that a simultaneous request for the same Lambda may or may not spin up a new container, ie while serial re…
> From my own reading of the Lambda docs, it seems that a simultaneous request for the same Lambda may or may not spin up a new container, ie while serial requests within the 15 minute timeout will likely reuse the same instance with the same frozen/cached tmp data, parallel requests do not have that assumption.
This is true. I learned this a hard way some time ago while trying to understand why our lambda function crashes from time to time. It turned out we didn't clean the /tmp folder and we used the space quite extensively. There's a good article on this:
https://aws.amazon.com/blogs/compute/container-reuse-in-lamb...
Re: Parallel C++ on AWS Lambda for CRISPR
#6What's the difference between this CRISPR search problem and DNA sequence alignment? There were extensive development in the latter and is highly optimized. The author seems to be coming up with solution from scratch.
Re: Parallel C++ on AWS Lambda for CRISPR
#7Articles like this are why I return to HN everyday. There are 3 terms in the title that I know well, but put them together and they are something completely new to me. And the cost savings here is pretty remarkable.
Re: Parallel C++ on AWS Lambda for CRISPR
#8Lambda is an exciting platform, but it does require some bending to get it to work for certain use cases. This reminds me of some problems we solved last year [0], which should not have been problems for a fairly straightforward application. [0]: https://iplayer.engineering/evaluating-tensorflow-models-in-...
I read that article before and just reread it now - thanks for writing it up, but I still have the same question I did the first time around. You posit: > Because we run 200 invocations or so in parallel we’ll only need to download the model once and save it there From my own reading of the Lambda docs, it seems that a simultaneous request for the same Lambda may or may not spin up a new container, ie while serial re…
The CloudWatch "keep warm" events are just one invocation, yes.
I do not remember having had issues with it, but honestly, I don't think I actually have stats on that anymore.
I've just checked in S3, but it doesn't look like we have request or data transfer metrics enabled on the model bucket. I may enable those next week to monitor the effectiveness of our strategy better.
Re: Parallel C++ on AWS Lambda for CRISPR
#9What's the difference between this CRISPR search problem and DNA sequence alignment? There were extensive development in the latter and is highly optimized. The author seems to be coming up with solution from scratch. https://en.wikipedia.org/wiki/Sequence_alignment
Original author here.
You're right - conceptually the CRISPR search problem and DNA sequence alignment are related. In both, you're looking for place where two (or more) sequences are very similar. I would say there are two major differences.
The first is in the goal of the search. Typically, alignment tools try to find the best positional alignment for two (or more) sequences. The CRISPR search problem is to find every possible match above some similarity threshold.
There are also a few constraints on the CRISPR search problem that allow us to make this much faster than a general DNA sequence alignment tool:
1) We know that that guide sizes tend to be very small (~20bp) 2) Part of the guide must match exactly (the PAM site), allowing us to restrict our search even further. 3) We don't need to worry about insertions or deletions in our search.
Using those three constraints, we can do this search a lot faster than a more general DNA alignment tool!
Re: Parallel C++ on AWS Lambda for CRISPR
#10What's the difference between this CRISPR search problem and DNA sequence alignment? There were extensive development in the latter and is highly optimized. The author seems to be coming up with solution from scratch. https://en.wikipedia.org/wiki/Sequence_alignment
(I work at Benchling and worked on the protein alignments tool.)
Adding to Vineet's response, Benchling has both CRISPR search and sequence alignment, and they have different use cases. CRISPR search uses the custom algorithm described in the article (and the "last post" linked at the top), while alignments use the two popular alignment tools Clustal Omega[1] and MAFFT[2]. It's certainly true that the general alignment problem is complex had has undergone plenty of research around performance and producing good results, which is why we used an off-the-shelf tool for computing the alignment itself (but a custom UI for viewing the result).