Ask HN: What would you compute on 2000 badly behaved worker nodes?

#1

Post by pedrombafonso » Mon, Feb 17, 2014, 6:03 PM UTC

I work for CrowdProcess, and we built a distributed computing platform that runs on Web Workers (badly behaved, i.e. volatile connections with some latency), to monetize websites replacing advertising.

We already have +2000 nodes and have been getting pretty good speedups in Monte-Carlo based algorithms. We also have the simplest ever API for a distributed computing platform.

What else would you build using it ?

Re: Ask HN: What would you compute on 2000 badly behaved worker nodes?

#2

Post by patio11 » Mon, Feb 17, 2014, 6:14 PM UTC

At a previous day job we used n-queens to benchmark distributed computing, as it is disgustingly parallelizable and produces nice visual results.

Re: Ask HN: What would you compute on 2000 badly behaved worker nodes?

#3

Post by pedrombafonso » Mon, Feb 17, 2014, 6:18 PM UTC

patio11 wrote: post #2

At a previous day job we used n-queens to benchmark distributed computing, as it is disgustingly parallelizable and produces nice visual results.

Thanks for your answer! Do you know about any real life application for that problem?

Re: Ask HN: What would you compute on 2000 badly behaved worker nodes?

#4

Post by patio11 » Mon, Feb 17, 2014, 6:24 PM UTC

pedrombafonso wrote: post #3

patio11 wrote: post #2

At a previous day job we used n-queens to benchmark distributed computing, as it is disgustingly parallelizable and produces nice visual results.

Thanks for your answer! Do you know about any real life application for that problem?

If you're freezing to death inside of a computer cluster n-queens will save your life.

Re: Ask HN: What would you compute on 2000 badly behaved worker nodes?

#5

Post by tlarkworthy » Mon, Feb 17, 2014, 6:35 PM UTC

CPLEX and Gurobi cost a lot but are used by big companies to solve mixed integer linear programming problems. You can exploit parralelism in the MIP part.

Operations use MIPs a lot. Usually via an excel spreadsheet :s However people expect a good interface to these problems, its not trivial

Re: Ask HN: What would you compute on 2000 badly behaved worker nodes?

#6

Post by Everlag » Mon, Feb 17, 2014, 6:36 PM UTC

coin miners where is any coin using scrypt?

Re: Ask HN: What would you compute on 2000 badly behaved worker nodes?

#7

Post by joaojeronimo » Mon, Feb 17, 2014, 6:43 PM UTC

Everlag wrote: post #6

coin miners where is any coin using scrypt?

that's a pretty good one, especially since mining Litecoins is still profitable on EC2 (at least last time I checked). I was planing to write something like that on the weekend but couldn't, will have to do it pretty soon...

Re: Ask HN: What would you compute on 2000 badly behaved worker nodes?

#8

Post by thaumaturgy » Mon, Feb 17, 2014, 6:57 PM UTC

Hmm. It's not clear from your documentation, is it possible to use xmlhttprequest through the web workers and get the response?

Because having thousands of systems as distributed web crawlers would be really really cool.

Re: Ask HN: What would you compute on 2000 badly behaved worker nodes?

#9

Post by bdcs » Mon, Feb 17, 2014, 7:05 PM UTC

If you can use GPUs then I think scrypt-based coin mining will be the most profitable thing to do. If not, then you need to find problems that are relatively fast on CPUs compared to GPUs, parallelizable, and low bandwidth. It will be a small intersection, but there will likely be something.

Re: Ask HN: What would you compute on 2000 badly behaved worker nodes?

#10

Post by joaojeronimo » Mon, Feb 17, 2014, 7:10 PM UTC

thaumaturgy wrote: post #8

Hmm. It's not clear from your documentation, is it possible to use xmlhttprequest through the web workers and get the response? Because having thousands of systems as distributed web crawlers would be really really cool.

It's not possible, sorry. It's possible if you ask us to access a certain address, but to the outside world it's not allowed. It would be pretty cool to have distributed web crawlers but it would also be extremely dangerous if someone decided to use CrowdProcess to do a DDoS.