Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speed up your Fabric deployments with Parex

paulbohm.com

Post reply on HN
Print view View original

1–9 of 9 posts

Re: Speed up your Fabric deployments with Parex

#3

Post by enki »

njharman wrote: post #2

Why isn't Fabric doing this? This would be basic feature of deployment software I would think. I vaguely remember the old, old Fabric (before they rewrote everything) did this.

i think they've been planning to for a while: http://code.fabfile.org/issues/show/19

i haven't really followed their progress too closely, but they seem to be close to releasing something for 1.0.

I think they do client-side threading while i prefer to to do fork() on the server to avoid multiple ssh connections and retain maximum control.

i'll reevaluate when they ship. till then parex works for me.

Re: Speed up your Fabric deployments with Parex

#4

Post by jacobian »

njharman wrote: post #2

Why isn't Fabric doing this? This would be basic feature of deployment software I would think. I vaguely remember the old, old Fabric (before they rewrote everything) did this.

There's a branch with this feature: http://github.com/goosemo/fabric. The current work's in the "mutliprocessing" branch. I'm using it for a big deploy right now, and it appears to be working well.

[edit: fixed my mistake about where the most current code lives]

Re: Speed up your Fabric deployments with Parex

#5

Post by goosemo »

enki wrote: post #3
njharman wrote: post #2

Why isn't Fabric doing this? This would be basic feature of deployment software I would think. I vaguely remember the old, old Fabric (before they rewrote everything) did this.

i think they've been planning to for a while: http://code.fabfile.org/issues/show/19 i haven't really followed their progress too closely, but they seem to be close to releasing something for 1.0. I think they do client-side threading while i prefer to to do fork() on the server to avoid multiple ssh connections and retain maximum control. i'll reevaluate when they ship. till then parex works for me.

The branch actually does client side forks, one for each server in a task. Then each call inside a task is run sequentially, as expected in normal fabric usage. I've also included the ability to set a pool size, so that one can manage the number of ssh connections open simultaneously.

Re: Speed up your Fabric deployments with Parex

#7

Post by enki »

goosemo wrote: post #5
enki wrote: post #3

Earlier quoted context omitted.

i think they've been planning to for a while: http://code.fabfile.org/issues/show/19 i haven't really followed their progress too closely, but they seem to be close to releasing something for 1.0. I think they do client-side threading while i prefer to to do fork() on the server to avoid multiple ssh connections and retain maximum control. i'll reevaluate when they ship. till then parex works for me.

The branch actually does client side forks, one for each server in a task. Then each call inside a task is run sequentially, as expected in normal fabric usage. I've also included the ability to set a pool size, so that one can manage the number of ssh connections open simultaneously.

ah - so what you are doing speeds up deploys to multiple servers, while what i'm doing is meant to speed up an individual deploy.

i mostly just got tired of waiting for an individual deploy to my staging server.

btw thanks for emailing me! - i'll definitely look at your branch and see if i can help! (wasn't aware of it until today)

Re: Speed up your Fabric deployments with Parex

#8

Post by goosemo »

enki wrote: post #7
goosemo wrote: post #5

Earlier quoted context omitted.

The branch actually does client side forks, one for each server in a task. Then each call inside a task is run sequentially, as expected in normal fabric usage. I've also included the ability to set a pool size, so that one can manage the number of ssh connections open simultaneously.

ah - so what you are doing speeds up deploys to multiple servers, while what i'm doing is meant to speed up an individual deploy. i mostly just got tired of waiting for an individual deploy to my staging server. btw thanks for emailing me! - i'll definitely look at your branch and see if i can help! (wasn't aware of it until today)

Yeah, thats the goal anyways. I tend to reach out to 50+ servers at a time, and iterating though that on any tasks that take over a min is mind numbing.

Well through this post perhaps the branch will be better known, but I haven't really posted much about it in that I though it'd be done sooner. Then any mention of a branch would be negated by it's inclusion in master.

Post reply on HN