Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel Programming with Python

chryswoods.com

Post reply on HN
Print view View original

121–130 of 147 posts

Re: Parallel Programming with Python

#121

Post by TickleSteve »

devxpy wrote: post #108
TickleSteve wrote: post #89

Earlier quoted context omitted.

I think you took that claim out of context: "By "perfect MT programs", I mean code that's easy to write and understand, that works with the same design approach in any programming language, and on any operating system, and that scales across any number of CPUs with zero wait states and no point of diminishing returns." That doesn't mean to say its "perfect" or "solves" multithreading, just that its easy to write and…

> That doesn't mean to say its "perfect" or "solves" multithreading, just that its easy to write and understand Try saying that out loud?

yes. That makes perfect sense...

easy to write and understand is something completely different to correctness, robustness, scalability, etc. All those must be considered if you think you have 'solved' parallelism, but they are orthogonal to 'easy to understand'.

Re: Parallel Programming with Python

#122

Post by TickleSteve »

devxpy wrote: post #109
TickleSteve wrote: post #102

Earlier quoted context omitted.

Nothing against zeromq, its good s/w, but like all tools it must be used appropriately. ...also, nanomsg is the 'improved' successor. Also, MPI isn't a 'niche' thing, its the way that a large proportion of high-performance applications have been implemented for a few decades (think Crays & weather prediction). Zeromq has a few simple web-apps using it (I exagerate slightly).

Seems like you know quite a lot about this topic. Do you have any projects of your own that I can see? Or do you only work for your employer or something?

Like the vast majority of software engineers, my work is not open source.

But since you seem like arguments from authority, I've got around 25 years experience in software ranging from hard-real-time embedded defence software to safety-critical train braking systems. I've been software architect on systems selling 10's of millions of products, currently working in the IoT space. I've architected and implemented software on servers, desktops, embedded and mobile platforms.

But no, you aren't likely to find my stuff on GitHub.

Re: Parallel Programming with Python

#123

Post by srean »

lozenge wrote: post #105
srean wrote: post #103

Earlier quoted context omitted.

> Because of copy-on-write fork magic, many multiprocessing configurations (including the default) can "send" that data to child processes in constant* time, if the data's already present in e.g. a global when children are created. Have you tried this or got it working ? The fly in the ointment is the reference count . Add a reference and BOOM you suddenly have a huge copy. It can be made to work efficiently in certa…

You can call gc.freeze that effectively sets all reference counts to infinity.

Yes! thanks, that slipped my mind

Re: Parallel Programming with Python

#124

Post by devxpy »

adw wrote: post #120
devxpy wrote: post #113

Earlier quoted context omitted.

'utterly perfect ' are not my words http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ Thought linking it there would make it better, but I'll just remove it... And you do make a good point. It doesn't really solve anything technically. But would you agree that it exposes a better API for doing much of the same stuff?

I wouldn't know without using it. That's where "software using this library" is a really useful bit of social proof. Think of Django; even without looking at the code you have a lot of evidence that it can conveniently solve a wide range of real problems.

Well HN I'd say is a pretty good place to raise social awareness

Re: Parallel Programming with Python

#125

Post by devxpy »

adw wrote: post #118
devxpy wrote: post #115

Earlier quoted context omitted.

Do you think using this over zmq in my thingy would really bring much improvement? I use it everyday for some of my home baked apps, and would love if this really made a difference.

That depends on your workload. 0MQ is fine software, but they solve different problems. The problem here is the claims you're making. You've written some utility classes around 0MQ for some applications, which is a real thing, so I'd rewrite your GitHub readme to just demonstrate what problems you've solved with it (and at what kind of scale). Making big, sweeping claims gets you into these kinds of threads, because…

Updated readme to be a little better

https://github.com/pycampers/zproc

I use it in a couple of my own stuff which I have open sourced yet.

I do plan to release them, and I hope I can prove the usefulness of the library using those...

professional (corporate) stuff would be a far fetch for me, obviously.

Re: Parallel Programming with Python

#126

Post by devxpy »

TickleSteve wrote: post #122
devxpy wrote: post #109

Earlier quoted context omitted.

Seems like you know quite a lot about this topic. Do you have any projects of your own that I can see? Or do you only work for your employer or something?

Like the vast majority of software engineers, my work is not open source. But since you seem like arguments from authority, I've got around 25 years experience in software ranging from hard-real-time embedded defence software to safety-critical train braking systems. I've been software architect on systems selling 10's of millions of products, currently working in the IoT space. I've architected and implemented softw…

Alright, thanks.

It's the internet, so have to make sure.

Most of my stuff is closed source as well.

Iot you say?

I made a couple of stuff myself, mostly using the micropython stack or the raspberry pi.

What do you generally work on?

Re: Parallel Programming with Python

#127

Post by devxpy »

TickleSteve wrote: post #102
devxpy wrote: post #85

Earlier quoted context omitted.

> I would suggest you don't make dramatic claims If you could point out some stuff from ZProc's page, that would be nice! > mpi is the grandfather of message passing libraries Never heard of it before, but just a simple google search reveals that it _might_ be more performant than zmq, but not as fault-tolerant and flexible. It really looks like a niche thing, from this comment by peter hintjens > Why smart cloud bui…

Nothing against zeromq, its good s/w, but like all tools it must be used appropriately. ...also, nanomsg is the 'improved' successor. Also, MPI isn't a 'niche' thing, its the way that a large proportion of high-performance applications have been implemented for a few decades (think Crays & weather prediction). Zeromq has a few simple web-apps using it (I exagerate slightly).

Is "Crays and weather prediction" really a part of everyday compute?

Re: Parallel Programming with Python

#128

Post by TickleSteve »

devxpy wrote: post #126
TickleSteve wrote: post #122

Earlier quoted context omitted.

Like the vast majority of software engineers, my work is not open source. But since you seem like arguments from authority, I've got around 25 years experience in software ranging from hard-real-time embedded defence software to safety-critical train braking systems. I've been software architect on systems selling 10's of millions of products, currently working in the IoT space. I've architected and implemented softw…

Alright, thanks. It's the internet, so have to make sure. Most of my stuff is closed source as well. Iot you say? I made a couple of stuff myself, mostly using the micropython stack or the raspberry pi. What do you generally work on?

Not MicroPython.

If you're interested in IoT (or embedded s/w in general), get away from MicroPython.

The primary characteristic of most embedded products is to be low-cost. When you're selling millions of products, cost counts. You can't waste cycles or resources on Python.

MicroPython is a toy for the 'makers'. Similarly the JS equivalents. No real high volume product would use those technologies.

Re: Parallel Programming with Python

#129

Post by devxpy »

TickleSteve wrote: post #128
devxpy wrote: post #126

Earlier quoted context omitted.

Alright, thanks. It's the internet, so have to make sure. Most of my stuff is closed source as well. Iot you say? I made a couple of stuff myself, mostly using the micropython stack or the raspberry pi. What do you generally work on?

Not MicroPython. If you're interested in IoT (or embedded s/w in general), get away from MicroPython. The primary characteristic of most embedded products is to be low-cost. When you're selling millions of products, cost counts. You can't waste cycles or resources on Python. MicroPython is a toy for the 'makers'. Similarly the JS equivalents. No real high volume product would use those technologies.

the esp8266 boards are like pretty cheap.

Won't argue about pyboard, because that's more of a gimmick. (Way too expensive for what it does)

I think the cost and time of developing on micropython vs a lower level language like C, would superseded the cost associated with wasted cpu cycles.

However, I agree with the fact that no real product would use mpy right now in production because of the infancy of the project.

It certainly looks promising.

It's definitely NOT a toy.

OTOH JS is just a bad choice for this kind of work IMO. ( I'm a firm believer that JS is just a bad choice for anything in general, but IOT is just madness)

Re: Parallel Programming with Python

#130

Post by TickleSteve »

devxpy wrote: post #129
TickleSteve wrote: post #128

Earlier quoted context omitted.

Not MicroPython. If you're interested in IoT (or embedded s/w in general), get away from MicroPython. The primary characteristic of most embedded products is to be low-cost. When you're selling millions of products, cost counts. You can't waste cycles or resources on Python. MicroPython is a toy for the 'makers'. Similarly the JS equivalents. No real high volume product would use those technologies.

the esp8266 boards are like pretty cheap. Won't argue about pyboard, because that's more of a gimmick. (Way too expensive for what it does) I think the cost and time of developing on micropython vs a lower level language like C, would superseded the cost associated with wasted cpu cycles. However, I agree with the fact that no real product would use mpy right now in production because of the infancy of the project. I…

No, uPython IS a toy.

In embedded s/w, the cost of the h/w outweighs everything else, ok so your 'easily developed' Python app will cost (say) 10K less to develop... but the resource requirements mean you need to go up $0.5 on the processor....

Oops, on your 500'000 devices you've suddenly wasted $250,000. All because you couldn't be bothered to save a few KB of RAM.

Python is a toy for embedded s/w.... dont get me going on catching bugs at runtime in an embedded system because you're using a dynamically typed language! Now you have to upgrade 500,000 device over-the air (cellular data costs) which will take a week, meanwhile your customers a fuming because their data is being lost!

seriously!

Post reply on HN