Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What if Python was natively distributable?

medium.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: What if Python was natively distributable?

#21

Post by westurner »

westurner wrote: post #7

cloudpickle serializes code without signatures; which is an RCE vuln. It is much safer to distribute signed code in signed packages out of band and send only non-executable data in messages. It is more safe to store distributed messages in a page with the NX bit flipped. A compromise of any client in this system results in DOS and arbitrary RCE; but that's an issue with most distributed task worker systems. To take a…

Basically, there's no good way to sidestep the authentication and authorization and resource quota controls of a resource grid scheduler.

With redundancy and idempotency, distributed computation can work.

In order to run computation distributedly with required keys, sharded distributed ledger protocol nodes cost the smart contracts that they execute in a virtual machine with network access limited to in-protocol messages. Each "smart contract" costs money to run redundantly on multiple nodes, and so it should or must have an account with a verifiably-sufficient balance in order to run.

Smart contracts must be uploaded using a private key with a corresponding public key.

Smart contracts are identified by and so are addressed by a hash of their bytecode.

Wool, Dask, and Celery @task don't solve for smart contract output storage costs with redundancy. They don't set up database(s) with replication large enough for each computation step for you. Dask and Celery model and track the state of each executed DAG centrally with logs as trustable as the centralized nodes which are a single point of failure.

Why isn't Docker Swarm - which L2 bridges amongst all nodes without restriction - appropriate for a given application, given the Access Control Lists and cloud (pod,) configuration necessary for e.g. AWS or GCP to prevent budget overflow? Why quota grid users somehow?

Serverless functions must be uploaded/deployed before being run, too. To orchestrate a bunch of web services is to execute the DAG and handle errors due to network, node, and service failures and latency

But then what protocol do the (serverless function) services all implement, so that we don't have to have a hodgepodge of API clients to use all the services in the grid?

With (serverless) functions bound to /URL routes, cost each function and estimate resource requirements to continue to run a function of that cost. To handle just benign resource exhaustion, say, Scale up the databases and/or redundant block storages, change other distributed computation grid parameters for that pod or those pods of resources which service a named, signed function like /api/v1/helloworld_v2?q=select%20* which is creating contention for the costed resources of the organization

On what signals do you scale up or scale down - within a resource budget - to afford fanning out over multiple actually parallel nodes to compute and sign and store the data?

Re: What if Python was natively distributable?

#22

Post by schmichael »

Absolutely wild to see none of the long lineage of similar attempts mentioned here. The earliest I could find with a quick search was Pyro which started in 1998 and still seems to be going: https://pyro4.readthedocs.io/en/stable/

RPyC came along in the aughts. There's a long history of "transparent clustering and rpc" efforts in Python that could be used or drawn on.

Sad to see that history ignored here.

Re: What if Python was natively distributable?

#23

Post by sporkl »

MomsAVoxell wrote: post #4

I ask this question, but instead: I ask it of Lua. As in, what if there was a Linux distro that focused, primarily, on building a Lua layer on top of everything, system-wise. Just replace all the standard stuff with one single, system-friendly language: Lua. C/C++ everything as it currently is: put Lua on top all the way to the Desktop. It’s only a thought experiment, except there are cases where I can see a way to u…

Sounds kind of like Arcan https://arcan-fe.com/about/

Re: What if Python was natively distributable?

#24

Post by whatevaa »

interiorchurch wrote: post #20
MomsAVoxell wrote: post #4

I ask this question, but instead: I ask it of Lua. As in, what if there was a Linux distro that focused, primarily, on building a Lua layer on top of everything, system-wise. Just replace all the standard stuff with one single, system-friendly language: Lua. C/C++ everything as it currently is: put Lua on top all the way to the Desktop. It’s only a thought experiment, except there are cases where I can see a way to u…

There used to be a roguelike game called Angband, which was written in C. There was a vibrant community around it, many of whom produced Angband variants by hacking the text config files and the C code. One developer got the idea of making most of the game scriptable in Lua, over a C core; which would, in theory, make even more people be able to hack at the game and produce variants. What happened was the Angband com…

Arrays starts at one scared everybody away.

Re: What if Python was natively distributable?

#26

Post by eesmith »

interiorchurch wrote: post #20
MomsAVoxell wrote: post #4

I ask this question, but instead: I ask it of Lua. As in, what if there was a Linux distro that focused, primarily, on building a Lua layer on top of everything, system-wise. Just replace all the standard stuff with one single, system-friendly language: Lua. C/C++ everything as it currently is: put Lua on top all the way to the Desktop. It’s only a thought experiment, except there are cases where I can see a way to u…

There used to be a roguelike game called Angband, which was written in C. There was a vibrant community around it, many of whom produced Angband variants by hacking the text config files and the C code. One developer got the idea of making most of the game scriptable in Lua, over a C core; which would, in theory, make even more people be able to hack at the game and produce variants. What happened was the Angband com…

When I was a wet behind the ears programmer, I learned a cautionary tale about Lisp and macros. Very smart people love macros and are good at them, and will modify things to fit their needs. Problem is, other people think different ways. A flexible language which is easy to mutate results in incompatible dialects.

The sage elders suggested that making the language harder to change (cough Python) is more likely to result in a single widely used dialect, with the differences at the library level rather than in language/macro level.

Re: What if Python was natively distributable?

#27

Post by bzurak »

ericreg92 wrote: post #3

Whats curious to me here is that the github repo linked to in the article seems to exactly recreate the "operating system" that is called an "opinion in the wrong layer". Perhaps I am misunderstanding this, but after looking at the code what exactly are we achieving here over other frameworks? The repo is obviously very new (and the author certainly seems busy), so perhaps a better question is what do we aim to achie…

For one, it's faster than prefect and ray so I can use it for more stuff without having to import a 10000000 line library.

Re: What if Python was natively distributable?

#28

Post by bzurak »

schmichael wrote: post #22

Absolutely wild to see none of the long lineage of similar attempts mentioned here. The earliest I could find with a quick search was Pyro which started in 1998 and still seems to be going: https://pyro4.readthedocs.io/en/stable/ RPyC came along in the aughts. There's a long history of "transparent clustering and rpc" efforts in Python that could be used or drawn on. Sad to see that history ignored here.

I guess I'm not old enough.

Re: What if Python was natively distributable?

#29

Post by bzurak »

westurner wrote: post #7

cloudpickle serializes code without signatures; which is an RCE vuln. It is much safer to distribute signed code in signed packages out of band and send only non-executable data in messages. It is more safe to store distributed messages in a page with the NX bit flipped. A compromise of any client in this system results in DOS and arbitrary RCE; but that's an issue with most distributed task worker systems. To take a…

I'm running this in a trusted environment, I'm not so ambitious as to try to make this some sort of whacky trustless distributed Python runtime. Just a fun project that's been marinating for a while, and now I have an army of clankers to do the dirty work of documenting and testing it.

Re: What if Python was natively distributable?

#30

Post by bzurak »

jjtheblunt wrote: post #18
extr wrote: post #8

LLM generated article.

I wonder if an LLM generated article would get the title to use proper English, though: "What if Python were natively distributable?". It's possible LLMs pick up improper English, of course, since proper is some measure of what used to be a norm, but may presently be perceived as outdated.

Is it possible it's both?
Post reply on HN