Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyPI Proxying for Docker Builds

robopenguins.com

Post reply on HN
Print view View original

1–7 of 7 posts

Re: PyPI Proxying for Docker Builds

#3

Post by riedel »

In the old world a simple global http proxy would suffice to catch all such traffic, but I guess pip is also protecting my package installation behaviour against eavesdropping of foreign intelligence.

Re: PyPI Proxying for Docker Builds

#4

Post by move-on-by »

> The main advantage of 1. is that it’s a bit simpler. My main concern with it is that multiple builds could occur in parallel that might cause issues. We had seen instance when running multiple instances of pip simultaneously caused the cache to become corrupted.

I believe that is what the ‘sharing’ option for cache mounts solve? Per docker docs[0]:

> One of ‘shared’, ‘private’, or ‘locked’. Defaults to ‘shared’. A shared cache mount can be used concurrently by multiple writers. private creates a new mount if there are multiple writers. locked pauses the second writer until the first one releases the mount.

So, for pip, can’t you just do:

    RUN --mount=type=cache,target=/root/.cache/pip,sharing=locked \
    pip install -r requirements.txt
Although I’m not sure how that works with different python versions… Does docker link the cache to the specific FROM image? Probably not. I suggest using the 'id' argument with the python version, maybe:

    RUN --mount=type=cache,target=/root/.cache/pip,sharing=locked,id=pip-cache-py3-12 \
    pip install -r requirements.txt
Although, the proxy solution is certainly a valid alternative.

[0]: https://docs.docker.com/reference/dockerfile/#run---mounttyp...

Re: PyPI Proxying for Docker Builds

#5

Post by martinky24 »

move-on-by wrote: post #4

> The main advantage of 1. is that it’s a bit simpler. My main concern with it is that multiple builds could occur in parallel that might cause issues. We had seen instance when running multiple instances of pip simultaneously caused the cache to become corrupted. I believe that is what the ‘sharing’ option for cache mounts solve? Per docker docs[0]: > One of ‘shared’, ‘private’, or ‘locked’. Defaults to ‘shared’. A…

Had no idea this feature existed, thanks for sharing!

Re: PyPI Proxying for Docker Builds

#6

Post by move-on-by »

martinky24 wrote: post #5
move-on-by wrote: post #4

> The main advantage of 1. is that it’s a bit simpler. My main concern with it is that multiple builds could occur in parallel that might cause issues. We had seen instance when running multiple instances of pip simultaneously caused the cache to become corrupted. I believe that is what the ‘sharing’ option for cache mounts solve? Per docker docs[0]: > One of ‘shared’, ‘private’, or ‘locked’. Defaults to ‘shared’. A…

Had no idea this feature existed, thanks for sharing!

I only discovered it a couple weeks ago, I’m glad I can help spread the word.

Re: PyPI Proxying for Docker Builds

#7

Post by 3np »

riedel wrote: post #3

In the old world a simple global http proxy would suffice to catch all such traffic, but I guess pip is also protecting my package installation behaviour against eavesdropping of foreign intelligence.

No you can totally still do this. May want to set the requests CA if you're using self-provisioned cert.
Post reply on HN