Live data from Hacker News

Signing the OPAM Repository: TUF Meets Git

opam.ocaml.org

11–20 of 21 posts

Re: Signing the OPAM Repository: TUF Meets Git

#11
What I do not get about these repositories is, why they have no namespacing or a built in democratic mechanism to transfer ownership. Because this does not exist, two related problems occur: Outdated packages squatting their own name and otherwise worse packages squatting a generic name.

The adoption problems Python 3 still has at this point, could be mitigated entirely if package owners could lose the name for outdated packages and it would be transferred to a Python 3 fork. npm has this problem on an even larger scale. The Python community can at least slowly move away from PIL and use Pillow instead, because the libraries are reasonably big. This is much harder to do with all of npms tiny packages. I doubt that the best solution always wins, because the best name is just an enormous advantage.

If names would be democratically assigned, we would get some kind of extended standard library. Newbies that want the best library for their problem that works with the current version of the language would not have to search for Beautifulsoup, voluptuous or requests, but could just use xml, validation and http.

Of course, this is a hard problem and you have to be careful. We do not want Java, we do not want broken dependencies. So my ideas a solution could be based on are these (example is for pip, but applies to others as well):

  # No fork exists
  pip install flask
  
  >Downloading/unpacking mitshuko.flask
  >…

  # A fork exists
  pip install flask

  >Multiple users have created a package named "flask". Please specify the owner
  >of the package or use "--most-popular" to install the one with the most 
  >installations in the last 3 months.
  >
  >*****************************************************************************
  >Attention: pip does not check if the package actually does the same thing, be
  >sure to check that before you switch to a package from another author.
  >*****************************************************************************
  >
  >mitsuhiko.flask (10000 installations)
  >forker.flask (3 installations)

  pip install --most-popular flask
  
  >Downloading/unpacking mitsuhiko.flask
  >…

  pip install -U flask

  # On upgrade, this message is only shown, when the used package is 
  # already at the current version, has not been updated for a long time
  # and a more popular option exists.

  >You have installed the current version of mitsuhiko.flask, however
  >mitsuhiko.flask has not been updated since 7 months and there is a more
  >popular package available of the same name from another author.
  >
  >*****************************************************************************
  >Attention: pip does not check if the package actually does the same thing, be
  >sure to check that before you switch to a package from another author.
  >*****************************************************************************
  >
  >forker.flask (5000 installations)
  >mitsuhiko.flask (4000 installations)

  pip install -U forker.flask

  >Downloading/unpacking mitshuko.flask
  >…

Re: Signing the OPAM Repository: TUF Meets Git

#12
post #7
post #3

(Edit: I think it is great that we're working on signing software. I think that is very important, so important that I want to see it done right. So I have questions, in particular about TUF vs the more knee-jerk alternatives. But I would like to see signing, whether through TUF or the more well-known options.) It seems that in order to add a new package or developer, a quorum of administrators must go to their secur…

You should briefly think how opam does updates currently: a repository on GitHub where developers submit their pull requests. This won't change too much. Guarantees are: if a new package from a new developer is submitted, basically none. The signing of delegation and developer keys (by repository maintainers) enables a web-of-trust style system: once package `foo` has been claimed by the well-known (and signed) devel…

Unrelated to my reservations about this system, I think the idea of computation combined with a git datastore is very interesting and possibly more generally applicable. I like the idea that the snapshot bot enforces additional guarantees on the evolution of the repository, like the ones you're describing here.

Re: Signing the OPAM Repository: TUF Meets Git

#13
post #9
post #8

Earlier quoted context omitted.

I cannot follow your claim "stronger procedures and guarantees than what we are getting here". CAs might be useful for a single resource, but we have here a bag of packages and want to distribute the whole repository (with end-to-end signatures). Also, please read the TUF paper and specification which elaborates on the motivation. And serving sth over http/s and the repository out-of-band opens possible attacks - tha…

The fundamental problem here is signing keys as authoritative over subsets of a namespace (I believe). CAs are pretty good at that, so we should compare TUF to the CA system (with pinning). In practice, that means that you have to get a certificate for your package, which also requires a DNS name and SSL hosting. You could also use the PKI infrastructure, but only to sign the packages, and not require TLS. If you wan…

It is not clear to me what you are reading in the proposal. There is no need for DNS and SSL hosting. Packages are put as tarball onto the Internet. Then in the opam-repository the metadata, including a signature over the tarball. The tarball is fetched (over any insecure connection) and its signature is verified. The naming and trust delegation happens within opam and the opam repository.

How is an update supposed to work if the connection to the timestamp is prevented/intercepted? How if the connection to the repository is prevented/intercepted? Using a single connection leads to less potential failure modes.

Re: Signing the OPAM Repository: TUF Meets Git

#14
post #7

Earlier quoted context omitted.

You should briefly think how opam does updates currently: a repository on GitHub where developers submit their pull requests. This won't change too much. Guarantees are: if a new package from a new developer is submitted, basically none. The signing of delegation and developer keys (by repository maintainers) enables a web-of-trust style system: once package `foo` has been claimed by the well-known (and signed) devel…

Unrelated to my reservations about this system, I think the idea of computation combined with a git datastore is very interesting and possibly more generally applicable. I like the idea that the snapshot bot enforces additional guarantees on the evolution of the repository, like the ones you're describing here.

Off-topic but have you come across Irmin? It seems like something you might be interested in.

https://mirage.io/blog/introducing-irmin

Re: Signing the OPAM Repository: TUF Meets Git

#15

What I do not get about these repositories is, why they have no namespacing or a built in democratic mechanism to transfer ownership. Because this does not exist, two related problems occur: Outdated packages squatting their own name and otherwise worse packages squatting a generic name. The adoption problems Python 3 still has at this point, could be mitigated entirely if package owners could lose the name for outda…

A name, once taken, cannot be unclaimed. If the package should be removed, an empty delegation will be put there.

Re: Signing the OPAM Repository: TUF Meets Git

#16
post #13
post #9

Earlier quoted context omitted.

The fundamental problem here is signing keys as authoritative over subsets of a namespace (I believe). CAs are pretty good at that, so we should compare TUF to the CA system (with pinning). In practice, that means that you have to get a certificate for your package, which also requires a DNS name and SSL hosting. You could also use the PKI infrastructure, but only to sign the packages, and not require TLS. If you wan…

It is not clear to me what you are reading in the proposal. There is no need for DNS and SSL hosting. Packages are put as tarball onto the Internet. Then in the opam-repository the metadata, including a signature over the tarball. The tarball is fetched (over any insecure connection) and its signature is verified. The naming and trust delegation happens within opam and the opam repository. How is an update supposed t…

I understand that; I am trying to explain one way that a CA-based system might work in response to your statement that the CA system was not applicable. I believe I understand the basics of the TUF system, though the mechanics of your update mechanism are still not entirely clear to me. I simply haven't seen a comparison to the (more naive?) alternatives like the one I am straw-man proposing.

I think that TUF and an https based system would both warn if an adversary or adverse conditions was blocking connectivity to the "root file". I don't think either can do anything more than warn. TUF makes it easier to mirror the root file (because it doesn't require HTTPS), so TUF has the advantage there. But I think an attacker could probably as easily (more easily?) block or intercept every TUF mirror (over HTTP) as they could block a small number of HTTPS update servers, so I'm not sure this is a huge advantage in real life.

I believe the same problems apply to delivery of the root file as to subsequent files; if you are concerned about malicious attackers interfering with file delivery it seems you should prefer HTTPS. Both TUF and HTTPS-for-root are basically agnostic to the delivery mechanism for subsequent files, I believe, so I don't see this as an advantage for either: for both an attacker can trivially block individual files over HTTP; if you use HTTPS as the transport it is still trivial to block connectivity altogether, and neither approach offers a solution to this no-connectivity attack.

Re: Signing the OPAM Repository: TUF Meets Git

#17
post #14

Earlier quoted context omitted.

Unrelated to my reservations about this system, I think the idea of computation combined with a git datastore is very interesting and possibly more generally applicable. I like the idea that the snapshot bot enforces additional guarantees on the evolution of the repository, like the ones you're describing here.

Off-topic but have you come across Irmin? It seems like something you might be interested in. https://mirage.io/blog/introducing-irmin

That's the sort of thing I'm very interested in - thanks for the link. I'm particularly interested in doing this actually over git, so the mapping to git is great (as you can probably infer, I think there are big advantages to leveraging existing "devil you know" systems!)

Re: Signing the OPAM Repository: TUF Meets Git

#18
post #6

Earlier quoted context omitted.

Not at all: adding new packages or developers is automatically signed (by the snapshot bot). Administrators may later sign the keys and delegations, individually or in batch, after verifying the developer identity out of band (the specification doesn't yet include the mechanisms for this verification, but with a git repo based on GitHub, the GitHub login gives a good start).

Thank you for the clarification. Can I install these "not yet trusted" packages? Presumably I can, I just get a big warning. And how do you deal with squatters - can I (as the "real developer") submit a second claim on an unverified package name (that an impostor has claimed)? Edit: And does this mean that the snapshot bot automatically merges (well-formed) pull requests for new packages?

The pull-requests for new packages are manually merged (as it the case now), automatically signed.

Names uniquely identify one package. A repository maintainer can replace the owner of a package, and clear the existing package contents. This shouldn't be done for a different package with the same name if any users still have the package installed, though, as they will consider that an up/down-grade.

Hope that clarifies a little!

Re: Signing the OPAM Repository: TUF Meets Git

#19
post #13

Earlier quoted context omitted.

It is not clear to me what you are reading in the proposal. There is no need for DNS and SSL hosting. Packages are put as tarball onto the Internet. Then in the opam-repository the metadata, including a signature over the tarball. The tarball is fetched (over any insecure connection) and its signature is verified. The naming and trust delegation happens within opam and the opam repository. How is an update supposed t…

I understand that; I am trying to explain one way that a CA-based system might work in response to your statement that the CA system was not applicable. I believe I understand the basics of the TUF system, though the mechanics of your update mechanism are still not entirely clear to me. I simply haven't seen a comparison to the (more naive?) alternatives like the one I am straw-man proposing. I think that TUF and an…

opam could show a warning if it didn't have an updated signed timestamp for a certain amount of time (1 day, 7 days). We had a similar warning for ClamAV when no anti-virus signature database updates were seen for a certain amount of time, usually indicating DNS problems or bad mirror selection.

Re: Signing the OPAM Repository: TUF Meets Git

#20
post #18

Earlier quoted context omitted.

Thank you for the clarification. Can I install these "not yet trusted" packages? Presumably I can, I just get a big warning. And how do you deal with squatters - can I (as the "real developer") submit a second claim on an unverified package name (that an impostor has claimed)? Edit: And does this mean that the snapshot bot automatically merges (well-formed) pull requests for new packages?

The pull-requests for new packages are manually merged (as it the case now), automatically signed. Names uniquely identify one package. A repository maintainer can replace the owner of a package, and clear the existing package contents. This shouldn't be done for a different package with the same name if any users still have the package installed, though, as they will consider that an up/down-grade. Hope that clarifi…

That does help me a lot - thanks! The manual "pre-approval process" via accepting the PR is what I was missing, and that does seem to fix most of the shortcomings of the underlying system (assuming a reliable human!) So you have a human that looks at incoming requests to prevent abuse, and to arbitrate problems. And then the snapshot server signs that result to allow distribution over HTTP, and then periodically it gets signed by the quorum of "gods" to indicate that these packages have the full seal of approval.

I think it is pretty specific to OCaml's current workflow and volume, but signing is better than not signing!

Post reply on HN