161–170 of 230 posts
Earlier quoted context omitted.
Yes, this is why I implemented hash-checking in pip ( https://pip.pypa.io/en/stable/topics/repeatable-installs/#ha... ). Running your own server is certainly another way to solve the problem (and lets you work offline), but keeping the pinning info in version control gives you a built-in audit trail, code reviews, and one fewer server to maintain.
Doesn't Poetry do this by default in its lockfile too?
Earlier quoted context omitted.
> pip provides isolation between workspaces that apt has never been able to do natively because it's too generic This is hardly a concern on production systems. It's been common practice for decades to deploy each service on a dedicated host, or at least a VM in order to have a degree of security isolation. [or a container if you don't care] > npm, cargo, etc showed how to include a dependency multiple times at diffe…
If you don't want or need version freedom, what is it about the existing deb/rpm ecosystems that don't work for you, either on their own or in conjunction with the various tools that bridge other ecosystems into them? (I'm a fan of dh_virtualenv, myself) Or is your lament that the world as a whole is worse off because others have eaten of the fruit of systems like npm and pip?
(and here I'm even getting downvoted for going against the hivemind)
I can only hope that after enough software supply chain attacks the industry will realize that distros were right all along.
This is why our build systems don’t use public repositories directly, and why we always pin to an exact version. Any third party dependencies (js/python/java/c/you-name-it) are manually uploaded to our Artifactory server- which itself has no internet access. All third party libraries are periodically checked for new versions, any security announcements etc, and only if we are happy do we update the internal repo. It…
We did this at one of my previous companies, and, of all the things that ever went wrong with our deploy processes, our internal PyPi server was literally never the culprit.
---
[0]: https://pythonwheels.com/
This is why our build systems don’t use public repositories directly, and why we always pin to an exact version. Any third party dependencies (js/python/java/c/you-name-it) are manually uploaded to our Artifactory server- which itself has no internet access. All third party libraries are periodically checked for new versions, any security announcements etc, and only if we are happy do we update the internal repo. It…
Own hosting, pinning, and checksum checks are defenses against network-based attacks and compromise of the package repository. They do nothing against trojans like these. You will be running your own pinned checksummed version of the malicious code. If you want to stop malware published by the legitimate package author, you need to review the code you're pulling in and/or tightly sandbox it (and effective sandboxing…
If you mean having humans inspect every line of code of every package, well... good luck with that. But, if you're talking about automated analysis and acceptance testing, then, I think you've got something.
Like unit testing, automated testing and analysis is never going to catch 100% of all issues, but it can definitely help your SRE team sleep at night.
It seems to me like one low hanging fruit to make a lot of these kinds of exploits significantly more difficult is protection at a language level about which libraries are allowed to make outgoing HTTP requests or access the file system. It would be great if I could mark in my requirements.txt that a specific dependency should not be allowed to access the file system or network, and have that transitively apply to ev…
Deno (a Node-like runtime by the original author of Node) has a security model kind of like this [0]. Its unfortunately not as granular as I think it should be (only operates on the module level and not individual dependencies), but its a start. [0] https://deno.land/manual/getting_started/permissions
It covers a lot of terrain including the connectivity permissions control.
I recommend it as an easy way to learn about Deno and how it is different from Node as it is today.
Node seems to have evolved to handle some of what Deno set out to do at the start. It is worth hearing from Dahl why Deno is still relevant and for what use cases.
Dahl speaks without ego and addresses interesting topics like the company built around Deno and monetization plans.
This is why our build systems don’t use public repositories directly, and why we always pin to an exact version. Any third party dependencies (js/python/java/c/you-name-it) are manually uploaded to our Artifactory server- which itself has no internet access. All third party libraries are periodically checked for new versions, any security announcements etc, and only if we are happy do we update the internal repo. It…
I did a bunch of nodejs stuff at my last gig. These teams had the of practice keeping packages up to date. Drove me frikkin nuts. So much churn, chaos. Is this a JavaScript thing? Carried over from frontend development? Exasperated, I finally stopped advocating for locking everything down. Everyone treated me like I was crazy. (Reproducible builds?? Pfft!!) Happens with enterprisey Java, Sprint, Maven projects too. (…
Gotta disagree with this part. If you're making a web app package updates really need to be done on a regular cadence, ideally every quarter, but twice a year at minimum IMO. In the .Net world at least it feels like most responsible open source maintainers make it relatively painless to upgrade between incremental major versions (e.g. v4 -> v5). If you put off upgrades until someone holds a gun to your head so that your dependencies are years out of date, you're much more likely to experience painful upgrades that require a lot of work.
This is why our build systems don’t use public repositories directly, and why we always pin to an exact version. Any third party dependencies (js/python/java/c/you-name-it) are manually uploaded to our Artifactory server- which itself has no internet access. All third party libraries are periodically checked for new versions, any security announcements etc, and only if we are happy do we update the internal repo. It…
I did a bunch of nodejs stuff at my last gig. These teams had the of practice keeping packages up to date. Drove me frikkin nuts. So much churn, chaos. Is this a JavaScript thing? Carried over from frontend development? Exasperated, I finally stopped advocating for locking everything down. Everyone treated me like I was crazy. (Reproducible builds?? Pfft!!) Happens with enterprisey Java, Sprint, Maven projects too. (…
The downside, however, is that, by design, you end up with packages that don't get upgraded regularly. That can cause problems down the road when you decide you do want to upgrade those packages.
For instance, there might be breaking changes, because you're jumping major versions. Of course, breaking changes are always a problem, but, if you're not regularly upgrading stuff, your team will tend to build on/build around the functionality of the old version.
That leads to some real fun come upgrade time. If, you're, say, 3 major versions behind the latest version, or whatever version you want to upgrade to that contains some Cool New Feature(tm) you really, really need, you might end up having to do this silly dance of upgrading major versions one at a time, in order to keep the complexity of the upgrade process as a whole under control.
Oh, and, sometimes things get deprecated. That's always fun to deal with.
So, TL;DR: Yes, pin versions! It's safer that way! Just be aware that, like most engineering decisions, there's a tradeoff here that saves you some pain now in exchange for some amount of a different kind of pain in the future.
Nice writeup, but the title flashes to '(1) New Message' and back twice a second. That's kind of silly in my opinion, from whom do I expect the message? I assume from the chatbot at the bottom right corner. Even so, to talk to it I would need to grant it access to some personal information. It all ends up leaving a bitter aftertaste. Whatever the message was, why not place it in a block of text somewhere less distrac…
This junk is appearing on more and more web sites, at least this one is clearly a bot. Plenty of sales sites will pretend a human is sending you a message, try and talk back and all of a sudden you're in a queue waiting for a reply. Another anti pattern for web.
This is the first time I've seen a page flashing the title like that. Extremely annoying and I closed the page before reading the article to the end. It reminded me of the times when pages used to do that with the browser status bar on the bottom of the window.
Earlier quoted context omitted.
Everyone I know uses some form of lock file, and most of the modern programming languages support it. As for upgrading only when absolutely necessary, let's be honest, nothing is absolutely necessary. If the software is old, or slow, or buggy, well dear users you'll just have to deal with it. In my experience however, it's easier to keep dependencies relatively up to date all the time, and do the occasional change th…
Both extremes are bad. If you never change anything, you are left behind on a lot of security updates and bug fixes. The longer you wait the harder it is to move. The “stay current” model comes with risks too. It’s just a matter of figuring out which manner has the better value to risk trade off, and how to mitigate the risks.
If it got a green build, it would make a PR with the upgrades, which you could then either choose to merge, or tell the bot to STFU about that dependency (or optionally, STFU until $SOME_NEWER_VERSION or higher is available, or there's a security issue with the current version).
If not, it would send a complain-y email to the dev team about it, which we could either silence or address by manually doing the upgrade.
This worked out rather well for us. I think the net effect of having the bot was to make sure we devs actually paid attention to what versions of our dependencies we were using.
Earlier quoted context omitted.
How do you deal with regressions? For example, we once upgraded the redis client. One brief revision of the parser submodule had an apparent resource leak. (I can't imagine how...) Causing all of our services to ABEND after a few hours. Because everything is updated aggressively, and there's so many dependencies, we couldn't easily back out changes. -- FWIW, Gilt's "Test Into Production" strategy is the first and onl…
When I see a regression, I look at what was recently updated (in the past hours, days), and it's usually one of those packages. Because frequent updates tend to be independent, it's usually not difficult to revert that change (eg. if I update react and lodash at the same time, chances are really good that I can revert one of those changes independently without any issues)
Also, you should try to update as few things as possible at once, and let your changes "soak" into production for a while before going in all "upgrade ALL the things!"
Why? Well, sometimes things that fail take a while to actually start showing you symptoms. Sometimes, the failure itself takes a while to propagate, simply due to the number of servers you have.
And, one of these days, cosmic rays, or some other random memory/disk error is going to flip a critical bit in some crucial file in that neato library you depend on. And, oh, the fouled up build is only going to go out to a subset of your machines.
You'll be glad you didn't "upgrade ALL the things!" when any of these things ends up happening to you.