31–40 of 57 posts
The team avoided dealing with upstream in exactly the same way that you avoided dealing with upstream. What's so difficult to understand?
Earlier quoted context omitted.
Could you explain a bit more? I think most pkg repos are just static directories. If you do https reencryption, and https certs do not match EVERYTHING will break.
Yes, it will. Normally, the IT department would roll-out their MitM CA certificates globally into Windows trusted certificates store or something like it. However, if you are trying to do CI/CD on a Linux server, you have to manually provide those CA certificates to the tools you are using. Many tools are developed in a kind of "HTTPS everywhere" bubble, so that they think HTTPS just works and don't provide any kind…
The real struggle for me is if the proxy requires authentication. It's very often not possible to configure a username/password and either way I don't want to put my employee password in every goddamn configuration file.
The CA certificate must be setup on Linux hosts, that's the bare minimum if the company wants to do SSL interception (add to /etc/pki/trust/anchors and call update-ca-trust), then things mostly work out of the box.
I've had one job where I spent a lot of time debugging and setting up PKI in a bank, have a postmortem here of how various libraries retrieve CA from the system (including obscure bugs around python ssl). https://thehftguy.com/2020/03/19/jp-morgan-postmortem-why-yo...
Of course the real solution is to have an internal mirror of linux/python/java packages.
Honestly, you have this problem because you put up with it, or are "forced" to use certain software. Why is it everytime I venture into Python, interesting language aside, there are always multitudes of dependency problems even tied to the OS? People put up with it. Choosing quality over fast-food is a choice. Languages such as golang has largely solved much of this while still being pretty portable, and devs are enc…
git clone https://example.com/myproject
cd myproject
/usr/bin/python3 -m venv env
source ./env/bin/activate
pip install requirements.txt
python3 src/myapp.pyRepeat 100 times, and now you have a real problem brewing."
A future job interview question that might be asked at my future company:
"Let's say you're using a library for specific functionality, a library that you haven't written. Now let's say that there's a bug in that library that you can't work around.
How would you debug that library?"
See, there's a differentiator there.
There are programmers who can debug libraries, and then there are programmers that can merely use libraries provided to them.
Given a choice, you want a programmer who can debug libraries working for your company...
> They responded. What did they do? They made it catch the situation where the dotfile read failed, and made it not blow up the whole program. Instead, they just carried on as if it wasn't there.
The tone here (and in subsequent paragraphs) seems to suggest that this is somehow the "wrong" answer.
Yet, it's exactly what I would've done, since not doing this is clearly wrong. The file's corrupted with no hope of recovery, is not apparently meant to be user-serviceable (so user configuration is highly unlikely to be at risk), and is evidently not essential for the proper operation of the program (nor does it, in general, hold anything especially important - and no, some counter for when to nag about updates ain't important in this situation). There is absolutely no reason why this file's corruption should prevent normal operation; therefore, gracefully catching this error and expanding the "file doesn't exist" case to "file doesn't exist or is otherwise corrupt/unusable" and proceeding normally is absolutely the right call.
In an ideal world, they'd fix the actual corruption, too, but preventing that corruption from being an issue is the first and most critical step. I'd hardly call making software less fragile (said fragility being exactly why so many "80%" libraries are indeed 80%) a "missed opportunity"; indeed, not doing this seems like a glaring missed opportunity to make the software more robust against issues far beyond those caused by the limitations of some library.
That is:
> I will never know why the team chose to handle my report by swallowing the error instead of dealing with upstream
Because not swallowing the error would be a patently broken design, regardless of whether or not there was some library involved.
----
EDIT:
> Yeah, that's right, because I worked for G or FB or whatever, somehow any time I have a problem with something, it's because I'm trying to do something at too big of a scale? Are you shitting me? COME ON.
I mean, yes? The situation described is very obviously one where the author is trying to use a tool clearly not designed for massively-parallel execution for a task involving massively-parallel execution. Why said author is somehow surprised that said tool breaks when put under that sort of stress is beyond me.
It's akin to buying a 1 ton jack from the auto parts store and then complaining to the minimum-wage employees thereof because when you tried to use it to change the treads on a 50-ton bulldozer the jack predictably flattened like a pancake.
There's always a point where off-the-shelf parts won't cut it and you gotta roll your own solution in-house. Evidently the author has a tendency to hit that point sooner rather than later. That's pretty rare, though, and condemning the very notion of an external library because of said libraries failing under very exceptional circumstances seems absurd.
That is: no shit most libraries are written for the 80%, because the 80% ain't got the resources or motivation to deal with the consequences of NIH syndrome like the 20% might. They gotta ship something, and can't afford to let perfect be the enemy of good.
The more reasonable approach (for those currently in the 80% but hoping to eventually be in the 20%) is to start with those off-the-shelf libraries, and then be prepared to fork 'em and use them as starting points for specialized approaches (or else swap them out with specialized replacements). In this regard the author's correct in that the package ecosystems of most languages are poorly suited to this, since they offer little to no mechanism for customizing dependencies without rather painful workarounds.
I have been thinking a lot lately about a possible solution for a small portion of this problem: Microdependencies. I'll explain in more detail in the context of JS, but it applies to other languages as well. Currently package repositories like NPM host 2 types of dependencies: big community packages (frameworks, database drivers, validation libraries, query builders,...) and smaller function-scoped utility packages…
CCAN (C Code Archive Network) http://ccodearchive.net/ follows a similar philosophy. CCAN contains various small, self contained snippet of C code are meant to be be downloaded in a local directory ("vendored") and used directly or with small modifications. The most straightforward way to use CCAN is to 1) `git clone` it in a subdir; 2) create a `config.h`; 3) include whatever `lib.h` you want in your project. Updati…
For the most part, these 3rd party dependencies work fine. In my experience it is fairly rare to encounter a show-stopping bug.
I have a simple mitigation for the "potholes" though - try to take the time skim through the code of the library ahead of time and try to figure out what it is doing. That way if you do hit a problem, you can fork and fix. Doing this can also give you a sense of how well-written a library is to begin with.
I have been thinking a lot lately about a possible solution for a small portion of this problem: Microdependencies. I'll explain in more detail in the context of JS, but it applies to other languages as well. Currently package repositories like NPM host 2 types of dependencies: big community packages (frameworks, database drivers, validation libraries, query builders,...) and smaller function-scoped utility packages…
Horrible idea, if a package get corrupted by an adversary not only would you have to clean the npm package but also hope everyone who committed the code into a repository gets the news and fixes it. At least with NPM there is the chance that a fixed version gets automatically pulled on the next build. In addition to that, most package managers rely on metadata, like description files, the bloat introduced by these mi…
I can definitely imagine a package manager that, in some way, differentiates between the two (in repo or not), whether manually specified or as OP suggests some distinction based on how 'big' the package is.
Right now, it feels too dichotomous. Either I use a package that itself relies on a ton of packages, and I won't read all the code changes, or I copy and paste bits of code into my repo and now have to manually update things of any consequence.
> I show up with a problem ("hey, this thing keeps getting corrupted because X and Y") and suddenly it's because I'm "from" G or FB or something and I "want unreasonable things" from their stuff. So, my request is invalid, thank you drive though. No, the answer is they're trying to deliver impact for the customer and Rachel is instead asking them to invest time in a solution that doesn't bring them any closer to that…
This is a pretty bad take. Rachel has said implicitly that there was no version with a fix available and explicitly that working with the maintainers to get a fix is often unreasonably difficult. This matches my own experience fairly well. So your ‘normal people’ solution won’t work. You may also want to consider that there are good reasons for engineers and engineering management to be default suspicious of 3rdparty…
The issue gets even harder when the code is proprietary. Had a situation with one of VMware's products that was being provided to us by a SAAS provider. It was a far amount of effort to a) convince the provider to file a bug with their vendor and b) then provide enough data to the VMware engineers so they could understand the value in prioritising a fix.
In our case, we identified the issue during a proof of concept so our vendor pushed VMware pretty hard because there was a sizeable contract at stake for them. Most engineers aren't as thorough in my experience.