Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The internet is held together with spit and baling wire

krebsonsecurity.com

Post reply on HN
Print view View original

61–70 of 176 posts

Re: The internet is held together with spit and baling wire

#61

Post by jl6 »

imchillyb wrote: post #19

Has no one here seen: https://www.reddit.com/r/cablefail/ There are 1000s of these pictures. The entire world's IT infrastructure has been held together with spaghetti-noodle-cabling and bodged patches for over three decades now. I'm even guilty of it. Most IT guys are guilty of it. Don't even get me started talking about how bodged together corporate codebases are. Banks are perhaps the worst offenders. Old hardware…

> Don't even get me started talking about how bodged together corporate codebases are. Banks are perhaps the worst offenders. Old hardware running with bandaids and bubblegum. Software that more than 5 different teams have had their fingers in, mucking about, and some codebases have orders of magnitude more than that.

And yet, perhaps counterintuitively, the main downside of such systems is that they are slow and expensive to change, not that they are unreliable.

Re: The internet is held together with spit and baling wire

#62

Post by cgh »

hypertele-Xii wrote: post #3
ragingrobot wrote: post #2

O'Brien: I'm afraid to touch anything. It's all cross-circuited and patched together - I can't make head nor tails of it. Bashir: Sounds like one of your repair jobs. Seriously though, it seems like every form of infrastructure we rely on is held together in such a fragile manner. I hate hate to think of the chaos should there be a major Internet and physical infra failure in close proximity, time-wise.

We have a pretty good track record keeping it running though; The Internet has never gone down! I'd be far more concerned with agricultural logistics, though we've never starved to extinction, either. Perhaps we've reached a point of positive no return: We can no longer cease to exist!

There’s a strong probabilistic argument that says otherwise: https://en.wikipedia.org/wiki/Doomsday_argument

Note that the argument is not dependent on any sort of cause, such as climate change or whatever. It is entirely probabilistic.

Re: The internet is held together with spit and baling wire

#63

Post by WJW »

999900000999 wrote: post #37
zwkrt wrote: post #8

Earlier quoted context omitted.

Unless you exist in a state of wild over abundance and are very conscientious, most working infrastructure is always at a state of almost-disrepair. If I operate a machine shop and I have 4 partially-stripped screwdrivers, I might get a new one, but inevitably I will use it until it is as stripped as all the rest, and I don’t dare throw away a tool that is currently even in partially working condition since I might n…

You could also break something with your dull tools. This is why you can't just drive a car until it breaks, you get it checked out. Then again, cars are a private good. When it's your property vs our property you have more of an incentive to take care of it.

Machine shops are also private goods and their owners have a lot of incentive to keep them working, so I don't think this example is very accurate. In any case the chance of breaking anything with a partially stripped screwdriver is pretty minimal.

Re: The internet is held together with spit and baling wire

#64

Post by charcircuit »

imchillyb wrote: post #19

Has no one here seen: https://www.reddit.com/r/cablefail/ There are 1000s of these pictures. The entire world's IT infrastructure has been held together with spaghetti-noodle-cabling and bodged patches for over three decades now. I'm even guilty of it. Most IT guys are guilty of it. Don't even get me started talking about how bodged together corporate codebases are. Banks are perhaps the worst offenders. Old hardware…

How "pretty" your cable routing is doesn't effect performance / reliability.

Re: The internet is held together with spit and baling wire

#65

Post by winternett »

fulafel wrote: post #53
winternett wrote: post #44

Earlier quoted context omitted.

Lets encrypt from what I understand require time consuming updates every few months. My host provider also does not allow me to install them manually, further complicating the process, and conveniently they sell certs for $125 a year... Per site. It's been a thorn in my side because we're too big to easily move now.

You are absolutely not meant to do the updates manually.

On one ISP that I host sites on, they restrict cert installs and don't allow SSH access. It's done in order to sell their cert services. I have too many sites on there to move easily... It's complicated. Eventually I'll bite the bullet and move to a new host. :/

Re: The internet is held together with spit and baling wire

#66

Post by dragonwriter »

jl6 wrote: post #61
imchillyb wrote: post #19

Has no one here seen: https://www.reddit.com/r/cablefail/ There are 1000s of these pictures. The entire world's IT infrastructure has been held together with spaghetti-noodle-cabling and bodged patches for over three decades now. I'm even guilty of it. Most IT guys are guilty of it. Don't even get me started talking about how bodged together corporate codebases are. Banks are perhaps the worst offenders. Old hardware…

> Don't even get me started talking about how bodged together corporate codebases are. Banks are perhaps the worst offenders. Old hardware running with bandaids and bubblegum. Software that more than 5 different teams have had their fingers in, mucking about, and some codebases have orders of magnitude more than that. And yet, perhaps counterintuitively, the main downside of such systems is that they are slow and exp…

> And yet, perhaps counterintuitively, the main downside of such systems is that they are slow and expensive to change, not that they are unreliable.

They are slow and expensive to change because they are maintained because they have been, at great expense and cost in both failures and remediation efforts, made tolerably reliable (but still extremely fragile) so long as things are exactly within certain expectations (which often have been narrowed from the intended design based on observed bugs that have been deemed too expensive to fix), and it is inordinately difficult to modify them without causing them to revert to a state of intolerable unreliability.

They are systems that generations have been spent reshaping business operations around their bugs to make them “reliable”.

Re: The internet is held together with spit and baling wire

#67

Post by winternett »

fulafel wrote: post #56
winternett wrote: post #49

Earlier quoted context omitted.

But if you're independently running, paying for, and managing multiple sites, it's a HUGE burden. It also kills innovation for independent devs and startups, and dramatically raises the cost/investment threshold for this kind of innovation. Pricing on cert services is also far too high when everyone's concern and agreement should be security as a basis for operations. It's not something that should be an upcharge or…

Setting up certbot is easy, not a big burden for indie devs. Or if you want to know nothing about tls & certs, just get hosting that comes with tls.

I wasn't writing to the update process, as much as the original installation of a cert.

On a house you own, you can change locks and keys any time you want to keep security up to date (for example).

no house in "move in ready condition" comes without sufficiently keyed door locks of some kind (on day1).

Re: The internet is held together with spit and baling wire

#68

Post by cgh »

imchillyb wrote: post #19

Has no one here seen: https://www.reddit.com/r/cablefail/ There are 1000s of these pictures. The entire world's IT infrastructure has been held together with spaghetti-noodle-cabling and bodged patches for over three decades now. I'm even guilty of it. Most IT guys are guilty of it. Don't even get me started talking about how bodged together corporate codebases are. Banks are perhaps the worst offenders. Old hardware…

I had a friend who worked in IT in a major Canadian bank some years back. He said they’d lost the source code (Cobol, I would imagine) for certain jobs and had taken to editing the binaries directly with a hex editor.

Re: The internet is held together with spit and baling wire

#69

Post by WJW »

winternett wrote: post #49
shadowgovt wrote: post #36

Earlier quoted context omitted.

But monitoring and maintenance are things someone needs to do if they operate a site, period.

But if you're independently running, paying for, and managing multiple sites, it's a HUGE burden. It also kills innovation for independent devs and startups, and dramatically raises the cost/investment threshold for this kind of innovation. Pricing on cert services is also far too high when everyone's concern and agreement should be security as a basis for operations. It's not something that should be an upcharge or…

Having just replaced my door lock, I can assure you that they too wear out and need replacing. (one of the springs inside broke)

The pricing on Let's Encrypt is literally zero, and they provide (also free of charge) the `certbot` utility which you can run as a cronjob and which will automatically renew your certificates for you. The whole thing comes extremely well documented and with install scripts that take less than a minute to download, verify and run. If you think even that is too much of a burden I don't think any topic in programming is simple enough.

Re: The internet is held together with spit and baling wire

#70

Post by dylan604 »

AceyMan wrote: post #48
zwkrt wrote: post #8

Earlier quoted context omitted.

Unless you exist in a state of wild over abundance and are very conscientious, most working infrastructure is always at a state of almost-disrepair. If I operate a machine shop and I have 4 partially-stripped screwdrivers, I might get a new one, but inevitably I will use it until it is as stripped as all the rest, and I don’t dare throw away a tool that is currently even in partially working condition since I might n…

A corollary to this might be "a new part on the shelf is not truly 'better' than a 90% worn out one still in service." Having ready spares is great but I'm sure the HN crowd knows how much crib-death there is on new, replacement bits of all kinds. Until it's actually been run-in for a while how do you really know it's going to work when you unbox it to replace a truly dead piece? The upshot of this is' RAID-6' type d…

>Having ready spares is great but I'm sure the HN crowd knows how much crib-death there is on new, replacement bits of all kinds. >Until it's actually been run-in for a while how do you really know it's going to work when you unbox it to replace a truly dead piece?

We had the unfortunate experience of installing 4 new 16 bay chasis with brand new drives (10+ years ago now). We designated one of the 16s as hot spare for each chasis, plus had 2 cold spares for each as well. 72 brand new drives in total. All from the same batch of drives from the manufacture. Set them all up on a Friday and configured all for RAID5 (pre-RAID6 availability). Plan was to let them build and have some burn-in time over the weekend for possible Monday availability. Monday provided us with multiple drive failures in each chasis. Drive manufacturer confirmed a batch batch from whichever plant, replaced all and delivered larger sizes for replacements. Luckily, they failed during burn-in rather than 1 week after deploying.

Post reply on HN