Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logging sucks

loggingsucks.com

Post reply on HN
Print view View original

201–210 of 232 posts

Re: Logging sucks

#201

Post by lll-o-lll »

otterley wrote: post #199
lll-o-lll wrote: post #197

Earlier quoted context omitted.

> And it is! The record is either in local storage or in central storage. But it isn’t! Because there are many hardware failure modes that mean that you aren’t getting your log back. For the same reason that you need acks=all in Kafka for zero data loss, or synchronous_commit = remote_flush in PostgreSQL, you need to commit your audit log to more than the local disk!

If your hardware and software can’t guarantee that writes are committed when they say they are, all bets are off. I am assuming a scenario in which your hardware and/or cloud provider doesn’t lie to you. In the world you describe, you don’t have any durability when the network is impaired. As a purchaser I would not accept such an outcome.

It’s about avoiding single points of failure.

> In the world you describe, you don’t have any durability when the network is impaired.

Yes, the real world. If you want durability, a single physical machine is never enough.

This is standard distributed computing, and we’ve had all (most) of the literature and understanding of this since the 70’s. It’s complicated, and painful to get right, which is why people normally default to a DB (or cloud managed service).

The reason this matters for this logging scenario is that I normally don’t care if I lose a bit of logging in a catastrophic failure case. It’s not ideal, but I’m trading RPO for performance. However, when regs say “thou shalt not lose thy data”, I move the other way. Which is why the streams are separate. It does impose an architectural design constraint because audit can’t be treated as a subset of logs.

Re: Logging sucks

#202

Post by mrkeen »

8n4vidtmkvmk wrote: post #47
mrkeen wrote: post #37

> Your logs are lying to you. Not maliciously. They're just not equipped to tell the truth. The best way to equip logs to tell the truth is to have other parts of the system consume them as their source of truth. Firstly: "what the system does" and "what the logs say" can't be two different things. Secondly: developers can't put less info into the logs than they should, because their feature simply won't work without…

That doesn't sound like a good plan. You're coupling logging with business logic. I don't want to have to think if i change a debug string am i going to break something.

> You're coupling logging with business logic

Yes, the system shall not report that "User null was created" if it was actually "User 123 that was created".

String? Not a chance, make a proper type-safe struct. UserCreated { "id": 123}

> I don't want to have to think if i change a debug string am i going to break something.

Good point, you should probably have a unit test somewhere.

Re: Logging sucks

#203

Post by mrkeen »

SoftTalker wrote: post #71
8n4vidtmkvmk wrote: post #47

Earlier quoted context omitted.

That doesn't sound like a good plan. You're coupling logging with business logic. I don't want to have to think if i change a debug string am i going to break something.

You're also assuming your log infrastructure is a lot more durable than most are. Generally, logging is not a guaranteed action. Writing a log message is not normally something where you wait for a disk sync before proceeding. Dropping a log message here or there is not a fatal error. Logs get rotated and deleted automatically. They are designed for retroactive use and best effort event recording, not assumed to be a…

> You're also assuming your log infrastructure is a lot more durable than most are.

Make actions, not assumptions. Instead of using a one machine storage system, distribute that storage across many machines. Then stop deleting them.

> Dropping a log message here or there is not a fatal error.

I would try to reallocate my effort budget to things that actually need to work.

Drop logging completely, and come back to it once you have a flawless record of everything the system did. The reconsider whether you need it.

Re: Logging sucks

#204

Post by manmal »

simonw wrote: post #198
manmal wrote: post #180

Earlier quoted context omitted.

The one on Generate AI seems a bit outdated. This was before Claude Code was released.

Most of that one still rings very true to me. I particularly liked this section: > Let’s start here: hiring engineers is not a process of “picking the best person for the job”. Hiring engineers is about composing teams. The smallest unit of software ownership is not the individual, it’s the team. Only teams can own, build, and maintain a corpus of software. It is inherently a collaborative, cooperative activity.

I totally agree with this part.

Right now, we are in a transitioning phase, where parts of a team might reject the notion of using AI, while others might be using it wisely, and still others might be auto-creating PRs without checking the output. These misalignments are a big problem in my view, and it’s hard to know (for anybody involved) during hiring what the stance really is because the latter group is often not honest about it.

Re: Logging sucks

#205

Post by otterley »

lll-o-lll wrote: post #201
otterley wrote: post #199

Earlier quoted context omitted.

If your hardware and software can’t guarantee that writes are committed when they say they are, all bets are off. I am assuming a scenario in which your hardware and/or cloud provider doesn’t lie to you. In the world you describe, you don’t have any durability when the network is impaired. As a purchaser I would not accept such an outcome.

It’s about avoiding single points of failure. > In the world you describe, you don’t have any durability when the network is impaired. Yes, the real world. If you want durability, a single physical machine is never enough. This is standard distributed computing, and we’ve had all (most) of the literature and understanding of this since the 70’s. It’s complicated, and painful to get right, which is why people normally…

> If you want durability, a single physical machine is never enough.

It absolutely can be. Perhaps you are unfamiliar with modern cloud block storage, or RAID backed by NVRAM? Both have durability far above and beyond a single physical disk. On AWS, for example, ec2 Block Express offers 99.999% durability. Alternatively, you can, of course, build your own RAID 1 volumes atop ordinary gp3 volumes if you like to design for similar loss probabilities.

Again, auditors do not care -- a fact you admitted yourself! They care about whether you took reasonable steps to ensure correctness and availability when needed. That is all.

> when regs say “thou shalt not lose thy data”, I move the other way. Which is why the streams are separate. It does impose an architectural design constraint because audit can’t be treated as a subset of logs.

There's no conflict between treating audit logs as logs -- which they are -- with having separate delivery streams and treatment for different retention and durability policies. Regardless of how you manage them, it doesn't change their fundamental nature. Don't confuse the nature of logs with the level of durability you want to achieve with them. They're orthogonal matters.

Re: Logging sucks

#206

Post by 0xbadcafebee »

minitech wrote: post #188
0xbadcafebee wrote: post #148

Earlier quoted context omitted.

Say they used AI to write it, it came out bad, and they published it anyway. They had the opportunity to "make it better" before publishing, but didn't. The only conclusion for this is, they just aren't good at writing. So whether AI is used or not, it'll suck either way. So there's no need to complain about the AI. It's like complaining that somebody typed a crappy letter rather than hand-wrote it. Either way the le…

Compared to human bad writing, AI writing tends to suck more verbosely and in exciting new ways (e.g. by introducing factual errors).

> AI writing tends to suck more verbosely

So, it's the style you oppose, the way a grammar nazi complains about "improper" English

> and in exciting new ways (e.g. by introducing factual errors).

Because factually incorrect comments didn't exist before AI?

Your concern is that you read something you don't like, so you pick the lowest-effort criteria to complain about. Speaks more about you than the original commenter.

Re: Logging sucks

#207

Post by 0xbadcafebee »

Dylan16807 wrote: post #182
0xbadcafebee wrote: post #148

Earlier quoted context omitted.

Say they used AI to write it, it came out bad, and they published it anyway. They had the opportunity to "make it better" before publishing, but didn't. The only conclusion for this is, they just aren't good at writing. So whether AI is used or not, it'll suck either way. So there's no need to complain about the AI. It's like complaining that somebody typed a crappy letter rather than hand-wrote it. Either way the le…

> The only conclusion for this is, they just aren't good at writing. Not true. It's likely an effort issue in that situation. And that kind of effort issue is good to call out, because it compounds the low quality.

I don't know if you're new to the internet, but low-effort comments have existed before AI, and will continue to exist regardless of AI.

Re: Logging sucks

#208

Post by lll-o-lll »

otterley wrote: post #205
lll-o-lll wrote: post #201

Earlier quoted context omitted.

It’s about avoiding single points of failure. > In the world you describe, you don’t have any durability when the network is impaired. Yes, the real world. If you want durability, a single physical machine is never enough. This is standard distributed computing, and we’ve had all (most) of the literature and understanding of this since the 70’s. It’s complicated, and painful to get right, which is why people normally…

> If you want durability, a single physical machine is never enough. It absolutely can be. Perhaps you are unfamiliar with modern cloud block storage, or RAID backed by NVRAM? Both have durability far above and beyond a single physical disk. On AWS, for example, ec2 Block Express offers 99.999% durability. Alternatively, you can, of course, build your own RAID 1 volumes atop ordinary gp3 volumes if you like to design…

> It absolutely can be. Perhaps you are unfamiliar with modern cloud block storage, or RAID backed by NVRAM? Both have durability far above and beyond a single physical disk. On AWS, for example, ec2 Block Express offers 99.999% durability. Alternatively, you can, of course, build your own RAID 1 volumes atop ordinary gp3 volumes if you like to design for similar loss probabilities.

Certainly you can solve for zero data loss (RPO=0) at the infrastructure level. It involves synchronously replicating that data to a separate physical location. If your threat model includes “fire in the dc”, reliable storage isn’t enough. To survive a site catastrophe with no data loss you must maintain a second, live copy (synchronous replication before ack) in another fault domain.

In practice, to my experience, this is done at the application level rather than trying to do so with infrastructure.

> There's no conflict between treating audit logs as logs -- which they are -- with having separate delivery streams and treatment for different retention and durability policies

It matters to me, because I don’t want to be dependent on a sync ack between two fault domains for 99.999% of my logs. I only care about this when the regulator says I must.

> Again, auditors do not care -- a fact you admitted yourself! They care about whether you took reasonable steps to ensure correctness and availability when needed. That is all.

I care about matching the solution to the regulation; which varies considerably by country and use-case. However there are multiple cases I have been involved with where the stipulation was “you must prove you cannot lose this data, even in the case of a site-wide catastrophe”. That’s what RPO zero means. It’s DR, i.e., after a disaster. For nearly everything 15 minutes is good, if not great. Not always.

Re: Logging sucks

#209

Post by otterley »

lll-o-lll wrote: post #208
otterley wrote: post #205

Earlier quoted context omitted.

> If you want durability, a single physical machine is never enough. It absolutely can be. Perhaps you are unfamiliar with modern cloud block storage, or RAID backed by NVRAM? Both have durability far above and beyond a single physical disk. On AWS, for example, ec2 Block Express offers 99.999% durability. Alternatively, you can, of course, build your own RAID 1 volumes atop ordinary gp3 volumes if you like to design…

> It absolutely can be. Perhaps you are unfamiliar with modern cloud block storage, or RAID backed by NVRAM? Both have durability far above and beyond a single physical disk. On AWS, for example, ec2 Block Express offers 99.999% durability. Alternatively, you can, of course, build your own RAID 1 volumes atop ordinary gp3 volumes if you like to design for similar loss probabilities. Certainly you can solve for zero d…

> It matters to me, because I don’t want to be dependent on a sync ack between two fault domains for 99.999% of my logs. I only care about this when the regulator says I must.

If you want synchronous replication across fault domains for a specific subset of logs, that’s your choice. My point is that treating them this way doesn’t make them not logs. They’re still logs.

I feel like we’re largely in violent agreement, other than whether you actually need to do this. I suspect you’re overengineering to meet an overly stringent interpretation of a requirement. Which regimes, specifically, dictated that you must have synchronous replication across fault domains, and for which set of data? As an attorney as well as a reliability engineer, I would love to see the details. As far as I know, no one - no one - has ever been held to account by a regulator for losing covered data due to a catastrophe outside their control, as long as they took reasonable measures to maintain compliance. RPO=0, in my experience, has never been a requirement with strict liability regardless of disaster scenario.

Re: Logging sucks

#210

Post by lll-o-lll »

otterley wrote: post #209
lll-o-lll wrote: post #208

Earlier quoted context omitted.

> It absolutely can be. Perhaps you are unfamiliar with modern cloud block storage, or RAID backed by NVRAM? Both have durability far above and beyond a single physical disk. On AWS, for example, ec2 Block Express offers 99.999% durability. Alternatively, you can, of course, build your own RAID 1 volumes atop ordinary gp3 volumes if you like to design for similar loss probabilities. Certainly you can solve for zero d…

> It matters to me, because I don’t want to be dependent on a sync ack between two fault domains for 99.999% of my logs. I only care about this when the regulator says I must. If you want synchronous replication across fault domains for a specific subset of logs, that’s your choice. My point is that treating them this way doesn’t make them not logs. They’re still logs. I feel like we’re largely in violent agreement,…

> I suspect you’re overengineering to meet an overly stringent interpretation of a requirement. Which regimes, specifically, dictated that you must have synchronous replication across fault domains, and for which set of data? As an attorney as well as a reliability engineer, I would love to see the details.

I can’t go into details about current cases with my current employer, unfortunately. Ultimately, the requirements go through legal and are subject to back and forth with representatives of the government(s) in question. As I said, the problem isn’t passing an audit, it’s getting the initial approval to implement the solution by demonstrating how the requirement will be satisfied. Also, cloud companies are in the same boat, and aren’t certified for use as a result.

This is the extreme end of when you need to be able to say “x definitely happened” or “y definitely didn’t happen” It’s still a “log” from the applications perspective, but really more of a transactional record that has legal weight. And because you can’t lose it, you can’t send it out the “logging” pipe (which for performance is going to sit in a memory buffer for a bit, a local disk buffer for longer, and then get replicated somewhere central), you send it out a transactional pipe and wait for the ack.

Having a gov tell us “this audit log must survive a dc fire” is a bit unusual, but dealing with the general requirement “we need this data to survive a dc fire”, is just another Tuesday. An audit log is nothing special if you are thinking of it as “data”.

You’re a reliability engineer, have you never been asked to ensure data cannot be lost in the event of a catastrophe? Do you agree that this requires synchronous external replication?

Post reply on HN