Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap outage postmortem

mail.tarsnap.com

Post reply on HN
Print view View original

331–338 of 338 posts

Re: Tarsnap outage postmortem

#331

Post by prmoustache »

Aachen wrote: post #318
prmoustache wrote: post #111

Earlier quoted context omitted.

I understand some websites are doing lots of funky stuff, but at this point the issue is on your particular client if it cannot zoom bare text correctly and smoothly. Any decent client should be able to allow you to zoom it so that you can maximize those 80 character column to the width of your phone screen in protrait mode regardless of font size and dpi.

Try it if you don't have a perfect 100% vision (20/20 this is called in USA I think?) and have a phone that fits in a normal pocket Optimal reading width for speed/comprehension is also fewer than 80 characters afaik. I think different sources I read years ago were undecided whether it's closer to 60- or 70-character lines. Either way, rescaling when there is no ASCII art or position-dependent characters seems rather…

I am using prescription glasses. That is what they are for. If you are using yours and still can't read small text, you need a visit to your optiometrist and get new ones.

Re: Tarsnap outage postmortem

#332

Post by prmoustache »

hollerith wrote: post #319
prmoustache wrote: post #111

Earlier quoted context omitted.

I understand some websites are doing lots of funky stuff, but at this point the issue is on your particular client if it cannot zoom bare text correctly and smoothly. Any decent client should be able to allow you to zoom it so that you can maximize those 80 character column to the width of your phone screen in protrait mode regardless of font size and dpi.

What about us poor slobs whose vision is less than perfect where the 80-character line is too small even when it takes up the entire width of the display? The way zooming is implemented in the mobile browsers, we must horizontally scroll back and forth for every line we read.

I am using prescription glasses. That is what they are for. Maybe you should use yours?

Re: Tarsnap outage postmortem

#333

Post by chibea »

mattbee wrote: post #76
sandgiant wrote: post #65

Earlier quoted context omitted.

Curious how much you backup, which version of restic you're running and why you think the deduplication is borderline unacceptable. There were several major (orders of magnitudes) improvements made to pruning within the past ~1 year, that's why I'm interested.

A straight upgrade, that I can do :) It's been running for years without one. I was only edgy about it because when it takes 36h it blocks the next daily backup, and I wondered whether that was going to get worse (it hasn't).

The max-unused percentage feature is well worth it to 80/20 the prune process and only prune the data which is easiest to prune away (i.e. not try to remove small files big packs but focus on packs which have lots of garbage).

In general, there's an unavoidable trade-off between creating many small packs (harder on metadata throughout the system, inside restic and on the backing store but more efficient to prune) versus creating big packs which are more easy on the metadata but might create big repack cost.

I guess a bit more intelligent repacking could avoid some of that cost by packing stuff together that might be more likely to get pruned together.

Re: Tarsnap outage postmortem

#334

Post by [deleted] »

prmoustache wrote: post #332
hollerith wrote: post #319

Earlier quoted context omitted.

What about us poor slobs whose vision is less than perfect where the 80-character line is too small even when it takes up the entire width of the display? The way zooming is implemented in the mobile browsers, we must horizontally scroll back and forth for every line we read.

I am using prescription glasses. That is what they are for. Maybe you should use yours?

[deleted]

Re: Tarsnap outage postmortem

#335

Post by hollerith »

prmoustache wrote: post #332
hollerith wrote: post #319

Earlier quoted context omitted.

What about us poor slobs whose vision is less than perfect where the 80-character line is too small even when it takes up the entire width of the display? The way zooming is implemented in the mobile browsers, we must horizontally scroll back and forth for every line we read.

I am using prescription glasses. That is what they are for. Maybe you should use yours?

I don't think my glasses can correct my vision well enough to be able to read 80-char-wide text on a smartphone in portrait mode. I can't say for sure because I don't own a smartphone--partly because I worry about being able to read web pages on it without regularly resorting to the aforementioned tedious horizontal scrolling.

I always wear my glassses when I'm using my iPad, which is in landscape mode almost all the time (the exceptions being the rare apps like Uber's that won't do landscape mode).

Re: Tarsnap outage postmortem

#336

Post by nijave »

idlewords wrote: post #220
jacquesm wrote: post #192

Earlier quoted context omitted.

In future postmortems (of which I hope there will be very few or even none) you may want to spell out your 'lessons learned' to show why particular items will never recur.

It always amuses me how people want reassurance that the next crisis will be a fresh, new problem, and not one the person can demonstrably solve. A lot of 'lessons learned' analysis boils down to this: in order to prevent a recurrence of X, we introduced complex subsystem Y, the unexpected effects of which you can read about in our next post-mortem.

Could be as simple as "test restore a new server every 1-2 years"

Re: Tarsnap outage postmortem

#337

Post by nijave »

zetalyrae wrote: post #95

>The process of recovering the EC2 instance state consists of two steps: First, reading all of the metadata headers from S3; and second, "replaying" all of those operations locally. (These cannot be performed at the same time, since the use of log-structured storage means that log entries are "rewritten" to free up storage when data is deleted; log entries contain sequence numbers to allow them to be replayed in the…

There are some established patterns for S3 as a database. It's extremely common in "data lakes" (throw data of various schemas in and use a tool that can parse at query time).

There's client libraries like Delta Lake that implement ACID on S3.

Much of the Grafana stack uses S3 for storage (Mimir/metrics, Loki/logs, Tempo/traces).

That said, I'm not sure about the implementation Tarsnap uses--if it's completely ad-hoc or based of other patterns/libraries.

Re: Tarsnap outage postmortem

#338

Post by khiqxj »

dang wrote: post #330
khiqxj wrote: post #274

Earlier quoted context omitted.

[flagged]

Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

[dead]
Post reply on HN