Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heavy SSD Writes from Firefox

servethehome.com

Post reply on HN
Print view View original

341–350 of 361 posts

Re: Heavy SSD Writes from Firefox

#341

Post by Freestyler_3 »

smegel wrote: post #334
lighttower wrote: post #8

Chrome, on my system, is even more abusive. Watch the size of the .config/google-chrome directory and you'll see that it grows to multi-GB in the profile file. There is a Linux utility that takes care of all browsers' abuse of your ssd called profile sync daemon, PSD. It's available in the debian repo or [1] for Ubuntu or [2] for source. It uses `overlay` filesystem to direct all writes to ram and only syncs back to…

How do we switch it off on Chrome (Windows)?

I have been wondering and asked the same for chrome based browser... did not find anything so far.

Re: Heavy SSD Writes from Firefox

#342

Post by c3833174 »

Animats wrote: post #197

Firefox is relying too much on session restore to deal with bugs in their code. Firefox needs to crash less. With all the effort going into multiprocess Firefox, Rust, and Servo, it should be possible to have one page abort without taking down the whole browser. About half the time, session restore can't restore the page that crashed Firefox anyway.

It has been months since the last crash on my machine and it has definitely improved over the years (3.0 used to crash much more frequently). And even on chrome which has been multiprocess from the beginning I've had the whole browser crash after closing a "sad tab".

Re: Heavy SSD Writes from Firefox

#343

Post by d0vs »

lighttower wrote: post #8

Chrome, on my system, is even more abusive. Watch the size of the .config/google-chrome directory and you'll see that it grows to multi-GB in the profile file. There is a Linux utility that takes care of all browsers' abuse of your ssd called profile sync daemon, PSD. It's available in the debian repo or [1] for Ubuntu or [2] for source. It uses `overlay` filesystem to direct all writes to ram and only syncs back to…

Anyone actually noticing a speedup from using this?

Re: Heavy SSD Writes from Firefox

#344

Post by rasz_pl »

daveguy wrote: post #316
rasz_pl wrote: post #304

Earlier quoted context omitted.

Times of samsung 830 using 27nm SLC are over, its Drives that reach >1PB or write endurance are no longer on the consumer market. You can buy them (SLC), but you wont after seeing the price. You can do nice tests that will last for hundreds of TBs .. until you pause for 24 hours and realize all the data is gone.

It was the 840 Pro that went 2.4 terabytes during a drive test that ended in 2015[0]. That was a 21nm tech. I think the endurance is better than you think. [0] http://techreport.com/review/27909/the-ssd-endurance-experim...

afair this test was run 24/7 with no power down time. Earlier test (the one with 830? I think he moved computer mid test and had to power everything down) showed drives look healthy until you let them sit offline for a while, then it turns out all data evaporated.

840 is especially broken and dramatically slows down when you dont touch the data for >month. The "fix" is firmware silently rewriting everything in the background burning thru the rewrite cycles.

Re: Heavy SSD Writes from Firefox

#345

Post by alphapapa »

acqq wrote: post #325
acqq wrote: post #265

Earlier quoted context omitted.

> regress startup performance No, for example, LZ4 is unbelievably fast: https://github.com/Cyan4973/lz4 almost 2 GB per second in decompression! I've just tried compressing some backupXX.session file (the biggest I've managed to find, just around 2 MB) and it compressed to 70% of the original, probably not enough to implement the compression -- and I suspect the reason is that the file contains too much base64 encod…

So "somebody" was me at the end, I've rolled up my sleeves and extracted the damn pictures: in my 1.7 MB session file approximately 1 MB were the base64 encodings of exactly 57 jpeg pictures that were all the result of the Google image search (probably of the single search). There were additionally a few pngs, representing "facebook, twitter, gplus" images and one "question mark sign" gif too.

Thank you for doing that. This explains much.

For a while now I have been running a cronjob to commit my profile's sessionstore-backups directory to a git repo every 5 minutes.

This is because, occasionally, when Firefox starts, it will load tabs in some kind of limbo state where the page is blank and the title is displayed in the tab--but if I refresh the page, it immediately goes to about:blank, and the original tab is completely lost.

When this happens, I can dig the tabs out of the recovery.js or previous.js files with a Python script that dumps the tabs out of the JSON, but if Firefox overwrites those files first, I can't. So the git repo lets me recover them no matter what.

What I have noticed is that the git repo grows huge very quickly. Right now my recovery.js file is 2.6 MB (which seems ridiculous for storing a list of URLs and title strings), but the git repo is 4.3 GB. If I run "git gc --aggressive" and wait a long time, it compresses down very well, to a few hundred MB.

But it's simply absurd how much data is being stored, and storing useless stuff like images in data URIs explains a lot.

Re: Heavy SSD Writes from Firefox

#346

Post by alphapapa »

sundarurfriend wrote: post #330
Yoric wrote: post #319

Earlier quoted context omitted.

Nobody is blaming the user for the 300Mb. It's just a factor we need to take in consideration. But yeah, storing favicon in Session Restore would be pretty bad. I didn't remember that it was the case, though.

As an Fx user, I'm glad that favicons are stored though. The icons are a much easier, quicker indicator of what the tab is, when I'm scrolling through dozens of tabs to see which one to click and load.

There's no need to store favicons in the session JSON. They're stored in the browser cache. If the cache gets cleared in the meantime, they can be redownloaded.

Re: Heavy SSD Writes from Firefox

#347

Post by alphapapa »

derefr wrote: post #242
cbhl wrote: post #198

Earlier quoted context omitted.

Until you have a NFS homedir...

On the other hand, SQLite with concurrent local and SSHFS-remote updates (where the SSHFS server is run with caching disabled) is entirely stable.

I'm curious, could you expand on this comment? Do you mean that you run Firefox from a profile loaded over SSHFS? I can only imagine that being unbearably slow because of I/O latency. And shouldn't disabling caching in SSHFS make it much, much worse?

Re: Heavy SSD Writes from Firefox

#348

Post by alphapapa »

Yoric wrote: post #156
nograpes wrote: post #130

Earlier quoted context omitted.

What would an updated version of Session Restore look like? I mean, what would be your vision for a new version?

I haven't worked on Session Restore in some time. But from the top of my head, it would be as follows. On the back-end, I believe that Session Restore should be backed by a database, with each tab updated independently, rather than a big bunch of JSON data. The rationale being that: - it's performance-critical; - it's safety-critical; - we have users with 300+ Mb of data in their Session Restore and JSON isn't meant…

Using a SQLite database for Session Restore seems to make sense. However, it would make it even more difficult to get data out of it when necessary. As ugly as a mess of JSON is, at least it's text, and I don't have to write a SQL query to grep for URLs.

Why not just use the filesystem? Imagine a layout like this:

    + profilename.default
    |--+ sessionrestore
       |--+ 
          |--+ window1
          |  |--+ tab1
          |  |  |--- cookies
          |  |  |--- formdata
          |  |  |--- title
          |  |  |--- url
          |  ---+ tab2   
          |     |--- cookies
          |     |--- formdata
          |     |--- title
          |     |--- url
          ---+ window2
             |--+ tab1
             |  |--- cookies
             |  |--- formdata
             |  |--- title
             |  |--- url
             ---+ tab2   
                |--- cookies
                |--- formdata
                |--- title
                |--- url
One directory per timestamp, containing one directory per window, containing one directory per tab, containing one file per data type. All plain-text. Change a tab, just write to that tab's files, and update the timestamp directory name. Every x minutes, make a new timestamp tree, keeping y old ones as backups.

No serializing and writing entire sessions to disk at once. Only small writes to small files. All plain-text, easy to read outside of the browser. Easy to copy, backup, modify, troubleshoot. No complex JSON, serializing, or database code. Use the write-to-temp-file-then-rename-over-existing-files paradigm to get atomic updates.

Need to know if part of a session on disk is stale? Check the mtime for that file, see if it's older than the last time that data was changed in memory.

Simple stuff. Straightforward. No overengineering. No bloat.

Why not do this?

Re: Heavy SSD Writes from Firefox

#349

Post by alphapapa »

db48x wrote: post #287
Yoric wrote: post #224

Earlier quoted context omitted.

I seem to remember that DOM Session Storage and Session Cookies are pretty big offenders. Other websites (e.g. Google Docs) use lots and lots and lots of hidden forms to remember lots and lots and lots of data. Also, people who keep Session Restore tabs open (you know, the tab that lets you restore your session, when you have crashed) and continue browsing – Firefox needs to store several nested Session Restore JSON…

I've always been impressed by that, because it's essentially bullet-proof.

I wish it were, but every now and then, I restart Firefox and get empty, title-only tabs that go to about:blank when I reload them, completely losing the tab. I have to use a Python script to dump the session data (which still appears in the JSON file, even though Firefox won't use it), eyeball the list to figure out which tab was broken, and reopen that URL manually.

Re: Heavy SSD Writes from Firefox

#350

Post by alphapapa »

pllbnk wrote: post #331
Yoric wrote: post #126

Hi, I’m one of the Firefox developers who was in charge of Session Restore, so I’m one of the culprits of this heavy SSD I/O. To make a long story short: we are aware of the problem, but fixing it for real requires completely re-architecturing Session Restore. That’s something we haven’t done yet, as Session Restore is rather safety-critical for many users, so this would need to be done very carefully, and with plent…

I think you are safe to rewrite it. It is not very reliable anyway as I have lost my tabs after crashes more than once. It might have been possible to save them but after a crash first thing I would do is restart Firefox and it would sometimes just completely overwrite the recovery.js. To mitigate this I have created a daemon to copy the file to a separate directory every few hours. I then delete those old versions m…

I have had the same problem. I finally made a git repo in the sessionrestore directory, setup a cronjob to commit it every 5 minutes, and then use this script to dump the session contents when necessary:

https://gist.github.com/alphapapa/bc7f3b25025fb99dad56

The git repo grows in size rapidly, but can be compressed way down. (Today it was 4.3 GB, but "git gc" compressed it to 230 MB). Every now and then you can blow away the repo or filter-branch to get rid of outdated sessions.

Post reply on HN