Live data from Hacker News

Atuin replaces your existing shell history with a SQLite database

github.com

111–120 of 198 posts

Re: Atuin replaces your existing shell history with a SQLite database

#111
post #98

Earlier quoted context omitted.

Disk space is the least of the problems garbage entries in command history cause. It's not just commands that are actual for the current session (like ls/cd/cat), it's also incorrect commands or just commands not worth retaining still being saved among with the useful commands. The most precious is user's time. And when you fzf part of the command with the regularly saved history and would like to re-execute somethin…

A typical command I run is never going to be something I look up again, so I would prefer to optimize for writing instead of reading. Dumping every command to a file adds no friction to my regular work, while attempting to categorize garbage commands would add a lot of friction. Also, when I do want to reference my deep history I often find that seeing the full list of what I was doing is helpful at getting myself ba…

> A typical command I run is never going to be something I look up again, so I would prefer to optimize for writing instead of reading. Dumping every command to a file adds no friction to my regular work

Then why write history at all? Just discard it.

Re: Atuin replaces your existing shell history with a SQLite database

#112

For people who have tried this, how good is the latency from when I press ctrl-r [first letter] to when I start seeing results? If it's not instant it's just going to frustrate me.

Maybe ctrl+r stress uses history command.

Re: Atuin replaces your existing shell history with a SQLite database

#113

For people who have tried this, how good is the latency from when I press ctrl-r [first letter] to when I start seeing results? If it's not instant it's just going to frustrate me.

I notice 0 latency (except when using the opt-in skim feature - we haven't properly optimised that yet). Latency is very important for us too. Granted, if you don't use an SSD, then you might encounter some startup lag

Re: Atuin replaces your existing shell history with a SQLite database

#114

It looks like it's not possible to export the history to a format the shell can import[0], so if I wanted to try this out, my commands would be locked there. It looks interesting, so if I'm overlooking a way to do this with fish, I'd experiment with it. [0]: https://github.com/ellie/atuin/issues/816

It’d be pretty trivial to do this yourself though, shell history files are very simple

Re: Atuin replaces your existing shell history with a SQLite database

#115
First of all, great name.

Second, I am in awe of how good your documentation is and how well you communicate about atuin to the world at large.

Does Atuin offer any features to toggle the capture of commands into its DB? Being able to opt-in or opt-out of Atuin history on a per-command basis would be pretty useful, especially because there is also the atuin sync feature.

I usually work with sensitive information inside a tmux session because, in the default bash configuration, most commands run in tmux never make it into bash history (I believe the last pane to exit is the only one that does make it). It seems I would have to manually go in and drop rows from the DB if I set up Atuin.

One of my products, bugout, has a command called "bugout trap". Not trying to push bugout here, but thought Atuin might benefit from some of the lessons we learned:

1. Because bugout trap is opt-in (you have to explicitly prefix your command with "bugout trap --", it also allows users to specify tags to make classifying commands easy. This is really useful for search - e.g. you can use queries like "!#exit:0 #db #migration #prod" to find all unsuccessful database migrations you attempted in your production environment.

2. bugout trap has a --env flag which gives users the option of pushing their environment variables into their history. This is really useful for programs that use a lot of environment variables. The safest way to use this is to first trap commands into your personal knowledge base with --env, then remove or redact any sensitive information, and only then share (in case you want to share with a team).

3. We thought that sharing would be useful for teams to build documentation on top of. Even we ourselves have very little adoption of that use case internally. We use it to keep a record of programs we run in our production environment (especially database migrations).

4. bugout trap also stores data that a program returns from stdout and stderr - this has been INCREDIBLY useful. I do want to add a mode that makes the capture of output optional, though, as currently bugout trap is unusable with things like server start commands which run continuously.

5. In general, I have found that command line history is very personal and private for developers so collaborative features are going to rightly be seen with skepticism.

Hope that helps anyone building similar tools.

    $ bugout trap --help
    Wraps a command, waits for it to complete, and then adds the result to a Bugout journal.

    Specify the wrapped command using "--" followed by the command:
            bugout trap [flags] -- 

    Usage:
    bugout trap [flags]

    Flags:
    -e, --env              Set this flag to dump the values of your current environment variables
    -h, --help             help for trap
    -j, --journal string   ID of journal
        --tags strings     Tags to apply to the new entry (as a comma-separated list of strings)
    -T, --title string     Title of new entry
    -t, --token string     Bugout access token to use for the request

Re: Atuin replaces your existing shell history with a SQLite database

#116

Earlier quoted context omitted.

This is also sub-optimal, as it causes another problem: some of the commands are part of a bigger sequence (the most import property here is that items inside sequences are ordered, the order of commands matters!), so by blindly deleting duplicates - you break sequences. In SQL there are sessions and transactions. In shell history - we don't have such entities and this sucks. One could configure their bash/zsh to sav…

Oh, we don't actually delete anything - just deduplicate for search. Sequential context is something we're building very soon. The idea: you search for "cd /project/dir", press TAB and it opens up a new pane in the tui. This will show the command +/- 10 commands. You can then navigate back in time. This could indeed be useful for managing that one setup command you always have to run in this project dir but never rem…

> Oh, we don't actually delete anything - just deduplicate for search.

Good to hear, but the point stands: so you deduplicate only for the view, not in the source, and thus the source remains contaminated with duplicates (at very least they cost some disk space and increase seek time).

As for the view: so, since you deduplicate the commands - you can't lookup the context (commands executed before & after)! Because each time the now deduplicated command was executed in the past - it had its own context!

Re: Atuin replaces your existing shell history with a SQLite database

#117
post #98

Earlier quoted context omitted.

A typical command I run is never going to be something I look up again, so I would prefer to optimize for writing instead of reading. Dumping every command to a file adds no friction to my regular work, while attempting to categorize garbage commands would add a lot of friction. Also, when I do want to reference my deep history I often find that seeing the full list of what I was doing is helpful at getting myself ba…

> A typical command I run is never going to be something I look up again, so I would prefer to optimize for writing instead of reading. Dumping every command to a file adds no friction to my regular work Then why write history at all? Just discard it.

Because even with the friction of my history not having been pruned or otherwise tidied it's still a really valuable resource when I do need it!

Re: Atuin replaces your existing shell history with a SQLite database

#118
post #59

Earlier quoted context omitted.

At 0.5GB that's 5k per entry -- what are you storing for each? For comparison, my (non-work) history since 2012 (plain text) is 181k entries, and takes 25MB. I store the command along with when and where you ran it. ( https://www.jefftk.com/p/logging-shell-history-in-zsh )

Ah, a fellow packrat! I have every command I ever typed into a shell since around 2005, and my history weighs in at 1 CD or 650MB (as of a couple of years ago) I'm probably being wasteful of space because I store each session in a separate file. I used to do a lot of data analysis at the shell back in the day, and found it useful to audit sequences of commands afterwards for mistakes, or to turn them into scripts.

This is so insane that I love it. Do you also save your belly button lint since 2005? Or nail clippings? :)

Re: Atuin replaces your existing shell history with a SQLite database

#119
post #95

Earlier quoted context omitted.

For 100k entries you can grep them instantaneously; there's no need to maintain an index.

Grep only works if you want an exact string match. If you want to find words out of order or support features like stemming, fts is necessary.

With a small enough corpus, full text search does not require an index to be instantaneous, and 100k entries is easily small enough for that.

Additionally, everything you describe can be phrased as a regular expression.

Re: Atuin replaces your existing shell history with a SQLite database

#120
post #59

Earlier quoted context omitted.

At 0.5GB that's 5k per entry -- what are you storing for each? For comparison, my (non-work) history since 2012 (plain text) is 181k entries, and takes 25MB. I store the command along with when and where you ran it. ( https://www.jefftk.com/p/logging-shell-history-in-zsh )

Ah, a fellow packrat! I have every command I ever typed into a shell since around 2005, and my history weighs in at 1 CD or 650MB (as of a couple of years ago) I'm probably being wasteful of space because I store each session in a separate file. I used to do a lot of data analysis at the shell back in the day, and found it useful to audit sequences of commands afterwards for mistakes, or to turn them into scripts.

Do you regularly back up your history?
Post reply on HN