Here's one that's not super well-known: writes to a pipe are atomic as long as the write size is Anyone know if there's a similar guarantee for files?
[deleted]
Things Unix can do atomically (2010)
51–60 of 68 posts
Re: Things Unix can do atomically (2010)
#52msync() with MS_INVALIDATE doesn't belong on this list. It has nothing to do with atomic memory access. msync() is used when flushing a mapped file to durable storage. I very often see this mistake of conflating flushing caches with atomic access of memory. What's committed to durable storage has nothing to do with what multiple processes will see when mapping a file. All that's needed is the initial mmap to share a…
Right. Msync is an ordering barrier. Barriers go hand in hand with atomic operations, but they are really about visibility, not atomicity.
Re: Things Unix can do atomically (2010)
#53Earlier quoted context omitted.
Not really. It just means that it's indivisible (the original meaning of "atom"). Either it succeeds or fails, you never have to worry about it being half-finished. This includes actions which are so small they are literally indivisible, or actions which roll back to the original state if they fail.
Not just about it only completing or failing, but another observer in the system should never be able to find it in the half-way state. To everyone but the implementer of the atomic operation, it has no half-way states.
In the database terminology that's usually called "Isolation" to keep the concept separate from the rather restricted definition of atomicity.
Re: Things Unix can do atomically (2010)
#54I always thought it would be a good idea for system calls to support transactions. Probably in a limited way because implementing general transactions would require massive changes to the kernel. But it would be nice to be able to do [error checking omitted]: begin (); fp = fopen ("file", "w"); fputs (content, fp); fclose (fp); commit (); It could solve the whole thing with ending up with zero-length files because yo…
Windows has it, but not many people seem to use it: https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
Re: Things Unix can do atomically (2010)
#55I always thought it would be a good idea for system calls to support transactions. Probably in a limited way because implementing general transactions would require massive changes to the kernel. But it would be nice to be able to do [error checking omitted]: begin (); fp = fopen ("file", "w"); fputs (content, fp); fclose (fp); commit (); It could solve the whole thing with ending up with zero-length files because yo…
Windows has this.
Re: Things Unix can do atomically (2010)
#56msync() with MS_INVALIDATE doesn't belong on this list. It has nothing to do with atomic memory access. msync() is used when flushing a mapped file to durable storage. I very often see this mistake of conflating flushing caches with atomic access of memory. What's committed to durable storage has nothing to do with what multiple processes will see when mapping a file. All that's needed is the initial mmap to share a…
I haven't tested but I would expect MS_INVALIDATE on a large buffer to be much faster than filling it a word at a time with __sync_val_compare_and_swap (each causing its own bus transaction).
MS_INVALIDATE is likely a no-op on any modernish Unix, including Linux. It is there to accommodate old systems with non-coherent mapped files and page caches or even multiple mappings of the same file.
Re: Things Unix can do atomically (2010)
#57In a few simple words, can someone explain what does "atomically" mean? I personally used this term when talking about some Redis operations, but never knew the real gist of the word and concepts behind it. I have a very brief understanding of the term and if I'd have to explain it to a person, I'd say it's "the operation that does not have any side effects when performing its unit of work". Is my understanding even…
It is helpful to understand what problem it solves.
Lets say that we have a banking application that consists of a program which updates someones bank account by $Y every time it is called. Y is the command line parameter. The program's algorithm is like this :
1. Read the current balance amount to X
2. Add Y to X and store it in Z
3. Write Z to the database.
This program cannot be called by multiple processes at the same time. Lets say that it is payday, the account holder holds two jobs and each employer is trying to deposit $10 into someone's account, at the same time. Both these processes call the program with Y = $10. What happens ?
1. Process 1 reads the current balance ( $100 ) to X
2. Now, process 2 reads the current balance ( $100 ) to X
3. Process 1 adds 10 to X ( Z = 110 )
4. Process 2 adds 10 to X ( Z = 110 )
5. Process 1 writes the updated value to the database ( Z = 110 )
6. Process 2 writes the updated value to the database ( Z = 110 )
Now the account reflects a balance of $110, when it should have reflected $120. What we need is a guarantee from the system that some actions will not be parallelized ( i.e, they will be atomic ). From TFA it is given that "mkdir" is an atomic operation in UNIX ( i.e, only one process can create a directory at the same time ). You can write the program with the following logic
1. mkdir /tmp/lock_dir
2. If above step was unsuccessful sleep 10 seconds and go back to step 1
3. Read current account balance to X
4. Add Y to X and store it in Z
5. Write Z to database
6. Remove /tmp/lock_dir
Multiple processes can invoke this program simultaneously.
Re: Things Unix can do atomically (2010)
#58I always thought it would be a good idea for system calls to support transactions. Probably in a limited way because implementing general transactions would require massive changes to the kernel. But it would be nice to be able to do [error checking omitted]: begin (); fp = fopen ("file", "w"); fputs (content, fp); fclose (fp); commit (); It could solve the whole thing with ending up with zero-length files because yo…
Windows has it, but not many people seem to use it: https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
Beware, it appears to be deprecated:
"Microsoft strongly recommends developers utilize alternative means to achieve your application’s needs. Many scenarios that TxF was developed for can be achieved through simpler and more readily available techniques. Furthermore, TxF may not be available in future versions of Microsoft Windows."
Re: Things Unix can do atomically (2010)
#59Earlier quoted context omitted.
edit: I just realized you said "renaming." Original comment left below, but I edited before I get downvoted for a classic reading comprehension fail. Atomicity requires that the leakage mentioned shall not occur from any context aside from its own internal context. That makes your example somewhat of a simplification because these state transitions are visible to other processes. It is a common mistake to try to use…
> It is a common mistake to try to use files for locking, for example, instead of using the more robust flock(1). Why is this a mistake? It is my understanding that, if all the locking you need is a simple mutex, creating a file with a well-defined name with O_CREAT | O_EXCL is atomic -- the file will either be created or not (in which case the call will fail with EEXIST), and no two processes can possibly both succe…
The flock() method is preferable when you don't need to use NFS because as you say it'll automatically clean the lock up if the process holding it dies.
This gets rid of all the edge cases with stale locks in one fell swoop.
But as you point out if you want to do this e.g. over NFS you should create a file, but then you need to deal with stale locks.
If you can at all avoid that using flock() is generally better.
Re: Things Unix can do atomically (2010)
#60It should be noted that those filesystem operations are only atomic with respect to an observer running on the same operating system incarnation. Whether they are also atomic across power loss depends on the filesystem (though with a modern journaling filesystem, that generally should be the case).
That's what people would expect. But there was some drama around ext4, renames and fsync a few years ago.
You mean around truncation and rewrite?