Can we do atomic write to a file by using raw_fd_ostream?

Hi all,

In a pass I’m using raw_fd_ostream to write a string to a file. Does raw_fd_ostream guarantee
the write is atomic when I do parallel compilation (currently I’m using -j8)? I have some errs() to
print information for debugging purposes and I saw that the printed information gets messed up sometime.
If I’m writing a string with the format of “A:B:C”, is it possible that I got “A1:B2:C1” and “A2:B1:C2”
instead of the correct one “A1:B1:C1” and “A2:B2:C2”? And if this is possible, how can I do atomic write to a file?

Thanks,

  • Jie

Hi Jie,

In a pass I’m using raw_fd_ostream to write a string to a file. Does raw_fd_ostream guarantee
the write is atomic when I do parallel compilation (currently I’m using -j8)?

raw_fd_ostream doesn't really attempt to solve this problem. All
writes get filtered through the write_impl function which calls POSIX
::write in chunks (large ones -- 1GB). Additionally, individual calls
to << operators may or may not be buffered, and if they are it's with
a fixed size buffer rather than a transaction based system.

According to Is file append atomic in UNIX? - Stack Overflow,
those individual calls to ::write are atomic and won't interfere
provided the file has been opened in append mode. So your goal should
be to ensure that each logging event gets implemented as a single call

The most straightforward way would obviously be to do it yourself. In
LLVM you might do your normal writing to a raw_string_ostream and then
call ::write once on that.

Cheers.

Tim.

The other solution would be to use ninja instead of make if you can. Ninja always buffers command output so that error messages from a parallel build don’t interleave. https://ninja-build.org/manual.html#_comparison_to_make