C++ modules questions

Our distributed build tool team is asking the following questions regarding C++ Modules. [My guesses in brackets. Please correct.]

  • Is the module creation mechanism (that fills the module cache) multi-process safe?

[yes]

  • If two or more compiler instances both try to create the same entry how is this arbitrated?

[O/S locks]

  • Does it allow for processes being terminated whilst creating cache entries? [I.e. are incomplete files cleaned up?]

[I don’t know]

Thanks.

-John

David, you've looked into this more recently than I have; can you
remember more details about our lock file mechanism?

Our distributed build tool team is asking the following questions regarding
C++ Modules. [My guesses in brackets. Please correct.]

- Is the module creation mechanism (that fills the module cache)
multi-process safe?

[yes]

Yes, it should be, in the sense that you should never see an invalid
PCM file. But IIRC it's racy and may perform redundant rebuilds. And
it's slow (see below).

- If two or more compiler instances both try to create the same entry how is
this arbitrated?

[O/S locks]

A lock file is used, and the waiting processes poll the filesystem
waiting for the resulting PCM file to appear. This polling starts by
waiting 1s and has exponential backoff. This leads to very high
overheads if modules need to be rebuilt, but no-one has fixed it to
use something more reasonable yet. We assume that file renames within
the module cache atomically create a complete destination file.

- Does it allow for processes being terminated whilst creating cache
entries? [I.e. are incomplete files cleaned up?]

[I don't know]

IIUC, no. Processes clean up after themselves, but there isn't a
mechanism for processes to clean up after other processes that died.

David, you've looked into this more recently than I have; can you
remember more details about our lock file mechanism?

Our distributed build tool team is asking the following questions regarding
C++ Modules. [My guesses in brackets. Please correct.]

- Is the module creation mechanism (that fills the module cache)
multi-process safe?

[yes]

Yes, it should be, in the sense that you should never see an invalid
PCM file. But IIRC it's racy and may perform redundant rebuilds. And
it's slow (see below).

- If two or more compiler instances both try to create the same entry how is
this arbitrated?

[O/S locks]

A lock file is used, and the waiting processes poll the filesystem
waiting for the resulting PCM file to appear. This polling starts by
waiting 1s and has exponential backoff. This leads to very high
overheads if modules need to be rebuilt, but no-one has fixed it to
use something more reasonable yet. We assume that file renames within
the module cache atomically create a complete destination file.

- Does it allow for processes being terminated whilst creating cache
entries? [I.e. are incomplete files cleaned up?]

[I don't know]

IIUC, no. Processes clean up after themselves, but there isn't a
mechanism for processes to clean up after other processes that died.

Actually we will delete lock files if the process that created them is no longer executing (we write the PID into the lock file). This is what causes the races that you mentioned, because we cannot tell that the file we delete is the same one we opened and determined to be ‘abandoned’. I don’t know of a safe way to fix this with our current approach without another layer of inter-process locking.

Ben