clang-tools-extra sphinx documentation was last updated a month ago

Hi,

I’ve recently noticed that files in http://clang.llvm.org/extra/clang-tidy/checks/ (and the rest of clang-tools-extra sphinx documents) had last been built on July 18 (by Sphinx 1.1.3). The rest of clang docs seem to build normally (using Sphinx 1.4.5), which makes me think that the issue might have something to do with the upgrade of sphinx. I’ve updated it locally and the issues should be fixed now in r279049.

The clang-sphinx-docs buildbot was green all this month (e.g. http://lab.llvm.org:8011/builders/clang-sphinx-docs/builds/15762), but I don’t know 1. whether it builds clang-tools-extra docs and 2. which version of Sphinx (and Pygments) it uses.

So the questions:

  1. Tanya, can we make the process of building docs for llvm.org more transparent? In the form of a status dashboard, logs of the last build or e-mail notifications, for example.
  2. Dmitri, can you update Sphinx on the clang-sphinx-docs buildbot to the same version that is used to build docs for llvm.org?
  3. Dmitri, could you include clang-tools-extra docs to the build steps on the clang-sphinx-docs buildbot?
  4. Can we provide a better coordination between documentation format and tools versions? Possible options are:
    a. Move in the direction of hermetic documentation builds. For instance, put a private copy of Sphinx to LLVM sources somewhere in llvm/utils/ and fix the documentation files when updating Sphinx.
    b. Strictly require a certain version of Sphinx when building documentation. Only works if there is a sane and reliable way to get a specific version of the package on every platform.
    c. At least, coordinate Sphinx updates on all docs buildbots and whatever is used to build docs for llvm.org.

Thanks!

– Alex

  2. Dmitri, can you update Sphinx on the clang-sphinx-docs buildbot to the
same version that is used to build docs for llvm.org?

Hi Alexander,

I'm running python-sphinx 1.4.5-1 from Debian unstable, this is the
latest version available in the repository. Downgrading the bot to
1.1.3 seems to be non-trivial (might require adding a Python
virtualenv or something like that).

  3. Dmitri, could you include clang-tools-extra docs to the build steps on
the clang-sphinx-docs buildbot?

I would not object to such a change, but I don't have the bandwidth to
make it right now. I would appreciate if someone submitted a patch.

Dmitri

We should be updating the website to a more modern Sphinx, not the
other way around.

I've just updated it for (what I thought was all docs), but it seems
clang-tools-extra has a different virtualenv.

I can try to update that, if I find the right place.

cheers,
--renato

> 2. Dmitri, can you update Sphinx on the clang-sphinx-docs buildbot to
the
> same version that is used to build docs for llvm.org?

Hi Alexander,

I'm running python-sphinx 1.4.5-1 from Debian unstable, this is the
latest version available in the repository. Downgrading the bot to
1.1.3 seems to be non-trivial (might require adding a Python
virtualenv or something like that).

We don't need to downgrade anything. The 1.1.3 was used to build llvm.org
docs _before_ the upgrade, now it's using 1.4.5 as well, so this is already
consistent. Nothing is needed here at this point.

> 3. Dmitri, could you include clang-tools-extra docs to the build steps
on
> the clang-sphinx-docs buildbot?

I would not object to such a change, but I don't have the bandwidth to
make it right now. I would appreciate if someone submitted a patch.

Where's the buildbot config? I can try to add the step.

Sorry for being unclear, 1.1.3 was the version that was able to build clang-tools-extra in July. And once Sphinx was upgraded to 1.4.5, the clang-tools-extra build started failing until I’ve fixed issues in r279049. Now http://clang.llvm.org/extra/ says “Created using Sphinx 1.4.5”, which is fine and consistent with the current setup of the clang-sphinx-docs buildbot, according to Dmitri.

The important question here is “can we make the process of building docs for llvm.org more transparent? In the form of a status dashboard, logs of the last build or e-mail notifications, for example.”. Is it something you can help with?

Sorry for being unclear, 1.1.3 _was_ the version that was able to build
clang-tools-extra in July. And once Sphinx was upgraded to 1.4.5, the
clang-tools-extra build started failing until I've fixed issues in r279049.
Now http://clang.llvm.org/extra/ says "Created using Sphinx 1.4.5", which is
fine and consistent with the current setup of the clang-sphinx-docs
buildbot, according to Dmitri.

Right, so my migration worked across the board (as I thought it
would). Good to know. :slight_smile:

The important question here is "can we make the process of building docs for
llvm.org more transparent? In the form of a status dashboard, logs of the
last build or e-mail notifications, for example.". Is it something you can
help with?

It could. Right now there are two processes:

1. A cron job in the LLVM server pulling the docs, building with
Sphinx and if all goes well, an rsync to the WWW directory.

2. Dmitry's buildbots, which do something similar, but doesn't copy
anything anywhere.

Making the buildbot push the docs to the server will involve some
SSH/FTP magic, which is doable, but non-trivial to make it secure.
WebDAV may work, but that's less trivial to setup.

Putting the buildbot in the LLVM server could be another solution, but
it would also increase the complexity, which is already non-trivial.

Tanya is moving all services out of that server and that'll help us
have better granularity and probably my second proposal would work
best.

But in the meantime, we made sure both bots and server have the same
Sphinx, so it should be mostly ok to rely on the buildbot.

cheers,
--renato

Sorry for being unclear, 1.1.3 _was_ the version that was able to build
clang-tools-extra in July. And once Sphinx was upgraded to 1.4.5, the
clang-tools-extra build started failing until I've fixed issues in r279049.
Now http://clang.llvm.org/extra/ says "Created using Sphinx 1.4.5", which is
fine and consistent with the current setup of the clang-sphinx-docs
buildbot, according to Dmitri.

Right, so my migration worked across the board (as I thought it
would). Good to know. :slight_smile:

The important question here is "can we make the process of building docs for
llvm.org more transparent? In the form of a status dashboard, logs of the
last build or e-mail notifications, for example.". Is it something you can
help with?

It could. Right now there are two processes:

1. A cron job in the LLVM server pulling the docs, building with
Sphinx and if all goes well, an rsync to the WWW directory.

2. Dmitry's buildbots, which do something similar, but doesn't copy
anything anywhere.

Making the buildbot push the docs to the server will involve some
SSH/FTP magic, which is doable, but non-trivial to make it secure.
WebDAV may work, but that's less trivial to setup.

Putting the buildbot in the LLVM server could be another solution, but
it would also increase the complexity, which is already non-trivial.

Tanya is moving all services out of that server and that'll help us
have better granularity and probably my second proposal would work
best.

But in the meantime, we made sure both bots and server have the same
Sphinx, so it should be mostly ok to rely on the buildbot.

I do not see a need to use another server. Having some sort of status page is an option. Email notification (which we use for other doc builds) would be too frequent.

I realize I am a bottleneck so I will be putting into place a way to distribute the work load.

/Tanya

> 2. Dmitri, can you update Sphinx on the clang-sphinx-docs buildbot to
the
> same version that is used to build docs for llvm.org?

Hi Alexander,

I'm running python-sphinx 1.4.5-1 from Debian unstable, this is the
latest version available in the repository. Downgrading the bot to
1.1.3 seems to be non-trivial (might require adding a Python
virtualenv or something like that).

We don't need to downgrade anything. The 1.1.3 was used to build llvm.org
docs _before_ the upgrade, now it's using 1.4.5 as well, so this is already
consistent. Nothing is needed here at this point.

> 3. Dmitri, could you include clang-tools-extra docs to the build
steps on
> the clang-sphinx-docs buildbot?

I would not object to such a change, but I don't have the bandwidth to
make it right now. I would appreciate if someone submitted a patch.

Where's the buildbot config? I can try to add the step.

I found something relevant. Here's the patch:
https://reviews.llvm.org/D23787