Building the Docs Pipeline#

Warning

Be aware that the doc-builder was developed originally for Linux, although, in theory, you can run it on any platform (supporting either docker or windows), it’s only tested it on Linux. If you find any windows related issues, feel free to open an issue for that to review it.

Note

Recommendation: You can use the convenience script if you build the docs regularly, as it will not re-download the dependencies.

If you have a slow internet connection, you can use GitHub Codespaces since it will help you to build the docs faster since our script downloads large dependency files.

To build our docs, we use Sphinx. Sphinx is an extendable documentation generator for Python. As our building pipeline is complex, we heavily customize Sphinx using custom and third party extensions. As well as having a convenience script to build the docs.

How the doc-builder is being run#

There are 2 ways to build the docs:

Through a convenience script, which is useful for local development.
Through a Docker image, which is the recommended way.

We will go through how they work in the following sections.

The convenience script#

make_docs_without_docker.sh is a convenience script to build the docs, which can be found in the doc-builder repository. It takes one argument, the path to a project to document. The project should have the following characteristics:

It should have a requirements.txt, or alternatively a requirements folder, which includes a requirements.txt and an optional optional.txt file.
It can have an optional optional.txt file, if not the script will simply ignore it.
It should have a docs folder, which contains an index.rst file. This file is the root of the documentation.
It can contain an optional docs/prebuild.sh file, which will be executed before the docs are built. This is useful if you need to install some dependencies for the docs to build.
It can contain an optional docs/partial_conf.py which is a partial Sphinx configuration file. This file will be imported with the default conf.py file located in the doc-builder repo.

Running the script:

./make_docs_without_docker.sh /path/to/project

will result in the creation of documentation for the project in the directory docs/build.

Options#

-h, --help: Show this help
-C, --no-cleanup: Disable the backup/cleanup procedure
-g, --git-add: Stage changed files before generating the docs
-s, --skip-dependencies-install: Skip installing dependencies using pip
-j, --jobs N: Build in parallel with N processes where possible (special value auto will set N to cpu-count)
-D setting: Override a setting in conf.py

The Docker image#

The Docker image unifyai/doc-builder works as a wrapper around the make_docs_without_docker.sh script. It runs the script on the /project directory, located in the container as shown here:

./make_docs_without_docker.sh /project

To build the docs through docker you use this command:

docker run -v /path/to/project:/project unifyai/doc-builder

You can also add options described in the The convenience script section.

docker run -v /path/to/project:/project unifyai/doc-builder --no-cleanup

How Ivy’s docs is structured#

Looking at Ivy docs, we can see that it is structured like this:

docs
├── index.rst
├── partial_conf.py
├── prebuild.sh
├── overview
│   ├── background.rst
│   ├── ...
│   └── ...
└── ...

Let’s go through each of these files and folders.

`index.rst`#

This is the root of the documentation. It is the first file that Sphinx will read when building the docs. It is also the file that will be displayed when you open the docs in a browser.

Here is a segment of the file:

.. include:: ../README.rst

.. toctree::
    :hidden:
    :maxdepth: -1
    :caption: Overview

    overview/background.rst
    overview/design.rst
    overview/related_work.rst
    overview/extensions.rst
    overview/contributing.rst
    overview/deep_dive.rst
    overview/faq.rst
    overview/glossary.rst

.. autosummary::
    :toctree: docs/functional
    :template: top_functional_toc.rst
    :caption: API Reference
    :recursive:
    :hide-table:

    ivy.functional.ivy

You can see here different reStructuredText directives. The first one is include, which simply includes the main README file of the project, this is a good place if you want to make the rendered docs look different from the README, or simply include it as is.

The second directive is toctree, which is used to create a table of contents. The :hidden: option hides the table of contents from the rendered docs, only keeping it on the left side of the docs, not inline in the page itself. The :maxdepth: option is used to specify how deep the table of contents should go. The :caption: option is used to specify the title of the table of contents. The rest of the arguments are the files that should be included in the table of contents. Which in recursively points to every page in this documentation, for example this page is included in the toctree of overview/deep_dive.rst, which is included in the toctree of index.rst. You can read more about the toctree directive in sphinx docs, from now on we’ll only explain the directives that are custom to Ivy’s doc-builder.

The last directive is autosummary, which is used to automatically generate a table of contents for a module, as well as the documentation itself automatically by discovering the docstrings of the module. This is a custom directive, built on the original autosummary extension. We will explain in detail how did we change it, in Custom Extensions.

`partial_conf.py`#

This is a partial Sphinx configuration file. Which is being imported in the conf.py, it’s used to customize options that are specific to the project being documented. While importing common configurations such as the theme, the extensions, etc in the original conf.py.

This is a part of partial_conf.py:

ivy_toctree_caption_map = {
    "ivy.functional.ivy": "Functions",
    "ivy.stateful": "Framework classes",
    "ivy.nested_array": "Nested array",
    "ivy.utils": "Utils",
    "ivy_tests.test_ivy.helpers": "Testing",
}

Here we are overriding the ivy_toctree_caption_map configuration, which is used to customize the title of the table of contents for each module. ivy_toctree_caption_map is one of the configuration options we have in our custom_autosummary extension, which will be covered extensively in Custom Extensions.

`prebuild.sh`#

This is an optional file, which is executed before the docs are built. This is useful if you need to install some dependencies for the docs to build. In Ivy’s case, we install torch then torch-scatter sequentially to avoid a bug in torch-scatter’s setup. And if we want to make any changes to the docker container before building the docs.

Custom Extensions#

As of writing this documentation, Ivy’s doc-builder is using 4 custom extensions:

custom_autosummary
discussion_linker
skippable_function
ivy_data

`custom_autosummary`#

This extension is a modified version of the original autosummary, which is used to discover and automatically document the docstrings of a module. This is done by generating “stub” rst files for each module listed in the autosummary directive, you can add a template for these stub files using the :template: option. Which can in turn include the autosummary directive again, recursing on the whole module.

Unfortunately, the original autosummary extension is very limited, forcing you to have a table of contents for each modules.

We’ll go through each option or configuration value added to the original autosummary

`:hide-table:`#

As the name suggests, the original behavior of autosummary is to generate a table of contents for each module. And it generates stub files only if the :toctree: option is specified. As we only need the toctree this option hides the table of contents, but it requires the :toctree: option to be specified.

`discussion_linker`#

Discussion linker is a simple extension that adds a link to our discord server, as well as specific discussion boards for each modules.

The directive is included like this:

.. discussion-links:: module.foo

First it will look for the discussion_channel_map configuration, in Ivy it looks like this:

discussion_channel_map = {
    ...,
    "ivy.functional.ivy.creation": ["1000043690254946374"],
    "ivy.functional.ivy.data_type": ["1000043749088436315"],
    ...,
}

The key is the module name, if it’s not found the discussion-link directive will render an empty node. The first and only value in the list is the channel id of the module, it is in a list as we used to have forums as well but they are removed now.

The output string is generated by a series of replaces on template strings, which are customizable using the config. To understand how it works, let’s look at the default configurations and their values:

discussion_paragraph: "This should have hopefully given you an overview of the {{submodule}} submodule, if you have any questions, please feel free to reach out on our [discord]({{discord_link}}) in the [{{submodule}} channel]({{channel_link}})!"
discord_link: "https://discord.gg/uYRmyPxMQq"
channel_link: "https://discord.com/channels/799879767196958751/{{channel_id}}"

Here is an example of how it works for ivy.functional.ivy.creation:

First we resolve the {{submodule}} template string, which is the last part of the module name, in this case it’s creation.

The result will be like this:

This should have hopefully given you an overview of the creation submodule, if you have any questions, please feel free to reach out on our [discord]({{discord_link}})!
Then we resolve the {{discord_link}} template string.

The result will be like this:

This should have hopefully given you an overview of the creation submodule, if you have any questions, please feel free to reach out on our [discord](https://discord.gg/uYRmyPxMQq)!
Then we resolve the {{channel_link}} template string.

The result will be like this:

This should have hopefully given you an overview of the creation submodule, if you have any questions, please feel free to reach out on our [discord](https://discord.gg/uYRmyPxMQq)!
We finally resolve {{channel_id}} template strings.

The result will be like this:

This should have hopefully given you an overview of the creation submodule, if you have any questions, please feel free to reach out on our [discord](https://discord.gg/uYRmyPxMQq)!
After that we render the node paragraph as if it’s a Markdown text resulting this:

This should have hopefully given you an overview of the creation submodule, if you have any questions, please feel free to reach out on our discord!

All of the above template strings can be customized using the configuration, so feel free to change them to your liking.

`skippable_function`#

This extension provides a custom auto documenter autoskippablemethod that skip functions that match values in skippable_method_attributes configuration.

This is an example of skippable_method_attributes configuration in partial_conf.py:

skippable_method_attributes = [
    {
        "__qualname__": "_wrap_function.<locals>.new_function"
    }
]

This will remove any function that has __qualname__ attribute equal to _wrap_function.<locals>.new_function.

`ivy_data`#

This is a custom documenter for autodoc that documents Ivy data attributes that live in ivy.functional.ivy, it will replace the module to ivy. instead of ivy.functional.ivy.<submodule>.

It’s used instead of simply using ivy.<data attribute> because data attributes have no __doc__ attribute, instead docs are discovered by parsing the source code itself. So for Sphinx to find the required docs, it needs to be supplied the full module name, then using the autoivydata directive will replace the module name to ivy..

Please refer to the auto documenter guide in sphinx documentation for more info.

Building the Docs Pipeline#

How the doc-builder is being run#

The convenience script#

Options#

The Docker image#

How Ivy’s docs is structured#

index.rst#

partial_conf.py#

prebuild.sh#

Custom Extensions#

custom_autosummary#

:hide-table:#

discussion_linker#

skippable_function#

ivy_data#