A lion, a head, and a dash of YAML

The Sphinx documentation tool provides tremendous extensibility and theming options; options which are massively underutilized. We demonstrate how you can go about writing your own extensions and themes for Sphinx and ultimately how you treat your docs like code.

If you work in open source, then it's increasingly likely that you've come across the Sphinx documentation tool. Originally written by and for the Python community, the past few years have seen an increasingly rapid adoption by non-Python projects. Recent adopters of Sphinx include the Linux kernel, Open vSwitch and the Dataplane Development Kit (DPDK), while other projects such as QEMU are preparing plans to migrate. Meanwhile, the Python ecosystem itself has only grown in strength and communities such as OpenStack have adopted the tool and have made extensive use of it.

Unfortunately, despite the widespread adoption and existence of multiple extensions, Sphinx is not always a very well understood tool. Extensions provide a pathway to greater functionality, for example, automatically generating documentation or generating docs in different output formats. Themes, meanwhile, provide a way to give a consistent, project-focused feel to your docs. The OpenStack community makes significant use of these extensibility features, which allow us to solve a number of problems in an automated manner.

- How can I avoid documenting a command line application twice: once in code and once in my docs?
- How can I take a large dataset and automatically convert this into multiple human-readable documents?
- How can I move documents about without breaking links?
- How can I automatically spell-check my documents?
- How can I style my docs to match my project website?

Through this talk we will demonstrate how each of the above problems, and how many more, can be solved in an easy and automated manner through the power of Sphinx extensions and themes. We also demonstrate how you can share your extensions and themes with the world, if you're so inclined

Presented at FOSDEM 2018

https://fosdem.org/2018/schedule/event/automating_documentation_with_sphinx_extensions/

Transcript

1. A lion, a head, and a dash of YAML Extending Sphinx to automate your documentation FOSDEM 2018 @stephenfin
2. reStructuredText, Docutils & Sphinx 1
3. A little reStructuredText ========================= This document demonstrates some basic features of |rst|. You can use **bold** and *italics*, along with ``literals``. It’s quite similar to `Markdown`_ but much more extensible. CommonMark may one day approach this [1]_, but today is not that day. `docutils`__ does all this for us. .. |rst| replace:: **reStructuredText** .. _Markdown: https://daringfireball.net/projects/markdown/ .. [1] https://talk.commonmark.org/t/444 .. __ http://docutils.sourceforge.net/
4. A little reStructuredText ========================= This document demonstrates some basic features of |rst|. You can use **bold** and *italics*, along with ``literals``. It’s quite similar to `Markdown`_ but much more extensible. CommonMark may one day approach this [1]_, but today is not that day. `docutils`__ does all this for us. .. |rst| replace:: **reStructuredText** .. _Markdown: https://daringfireball.net/projects/markdown/ .. [1] https://talk.commonmark.org/t/444 .. __ http://docutils.sourceforge.net/
5. A little reStructuredText This document demonstrates some basic features of reStructuredText. You can use bold and italics, along with literals. It’s quite similar to Markdown but much more extensible. CommonMark may one day approach this [1], but today is not that day. docutils does all this for us. [1] https://talk.commonmark.org/t/444/
6. A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.
7. A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.
8. A little more reStructuredText The extensibility really comes into play with directives and roles. We can do things like link to RFCs (RFC 2324, anyone?) or generate some more advanced formatting (I do love me some H 2 O). Warning The power can be intoxicating. Of course, all the stuff we showed previously still works!. The only limit is your imagination/interest.
9. reStructuredText provides the syntax Docutils provides the parsing
10. reStructuredText provides the syntax Docutils provides the parsing Sphinx provides the cross-referencing and file generation
11. Docutils use readers, parsers, transforms, and writers Docutils works with individual files
12. Docutils use readers, parsers, transforms, and writers Docutils works with individual files Sphinx uses readers, writers, transforms, and builders Sphinx works with multiple, cross-referenced files
13. Documentation tool Multiple output formats Extensive cross-referencing support Extensions
14. Documentation tool Multiple output formats Extensive cross-referencing support Extensions
15. sphinx-quickstart sphinx-build sphinx-apidoc sphinx-autogen
16. sphinx-quickstart sphinx-build sphinx-apidoc sphinx-autogen
17. Let’s Get To Extending... 2
18. Current version of Sphinx (1.7.0) - APIs may change Python knowledge is expected Some possible references to OpenStack projects See github.com/stephenfin/fosdem-sphinx-demo for more
19. Extensions are registered via sphinx.application.Application add_builder add_config_value add_domain add_event add_node add_directive add_role connect, disconnect ... (Builders) (Config Values) (Domains) (Events) (docutils Nodes) (Directives) (Interpreted Text Roles, a.k.a. Roles) (Hooks) ...
20. Extensions are registered via sphinx.application.Application add_builder add_config_value add_domain add_event add_node add_directive add_role connect, disconnect ... (Builders) (Config Values) (Domains) (Events) (docutils Nodes) (Directives) (Interpreted Text Roles, a.k.a. Roles) (Hooks) ...
21. Interpreted Text Roles 3 (a.k.a. roles)
22. A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.
23. A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.
24. def xyz_role(name, rawtext, text, lineno, inliner, options={}, content=[]): # code... def setup(app): app.add_role('xyz', xyz_role) return {'version': '1.0', 'parallel_read_safe': True}
25. Fixes ===== * #2951: Add ``--implicit-namespaces`` PEP-0420 support to apidoc. * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * #2471: Add config variable for default doctest flags. * Convert linkcheck builder to requests for better encoding handling * #2463, #2516: Add keywords of "meta" directive to search index source/changes.rst
26. Fixes ===== * #2951: Add ``--implicit-namespaces`` PEP-0420 support to apidoc. * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * #2471: Add config variable for default doctest flags. * Convert linkcheck builder to requests for better encoding handling * #2463, #2516: Add keywords of "meta" directive to search index source/changes.rst
27. Fixes ===== * #2951: Add ``--implicit-namespaces`` PEP-0420 support to apidoc. * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * #2471: Add config variable for default doctest flags. * Convert linkcheck builder to requests for better encoding handling * #2463, #2516: Add keywords of "meta" directive to search index source/changes.rst
28. Fixes ===== * Add ``--implicit-namespaces`` PEP-0420 support to apidoc (:ghissue:`2951`). * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * Add config variable for default doctest flags (:ghissue:`2471`). * Convert linkcheck builder to requests for better encoding handling * Add keywords of "meta" directive to search index (:ghissue:`2463`, :ghissue:`2516`) source/changes.rst
29. Fixes ===== * Add ``--implicit-namespaces`` PEP-0420 support to apidoc (:ghissue:`2951`). * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * Add config variable for default doctest flags (:ghissue:`2471`). * Convert linkcheck builder to requests for better encoding handling * Add keywords of "meta" directive to search index (:ghissue:`2463`, :ghissue:`2516`) source/changes.rst
30. from docutils import nodes BASE_URL = 'https://github.com/sphinx-doc/sphinx/issues/{}' def github_issue(name, rawtext, text, lineno, inliner, options={}, content=[]): refuri = BASE_URL.format(text) node = nodes.reference(rawtext, text, refuri=refuri, **options) return [node], [] def setup(app): app.add_role('ghissue', github_issue) return {'version': '1.0', 'parallel_read_safe': True} ext/issue_role.py
31. from docutils import nodes BASE_URL = 'https://github.com/sphinx-doc/sphinx/issues/{}' def github_issue(name, rawtext, text, lineno, inliner, options={}, content=[]): refuri = BASE_URL.format(text) node = nodes.reference(rawtext, text, refuri=refuri, **options) return [node], [] def setup(app): app.add_role('ghissue', github_issue) return {'version': '1.0', 'parallel_read_safe': True} ext/issue_role.py
32. Fixes ===== * Add ``--implicit-namespaces`` PEP-0420 support to apidoc (:ghissue:`2951`) * Add ``:caption:`` option for sphinx.ext.inheritance_diagram * Add config variable for default doctest flags (:ghissue:`2471`) * Convert linkcheck builder to requests for better encoding handling * Add keywords of "meta" directive to search index (:ghissue:`2463`, :ghissue:`2516`) source/changes.rst
33. Fixes • Add --implicit-namespaces PEP-0420 support to apidoc (2951) • Add :caption: option for sphinx.ext.inheritance_diagram • Add config variable for default doctest flags (2471) • Convert linkcheck builder to requests for better encoding handling • Add keywords of “meta” directive to search index (2463, 2516) build/changes.html
34. Directives 4
35. A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.
36. A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.
37. from docutils import nodes from docutils.parsers.rst import Directive class XYZDirective(Directive): def run(self): section = nodes.section(ids=['test']) section += nodes.title(text='Test') section += nodes.paragraph(text='Hello, world!') return [section] def setup(app): app.add_directive('xyz-directive', XYZDirective) return {'version': '1.0', 'parallel_read_safe': True}
38. Issues ====== Add keywords of "meta" directive to search index (#2463) -------------------------------------------------------- Opened by TimKam :: It would be great to have the keywords of `meta` directives included in the search index. Like this, one can help users who are searching for a synonym of the "correct" term through simply adding synonyms as keywords to a meta directive on the corresponding page. source/issues.rst
39. Issues ====== Add keywords of "meta" directive to search index (#2463) -------------------------------------------------------- Opened by TimKam :: It would be great to have the keywords of `meta` directives included in the search index. Like this, one can help users who are searching for a synonym of the "correct" term through simply adding synonyms as keywords to a meta directive on the corresponding page. source/issues.rst
40. Issues ====== .. github-issue:: 2463 source/issues.rst
41. from docutils import nodes from docutils.parsers.rst import Directive import requests URL = 'https://api.github.com/repos/sphinx-doc/sphinx/issues/{}' def get_issue(issue_id): issue = requests.get(URL.format(issue_id)).json() title = '%s (#%s)' % (issue['title'], issue_id) owner = 'Opened by %s' issue['user']['login'] return issue_id, title, issue['body'], owner ... ext/issue_directive.py
42. ... class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py
43. ... class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py
44. ... class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py
45. ... def setup(app): app.add_directive('github-issue', ShowGitHubIssue) return {'version': '1.0', 'parallel_read_safe': True} ext/issue_directive.py
46. Issues ====== .. github-issue:: 2463 source/issues.rst
47. Issues Add keywords of “meta” directive to search index (#2463) Opened by TimKam build/issues.html It would be great to have the keywords of `meta` directives included in the search index. Like this, one can help users who are searching for a synonym of the "correct" term through simply adding synonyms as keywords to a `meta` directive on the corresponding page.
48. class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py
49. class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py
50. class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) result = statemachine.ViewList() for line in format_issue(issue): result.append(line, '<' + __name__ + '>') node = nodes.section(document=self.state.document) nested_parse_with_titles(self.state, result, node) return node.children ext/issue_directive.py
51. class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) result = statemachine.ViewList() for line in format_issue(issue): result.append(line, '<' + __name__ + '>') node = nodes.section(document=self.state.document) nested_parse_with_titles(self.state, result, node) return node.children ext/issue_directive.py
52. def format_issue(issue): num, title, body, owner = issue yield title yield '=' * len(title) yield '' yield '%s ::' % owner yield '' for line in body.splitlines(): yield ' %s' % line if line else '' ext/issue_directive.py
53. Issues Add keywords of “meta” directive to search index (#2463) Opened by TimKam It would be great to have the keywords of `meta` directives included in the search index. Like this, one can help users who are searching for a synonym of the "correct" term through simply adding synonyms as keywords to a `meta` directive on the corresponding page. build/issues.html
54. Events 5
55. builder-inited(app) config-inited(app, config) source-read(app, docname, source) doctree-read(app, doctree) ...
56. from docutils import nodes from docutils.parsers.rst import Directive def builder_inited_handler(app): # code here... def setup(app): app.connect('builder-inited', builder_inited_handler)
57. issues.rst issue-2951.rst issue-2463.rst issue-2516.rst issue-2471.rst ...
58. from docutils import nodes from docutils.parsers.rst import Directive import requests URL = 'https://api.github.com/repos/sphinx-doc/sphinx/issues/' def get_issues(): issues = requests.get(URL).json() for issue in issues: title = '%s (#%s)' % (issue['title'], issue['number']) owner = 'Opened by %s' % issue['user']['login'] yield issue['number'], title, issue['body'], owner ext/issue_event.py
59. ... def generate_issue_docs(app): for num, title, body, owner in get_issues(): filename = os.path.join(app.srcdir, 'issues', '%s.rst' % num) with io.open(filename, 'w') as issue_doc: print(title, file=issue_doc) print('=' * len(title), file=issue_doc) print('', file=issue_doc) print('%s ::' % owner, file=issue_doc) print('', file=issue_doc) for line in body.splitlines(): print(' %s' % line if line else '', file=issue_doc) ext/issue_event.py
60. ... def setup(app): app.connect('builder-inited', generate_issue_docs) return {'version': '1.0', 'parallel_read_safe': True} ext/issue_event.py
61. Issues ====== .. toctree:: :maxdepth: 1 :glob: issues/* source/index.rst
62. Issues • Drop special support for rst2pdf (#4463) • Proposal: Integrate source_suffix and source_parsers (#4474) • [RFC] Implement delayed resolution in TOC (#4475) • Not possible to update individual ‘po’ files (#4476) • Build fails during eclim (aur) build: Babel data files not available (#4481) • Proposal: Allow to switch parsers on parsing document (#4482) • Integrate source suffix and source parsers (#4483) • … build/index.html
63. Enabling Your Extensions 5
64. import os import sys sys.path.insert(0, os.path.abspath('../ext')) extensions = [ 'issue_role', 'issue_directive', 'issue_event', ] source/conf.py
65. import os import sys sys.path.insert(0, os.path.abspath('../ext')) extensions = [ 'issue_role', 'issue_directive', 'issue_event', 'oslo_config.sphinxext', ] source/conf.py
66. Wrap Up 6
67. Extensions are registered via sphinx.application.Application add_builder add_config_value add_domain add_event add_node add_directive add_role connect, disconnect ... (Builders) (Config Values) (Domains) (Events) (docutils Nodes) (Directives) (Interpreted Text Roles, a.k.a. Roles) (Hooks) ...
68. Extensions are registered via sphinx.application.Application add_builder add_config_value add_domain add_event add_node add_directive add_role connect, disconnect ... (Builders) (Config Values) (Domains) (Events) (docutils Nodes) (Directives) (Interpreted Text Roles, a.k.a. Roles) (Hooks) ...
69. Extensions are registered via sphinx.application.Application Builder-specific Extensions (HTML themes, LaTeX templates, …) (Post) Transforms Translators Parsers Search languages ...
70. Fin
71. A lion, a head, and a dash of YAML Extending Sphinx to automate your documentation FOSDEM 2018 @stephenfin
72. References • Quick reStructuredText • Docutils Reference Guide ◦ reStructuredText Markup Specification ◦ reStructuredText Directives ◦ reStructuredText Interpreted Text Roles • Docutils How-Tos ◦ Creating reStructuredText Interpreted Text Roles ◦ Creating reStructuredText Directives • Docutils Hacker’s Guide • Sphinx Tutorial: Writing a simple extension
73. References • Defining Custom Roles in Sphinx -- Doug Hellmann • The Power of Sphinx - Integrating Jinja with RST -- Eric Holscher • Docutils Snippets -- Aurélien Gâteau • OpenStack + Sphinx In A Tree -- Stephen Finucane ( )