A Lion, a Head, and a Dash of YAML (PyCon Limerick 2020)

Presented at [PyCon Limerick 2020](https://python.ie/pycon-limerick-2020/).

Transcript

1. A lion, a head, and a dash of YAML Extending Sphinx to automate your documentation PyCon Limerick 2020 @stephenfin
2. reStructuredText, Docutils & Sphinx 1
3. A little reStructuredText ========================= This document demonstrates some basic features of |rst|. You can use **bold** and *italics*, along with ``literals``. It’s quite similar to `Markdown`_ but much more extensible. CommonMark may one day approach this [1]_, but today is not that day. `docutils`__ does all this for us. .. |rst| replace:: **reStructuredText** .. _Markdown: https://daringfireball.net/projects/markdown/ .. [1] https://talk.commonmark.org/t/444 .. __ http://docutils.sourceforge.net/
4. A little reStructuredText ========================= This document demonstrates some basic features of |rst|. You can use **bold** and *italics*, along with ``literals``. It’s quite similar to `Markdown`_ but much more extensible. CommonMark may one day approach this [1]_, but today is not that day. `docutils`__ does all this for us. .. |rst| replace:: **reStructuredText** .. _Markdown: https://daringfireball.net/projects/markdown/ .. [1] https://talk.commonmark.org/t/444 .. __ http://docutils.sourceforge.net/
5. A little reStructuredText This document demonstrates some basic features of reStructuredText. You can use bold and italics, along with literals. It’s quite similar to Markdown but much more extensible. CommonMark may one day approach this [1], but today is not that day. docutils does all this for us. [1] https://talk.commonmark.org/t/444/
6. A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.
7. A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.
8. A little more reStructuredText The extensibility really comes into play with directives and roles. We can do things like link to RFCs (RFC 2324, anyone?) or generate some more advanced formatting (I do love me some H 2 O). Warning The power can be intoxicating. Of course, all the stuff we showed previously still works!. The only limit is your imagination/interest.
9. reStructuredText provides the syntax Docutils provides the parsing
10. reStructuredText provides the syntax Docutils provides the parsing Sphinx provides the cross-referencing and file generation
11. Docutils use readers, parsers, transforms, and writers Docutils works with individual files
12. Docutils use readers, parsers, transforms, and writers Docutils works with individual files Sphinx uses readers, writers, transforms, and builders Sphinx works with multiple, cross-referenced files
13. Documentation tool Multiple output formats Extensive cross-referencing support Extensions
14. Documentation tool Multiple output formats Extensive cross-referencing support Extensions
15. sphinx-quickstart sphinx-build sphinx-apidoc sphinx-autogen
16. sphinx-quickstart sphinx-build sphinx-apidoc sphinx-autogen
17. Let’s Get To Extending... 2
18. Current version of Sphinx (2.4.0) - APIs may change Python knowledge is expected Some possible references to OpenStack projects See github.com/stephenfin/fosdem-sphinx-demo for more
19. Extensions are registered via sphinx.application.Sphinx add_builder add_config_value add_domain add_event add_node add_directive add_role connect, disconnect ... (Builders) (Config Values) (Domains) (Events) (docutils Nodes) (Directives) (Interpreted Text Roles, a.k.a. Roles) (Hooks) ...
20. Extensions are registered via sphinx.application.Sphinx add_builder add_config_value add_domain add_event add_node add_directive add_role connect, disconnect ... (Builders) (Config Values) (Domains) (Events) (docutils Nodes) (Directives) (Interpreted Text Roles, a.k.a. Roles) (Hooks) ...
21. Interpreted Text Roles 3 (a.k.a. roles)
22. A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.
23. A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.
24. def xyz_role(name, rawtext, text, lineno, inliner, options={}, content=[]): # code... def setup(app): app.add_role('xyz', xyz_role) return {'version': '1.0', 'parallel_read_safe': True}
25. Fixes ===== * #2951: Add ``--implicit-namespaces`` PEP-0420 support to apidoc. * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * #2471: Add config variable for default doctest flags. * Convert linkcheck builder to requests for better encoding handling * #2463, #2516: Add keywords of "meta" directive to search index source/changes.rst
26. Fixes ===== * #2951: Add ``--implicit-namespaces`` PEP-0420 support to apidoc. * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * #2471: Add config variable for default doctest flags. * Convert linkcheck builder to requests for better encoding handling * #2463, #2516: Add keywords of "meta" directive to search index source/changes.rst
27. Fixes ===== * #2951: Add ``--implicit-namespaces`` PEP-0420 support to apidoc. * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * #2471: Add config variable for default doctest flags. * Convert linkcheck builder to requests for better encoding handling * #2463, #2516: Add keywords of "meta" directive to search index source/changes.rst
28. Fixes ===== * Add ``--implicit-namespaces`` PEP-0420 support to apidoc (:ghissue:`2951`). * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * Add config variable for default doctest flags (:ghissue:`2471`). * Convert linkcheck builder to requests for better encoding handling * Add keywords of "meta" directive to search index (:ghissue:`2463`, :ghissue:`2516`) source/changes.rst
29. Fixes ===== * Add ``--implicit-namespaces`` PEP-0420 support to apidoc (:ghissue:`2951`). * Add ``:caption:`` option for sphinx.ext.inheritance_diagram. * Add config variable for default doctest flags (:ghissue:`2471`). * Convert linkcheck builder to requests for better encoding handling * Add keywords of "meta" directive to search index (:ghissue:`2463`, :ghissue:`2516`) source/changes.rst
30. from docutils import nodes BASE_URL = 'https://github.com/sphinx-doc/sphinx/issues/{}' def github_issue(name, rawtext, text, lineno, inliner, options={}, content=[]): refuri = BASE_URL.format(text) node = nodes.reference(rawtext, text, refuri=refuri, **options) return [node], [] def setup(app): app.add_role('ghissue', github_issue) return {'version': '1.0', 'parallel_read_safe': True} ext/issue_role.py
31. from docutils import nodes BASE_URL = 'https://github.com/sphinx-doc/sphinx/issues/{}' def github_issue(name, rawtext, text, lineno, inliner, options={}, content=[]): refuri = BASE_URL.format(text) node = nodes.reference(rawtext, text, refuri=refuri, **options) return [node], [] def setup(app): app.add_role('ghissue', github_issue) return {'version': '1.0', 'parallel_read_safe': True} ext/issue_role.py
32. Fixes ===== * Add ``--implicit-namespaces`` PEP-0420 support to apidoc (:ghissue:`2951`) * Add ``:caption:`` option for sphinx.ext.inheritance_diagram * Add config variable for default doctest flags (:ghissue:`2471`) * Convert linkcheck builder to requests for better encoding handling * Add keywords of "meta" directive to search index (:ghissue:`2463`, :ghissue:`2516`) source/changes.rst
33. Fixes • Add --implicit-namespaces PEP-0420 support to apidoc (2951) • Add :caption: option for sphinx.ext.inheritance_diagram • Add config variable for default doctest flags (2471) • Convert linkcheck builder to requests for better encoding handling • Add keywords of “meta” directive to search index (2463, 2516) build/changes.html
34. Directives 4
35. A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.
36. A little more reStructuredText ============================== The extensibility really comes into play with directives and roles. We can do things like link to RFCs (:RFC:`2324`, anyone?) or generate some more advanced formatting (I do love me some H\ :sub:`2`\ O). .. warning:: The power can be intoxicating. Of course, all the stuff we showed previously *still works!*. The only limit is your imagination/interest.
37. from docutils import nodes from docutils.parsers.rst import Directive class XYZDirective(Directive): def run(self): section = nodes.section(ids=['test']) section += nodes.title(text='Test') section += nodes.paragraph(text='Hello, world!') return [section] def setup(app): app.add_directive('xyz-directive', XYZDirective) return {'version': '1.0', 'parallel_read_safe': True}
38. Issues ====== Add keywords of "meta" directive to search index (#2463) -------------------------------------------------------- Opened by TimKam :: It would be great to have the keywords of `meta` directives included in the search index. Like this, one can help users who are searching for a synonym of the "correct" term through simply adding synonyms as keywords to a meta directive on the corresponding page. source/issues.rst
39. Issues ====== Add keywords of "meta" directive to search index (#2463) -------------------------------------------------------- Opened by TimKam :: It would be great to have the keywords of `meta` directives included in the search index. Like this, one can help users who are searching for a synonym of the "correct" term through simply adding synonyms as keywords to a meta directive on the corresponding page. source/issues.rst
40. Issues ====== .. github-issue:: 2463 source/issues.rst
41. from docutils import nodes from docutils.parsers.rst import Directive import requests URL = 'https://api.github.com/repos/sphinx-doc/sphinx/issues/{}' def get_issue(issue_id): issue = requests.get(URL.format(issue_id)).json() title = '%s (#%s)' % (issue['title'], issue_id) owner = 'Opened by %s' issue['user']['login'] return issue_id, title, issue['body'], owner ... ext/issue_directive.py
42. ... class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py
43. ... class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py
44. ... class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py
45. ... def setup(app): app.add_directive('github-issue', ShowGitHubIssue) return {'version': '1.0', 'parallel_read_safe': True} ext/issue_directive.py
46. Issues ====== .. github-issue:: 2463 source/issues.rst
47. Issues Add keywords of “meta” directive to search index (#2463) Opened by TimKam build/issues.html It would be great to have the keywords of `meta` directives included in the search index. Like this, one can help users who are searching for a synonym of the "correct" term through simply adding synonyms as keywords to a `meta` directive on the corresponding page.
48. class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py
49. class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) section = nodes.section(ids=['github-issue-%s' % issue[0]]) section += nodes.title(text=issue[1]) section += nodes.paragraph(text='Opened by %s' % issue[3]) section += nodes.literal_block(text=issue[2]) return [section] ext/issue_directive.py
50. class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) result = statemachine.ViewList() for line in format_issue(issue): result.append(line, '<' + __name__ + '>') node = nodes.section(document=self.state.document) nested_parse_with_titles(self.state, result, node) return node.children ext/issue_directive.py
51. class ShowGitHubIssue(Directive): required_arguments = 1 def run(self): issue = get_issue(self.arguments[0]) result = statemachine.ViewList() for line in format_issue(issue): result.append(line, '<' + __name__ + '>') node = nodes.section(document=self.state.document) nested_parse_with_titles(self.state, result, node) return node.children ext/issue_directive.py
52. def format_issue(issue): num, title, body, owner = issue yield title yield '=' * len(title) yield '' yield '%s ::' % owner yield '' for line in body.splitlines(): yield ' %s' % line if line else '' ext/issue_directive.py
53. Issues Add keywords of “meta” directive to search index (#2463) Opened by TimKam It would be great to have the keywords of `meta` directives included in the search index. Like this, one can help users who are searching for a synonym of the "correct" term through simply adding synonyms as keywords to a `meta` directive on the corresponding page. build/issues.html
54. Events 5
55. builder-inited(app) config-inited(app, config) source-read(app, docname, source) doctree-read(app, doctree) ...
56. from docutils import nodes from docutils.parsers.rst import Directive def builder_inited_handler(app): # code here... def setup(app): app.connect('builder-inited', builder_inited_handler)
57. issues.rst issue-2951.rst issue-2463.rst issue-2516.rst issue-2471.rst ...
58. from docutils import nodes from docutils.parsers.rst import Directive import requests URL = 'https://api.github.com/repos/sphinx-doc/sphinx/issues/' def get_issues(): issues = requests.get(URL).json() for issue in issues: title = '%s (#%s)' % (issue['title'], issue['number']) owner = 'Opened by %s' % issue['user']['login'] yield issue['number'], title, issue['body'], owner ext/issue_event.py
59. ... def generate_issue_docs(app): for num, title, body, owner in get_issues(): filename = os.path.join(app.srcdir, 'issues', '%s.rst' % num) with io.open(filename, 'w') as issue_doc: print(title, file=issue_doc) print('=' * len(title), file=issue_doc) print('', file=issue_doc) print('%s ::' % owner, file=issue_doc) print('', file=issue_doc) for line in body.splitlines(): print(' %s' % line if line else '', file=issue_doc) ext/issue_event.py
60. ... def setup(app): app.connect('builder-inited', generate_issue_docs) return {'version': '1.0', 'parallel_read_safe': True} ext/issue_event.py
61. Issues ====== .. toctree:: :maxdepth: 1 :glob: issues/* source/index.rst
62. Issues • Drop special support for rst2pdf (#4463) • Proposal: Integrate source_suffix and source_parsers (#4474) • [RFC] Implement delayed resolution in TOC (#4475) • Not possible to update individual ‘po’ files (#4476) • Build fails during eclim (aur) build: Babel data files not available (#4481) • Proposal: Allow to switch parsers on parsing document (#4482) • Integrate source suffix and source parsers (#4483) • … build/index.html
63. Enabling Your Extensions 5
64. import os import sys sys.path.insert(0, os.path.abspath('../ext')) extensions = [ 'issue_role', 'issue_directive', 'issue_event', ] source/conf.py
65. import os import sys sys.path.insert(0, os.path.abspath('../ext')) extensions = [ 'issue_role', 'issue_directive', 'issue_event', 'oslo_config.sphinxext', ] source/conf.py
66. Wrap Up 6
67. Extensions are registered via sphinx.application.Sphinx add_builder add_config_value add_domain add_event add_node add_directive add_role connect, disconnect ... (Builders) (Config Values) (Domains) (Events) (docutils Nodes) (Directives) (Interpreted Text Roles, a.k.a. Roles) (Hooks) ...
68. Extensions are registered via sphinx.application.Sphinx add_builder add_config_value add_domain add_event add_node add_directive add_role connect, disconnect ... (Builders) (Config Values) (Domains) (Events) (docutils Nodes) (Directives) (Interpreted Text Roles, a.k.a. Roles) (Hooks) ...
69. Extensions are registered via sphinx.application.Sphinx Builder-specific Extensions (HTML themes, LaTeX templates, …) (Post) Transforms Translators Parsers Search languages ...
70. Fin
71. A lion, a head, and a dash of YAML Extending Sphinx to automate your documentation PyCon Limerick 2020 @stephenfin
72. References • Quick reStructuredText • Docutils Reference Guide ◦ reStructuredText Markup Specification ◦ reStructuredText Directives ◦ reStructuredText Interpreted Text Roles • Docutils How-Tos ◦ Creating reStructuredText Interpreted Text Roles ◦ Creating reStructuredText Directives • Docutils Hacker’s Guide • Sphinx Tutorial: Writing a simple extension
73. References • Defining Custom Roles in Sphinx -- Doug Hellmann • The Power of Sphinx - Integrating Jinja with RST -- Eric Holscher • Docutils Snippets -- Aurélien Gâteau • OpenStack + Sphinx In A Tree -- Stephen Finucane ()