jyapayne/Nim
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docgen: implement cross-document links (#20990)

Browse source 
* docgen: implement cross-document links

Fully implements https://github.com/nim-lang/RFCs/issues/125
Follow-up of: https://github.com/nim-lang/Nim/pull/18642 (for internal links)
and https://github.com/nim-lang/Nim/issues/20127.

Overview
--------

Explicit import-like directive is required, called `.. importdoc::`.
(the syntax is % RST, Markdown will use it for a while).

Then one can reference any symbols/headings/anchors, as if they
were in the local file (but they will be prefixed with a module name
or markup document in link text).
It's possible to reference anything from anywhere (any direction
in `.nim`/`.md`/`.rst` files).

See `doc/docgen.md` for full description.

Working is based on `.idx` files, hence one needs to generate
all `.idx` beforehand. A dedicated option `--index:only` is introduced
(and a separate stage for `--index:only` is added to `kochdocs.nim`).

Performance note
----------------

Full run for `./koch docs` now takes 185% of the time before this PR.
(After: 315 s, before: 170 s on my PC).
All the time seems to be spent on `--index:only` run, which takes
almost as much (85%) of normal doc run -- it seems that most time
is spent on file parsing, turning off HTML generation phase has not
helped much.
(One could avoid it by specifying list of files that can be referenced
and pre-processing only them. But it can become error-prone and I assume
that these linke will be **everywhere** in the repository anyway,
especially considering https://github.com/nim-lang/RFCs/issues/478.
So every `.nim`/`.md` file is processed for `.idx` first).

But that's all without significant part of repository converted to
cross-module auto links. To estimate impact I checked the time for
`doc`ing a few files (after all indexes have been generated), and
everywhere difference was **negligible**.
E.g. for `lib/std/private/osfiles.nim` that `importdoc`s large
`os.idx` and hence should have been a case with relatively large
performance impact, but:

* After: 0.59 s.
* Before: 0.59 s.

So Nim compiler works so slow that doc part basically does not matter :-)

Testing
-------

1) added `extlinks` test to `nimdoc/`
2) checked that `theindex.html` is still correct
2) fixed broken auto-links for modules that were derived from `os.nim`
   by adding appropriate ``importdoc``

Implementation note
-------------------

Parsing and formating of `.idx` entries is moved into a dedicated
`rstidx.nim` module from `rstgen.nim`.

`.idx` file format changed:

* fields are not escaped in most cases because we need original
  strings for referencing, not HTML ones
  (the exception is linkTitle for titles and headings).
  Escaping happens later -- on the stage of `rstgen` buildIndex, etc.
* all lines have fixed number of columns 6
* added discriminator tag as a first column,
  it always allows distinguish Nim/markup entries, titles/headings, etc.
  `rstgen` does not rely any more (in most cases) on ad-hoc logic
  to determine what type each entry is.
* there is now always a title entry added at the first line.
* add a line number as 6th column
* linkTitle (4th) column has a different format: before it was like
  `module: funcName()`, now it's `proc funcName()`.
  (This format is also propagated to `theindex.html` and search results,
  I kept it that way since I like it more though it's discussible.)
  This column is what used for Nim symbols resolution.
* also changed details on column format for headings and titles:
  "keyword" is original, "linkTitle" is HTML one

* fix paths on Windows + more clear code

* Update compiler/docgen.nim

Co-authored-by: Andreas Rumpf <rumpf_a@web.de>

* Handle .md and .nim paths uniformly in findRefFile

* handle titles better + more comments

* don't allow markup overwrite index title for .nim files

Co-authored-by: Andreas Rumpf <rumpf_a@web.de>
This commit is contained in:
Andrey Makarov 2023-01-04 23:19:01 +03:00 • committed by GitHub
commit 2620da9bf9
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
45 changed files with 1863 additions and 491 deletions
Download patch file Download diff file Expand all files Collapse all files

4
doc/advopt.txt
View file

@ -130,7 +130,9 @@ Advanced options:
select which memory management to use; default is 'orc'
--exceptions:setjmp|cpp|goto|quirky
select the exception handling implementation
--index:on|off turn index file generation on|off
--index:on|off|only docgen: turn index file generation? (`only` means
not generate output files like HTML)
--noImportdoc:on|off disable loading documentation ``.idx`` files?
--putenv:key=value set an environment variable
--NimblePath:PATH add a path for Nimble support
--noNimblePath deactivate the Nimble path

309
doc/docgen.md
View file

@ -9,6 +9,7 @@
.. include:: rstcommon.rst
.. contents::
.. importdoc:: markdown_rst.md, compiler/docgen.nim
Introduction
============
@ -103,6 +104,64 @@ won't influence RST formatting.
## Paragraph.
```
Structuring output directories
------------------------------
Basic directory for output is set by `--outdir:OUTDIR`:option: switch,
by default `OUTDIR` is ``htmldocs`` sub-directory in the directory of
the processed file.
There are 2 basic options as to how generated HTML output files are stored:
1) complex hierarchy when docgen-compiling with `--project`:option:,
which follows directory structure of the project itself.
So `nim doc`:cmd: replicates project's directory structure
inside `--outdir:OUTDIR`:option: directory.
`--project`:option: is well suited for projects that have 1 main module.
File name clashes are impossible in this case.
2) flattened structure, where user-provided script goes through all
needed input files and calls commands like `nim doc`:cmd:
with `--outdir:OUTDIR`:option: switch, thus putting all HTML (and
``.idx``) files into 1 directory.
.. Important:: Make sure that you don't have files with same base name
like ``x.nim`` and ``x.md`` in the same package, otherwise you'll
have name conflict for ``x.html``.
.. Tip:: To structure your output directories and avoid file name
clashes you can split your project into
different *packages* -- parts of your repository that are
docgen-compiled with different `--outdir:OUTDIR`:option: options.
An example of such strategy is Nim repository itself which has:
* its stdlib ``.nim`` files from different directories and ``.md``
documentation from ``doc/`` directory are all docgen-compiled
into `--outdir:web/upload/<version>/`:option: directory
* its ``.nim`` files from ``compiler/`` directory are docgen-compiled
into `--outdir:web/upload/<version>/compiler/`:option: directory.
Interestingly, it's compiled with complex hierarchy using
`--project`:option: switch.
Contents of ``web/upload/<version>`` are then deployed into Nim's
Web server.
This output directory structure allows to work correctly with files like
``compiler/docgen.nim`` (implementation) and ``doc/docgen.md`` (user
documentation) in 1 repository.
Index files
-----------
Index (``.idx``) files are used for 2 different purposes:
1. easy cross-referencing between different ``.nim`` and/or ``.md`` / ``.rst``
files described in [Nim external referencing]
2. creating a whole-project index for searching of symbols and keywords,
see [Buildindex command].
Document Types
==============
@ -226,13 +285,46 @@ Note that the `jsondoc`:option: command outputs its JSON without pretty-printing
while `jsondoc0`:option: outputs pretty-printed JSON.
Referencing Nim symbols: simple documentation links
===================================================
Simple documentation links
==========================
You can reference Nim identifiers from Nim documentation comments, currently
only inside their ``.nim`` file (or inside a ``.rst`` file included from
a ``.nim``). The point is that such links will be resolved automatically
by `nim doc`:cmd: (or `nim jsondoc`:cmd: or `nim doc2tex`:cmd:).
It's possible to use normal Markdown/RST syntax to *manually*
reference Nim symbols using HTML anchors, however Nim has an *automatic*
facility that makes referencing inside ``.nim`` and ``.md/.rst`` files and
between them easy and seamless.
The point is that such links will be resolved automatically
by `nim doc`:cmd: (or `md2html`:option:, or `jsondoc`:option:,
or `doc2tex`:option:, ...). And, unlike manual links, such automatic
links **check** that their target exists -- a warning is emitted for
any broken link, so you avoid broken links in your project.
Nim treats both ``.md/.rst`` files and ``.nim`` modules (their doc comment
part) as *documents* uniformly.
Hence all directions of referencing are equally possible having the same syntax:
1. ``.md/rst`` -> itself (internal). See [Markup local referencing].
2. ``.md/rst`` -> external ``.md/rst``. See [Markup external referencing].
To summarize, referencing in `.md`/`.rst` files was already described in
[Nim-flavored Markdown and reStructuredText]
(particularly it described usage of index files for referencing),
while in this document we focus on Nim-specific details.
3. ``.md/rst`` -> external ``.nim``. See [Nim external referencing].
4. ``.nim`` -> itself (internal). See [Nim local referencing].
5. ``.nim`` -> external ``.md/rst``. See [Markup external referencing].
6. ``.nim`` -> external ``.nim``. See [Nim external referencing].
To put it shortly, local referencing always works out of the box,
external referencing requires to use ``.. importdoc:: <file>``
directive to import `file` and to ensure that the corresponding
``.idx`` file was generated.
Nim local referencing
---------------------
You can reference Nim identifiers from Nim documentation comments
inside their ``.nim`` file (or inside a ``.rst`` file included from
a ``.nim``).
This pertains to any exported symbol like `proc`, `const`, `iterator`, etc.
Syntax for referencing is basically a normal RST one: addition of
underscore `_` to a *link text*.
@ -405,6 +497,143 @@ recognized fine:
...
## Ref. `CopyFlag enum`_
Nim external referencing
------------------------
Just like for [Markup external referencing], which saves markup anchors,
the Nim symbols are also saved in ``.idx`` files, so one needs
to generate them beforehand, and they should be loaded by
an ``.. importdoc::`` directive. Arguments to ``.. importdoc::`` is a
comma-separated list of Nim modules or Markdown/RST documents.
`--index:only`:option: tells Nim to only generate ``.idx`` file and
do **not** attempt to generate HTML/LaTeX output.
For ``.nim`` modules there are 2 alternatives to work with ``.idx`` files:
1. using [Project switch] implies generation of ``.idx`` files,
however, if ``importdoc`` is called on upper modules as its arguments,
their ``.idx`` are not yet created. Thus one should generate **all**
required ``.idx`` first:
```cmd
nim doc --project --index:only <main>.nim
nim doc --project <main>.nim
```
2. or run `nim doc --index:only <module.nim>`:cmd: command for **all** (used)
Nim modules in your project. Then run `nim doc <module.nim>` on them for
output HTML generation.
.. Warning:: A mere `nim doc --index:on`:cmd: may fail on an attempt to do
``importdoc`` from another module (for which ``.idx`` was not yet
generated), that's why `--index:only`:option: shall be used instead.
For ``.md``/``.rst`` markup documents point 2 is the only option.
Then, you can freely use something like this in ``your_module.nim``:
```nim
## .. importdoc:: user_manual.md, another_module.nim
...
## Ref. [some section from User Manual].
...
## Ref. [proc f]
## (assuming you have a proc `f` in ``another_module``).
```
and compile it by `nim doc`:cmd:. Note that link text will
be automatically prefixed by the module name of symbol,
so you will see something like "Ref. [another_module: proc f](#)"
in the generated output.
It's also possible to reference a whole module by prefixing or
suffixing full canonical module name with "module":
Ref. [module subdir/name] or [subdir/name module].
Markup documents as a whole can be referenced just by their title
(or by their file name if the title was not set) without any prefix.
.. Tip:: During development process the stage of ``.idx`` files generation
can be done only *once*, after that you use already generated ``.idx``
files while working with a document *being developed* (unless you do
incompatible changes to *referenced* documents).
.. Hint:: After changing a *referenced* document file one may need
to regenerate its corresponding ``.idx`` file to get correct results.
Of course, when referencing *internally* inside any given ``.nim`` file,
it's not needed, one can even immediately use any freshly added anchor
(a document's own ``.idx`` file is not used for resolving its internal links).
If an ``importdoc`` directive fails to find a ``.idx``, then an error
is emitted.
In case of such compilation failures please note that:
* **all** relative paths, given to ``importdoc``, relate to insides of
``OUTDIR``, and **not** project's directory structure.
* ``importdoc`` searches for ``.idx`` in `--outdir:OUTDIR`:option: directory
(``htmldocs`` by default) and **not** around original modules, so:
.. Tip:: look into ``OUTDIR`` to understand what's going on.
* also keep in mind that ``.html`` and ``.idx`` files should always be
output to the same directory, so check this and, if it's not true, check
that both runs *with* and *without* `--index:only`:option: have all
other options the same.
To summarize, for 2 basic options of [Structuring output directories]
compilation options are different:
1) complex hierarchy with `--project`:option: switch.
As the **original** project's directory structure is replicated in
`OUTDIR`, all passed paths are related to this structure also.
E.g. if a module ``path1/module.nim`` does
``.. importdoc:: path2/another.nim`` then docgen tries to load file
``OUTDIR/path1/path2/another.idx``.
.. Note:: markup documents are just placed into the specified directory
`OUTDIR`:option: by default (i.e. they are **not** affected by
`--project`:option:), so if you have ``PROJECT/doc/manual.md``
document and want to use complex hirearchy (with ``doc/``),
compile it with `--docroot`:option:\:
```cmd
# 1st stage
nim md2html --outdir:OUTDIR --docroot:/absolute/path/to/PROJECT \
--index:only PROJECT/doc/manual.md
...
# 2nd stage
nim md2html --outdir:OUTDIR --docroot:/absolute/path/to/PROJECT \
PROJECT/doc/manual.md
```
Then the output file will be placed as ``OUTDIR/doc/manual.idx``.
So if you have ``PROJECT/path1/module.nim``, then ``manual.md`` can
be referenced as ``../doc/manual.md``.
2) flattened structure.
E.g. if a module ``path1/module.nim`` does
``.. importdoc:: path2/another.nim`` then docgen tries to load
``OUTDIR/path2/another.idx``, so the path ``path1``
does not matter and providing ``path2`` can be useful only
in the case it contains another package that was placed there
using `--outdir:OUTDIR/path2`:option:.
The links' text will be prefixed as ``another: ...`` in both cases.
.. Warning:: Again, the same `--outdir:OUTDIR`:option: option should
be provided to both `doc --index:only`:option: /
`md2html --index:only`:option: and final generation by
`doc`:option:/`md2html`:option: inside 1 package.
To temporarily disable ``importdoc``, e.g. if you don't need
correct link resolution at the moment, use a `--noImportdoc`:option: switch
(only warnings about unresolved links will be generated for external references).
Related Options
===============
@ -434,10 +663,21 @@ index file is line-oriented (newlines have to be escaped). Each line
represents a tab-separated record of several columns, the first two mandatory,
the rest optional. See the [Index (idx) file format] section for details.
.. Note:: `--index`:option: switch only affects creation of ``.idx``
index files, while user-searchable Index HTML file is created by
`buildIndex`:option: commmand.
Buildindex command
------------------
Once index files have been generated for one or more modules, the Nim
compiler command `buildIndex directory` can be run to go over all the index
compiler command `nim buildIndex directory`:cmd: can be run to go over all the index
files in the specified directory to generate a [theindex.html](theindex.html)
file.
file:
```cmd
nim buildIndex -o:path/to/htmldocs/theindex.html path/to/htmldocs
```
See source switch
-----------------
@ -568,10 +808,22 @@ references so they can be later concatenated into a big index file with
the file format in detail.
Index files are line-oriented and tab-separated (newline and tab characters
have to be escaped). Each line represents a record with at least two fields
but can have up to four (additional columns are ignored). The content of these
columns is:
have to be escaped). Each line represents a record with 6 fields.
The content of these columns is:
0. Discriminator tag denoting type of the index entry, allowed values are:
`markupTitle`
: a title for ``.md``/``.rst`` document
`nimTitle`
: a title of ``.nim`` module
`heading`
: heading of sections, can be both in Nim and markup files
`idx`
: terms marked with :idx: role
`nim`
: a Nim symbol
`nimgrp`
: a Nim group for overloadable symbols like `proc`s
1. Mandatory term being indexed. Terms can include quoting according to
Nim's rules (e.g. \`^\`).
2. Base filename plus anchor hyperlink (e.g. ``algorithm.html#*,int,SortOrder``).
@ -581,29 +833,20 @@ columns is:
not for an API symbol but for a TOC entry.
4. Optional title or description of the hyperlink. Browsers usually display
this as a tooltip after hovering a moment over the hyperlink.
5. A line number of file where the entry was defined.
The index generation tools try to differentiate between documentation
generated from ``.nim`` files and documentation generated from ``.txt`` or
``.rst`` files. The former are always closely related to source code and
consist mainly of API entries. The latter are generic documents meant for
human reading.
The index generation tools differentiate between documentation
generated from ``.nim`` files and documentation generated from ``.md`` or
``.rst`` files by tag `nimTitle` or `markupTitle` in the 1st line of
the ``.idx`` file.
To differentiate both types (documents and APIs), the index generator will add
to the index of documents an entry with the title of the document. Since the
title is the topmost element, it will be added with a second field containing
just the filename without any HTML anchor. By convention, this entry without
anchor is the *title entry*, and since entries in the index file are added as
they are scanned, the title entry will be the first line. The title for APIs
is not present because it can be generated concatenating the name of the file
to the word **Module**.
Normal symbols are added to the index with surrounding whitespaces removed. An
exception to this are the table of content (TOC) entries. TOC entries are added to
the index file with their third column having as much prefix spaces as their
level is in the TOC (at least 1 character). The prefix whitespace helps to
filter TOC entries from API or text symbols. This is important because the
amount of spaces is used to replicate the hierarchy for document TOCs in the
final index, and TOC entries found in ``.nim`` files are discarded.
.. TODO Normal symbols are added to the index with surrounding whitespaces removed. An
exception to this are the table of content (TOC) entries. TOC entries are added to
the index file with their third column having as much prefix spaces as their
level is in the TOC (at least 1 character). The prefix whitespace helps to
filter TOC entries from API or text symbols. This is important because the
amount of spaces is used to replicate the hierarchy for document TOCs in the
final index, and TOC entries found in ``.nim`` files are discarded.
Additional resources
@ -615,6 +858,8 @@ Additional resources
[Markdown and RST markup languages](markdown_rst.html), which also
contains the list of implemented features of these markup languages.
* the implementation is in [module compiler/docgen].
The output for HTML and LaTeX comes from the ``config/nimdoc.cfg`` and
``config/nimdoc.tex.cfg`` configuration files. You can add and modify these
files to your project to change the look of the docgen output.

84
doc/markdown_rst.md
View file

@ -9,6 +9,8 @@ Nim-flavored Markdown and reStructuredText
.. include:: rstcommon.rst
.. contents::
.. importdoc:: docgen.md
Both `Markdown`:idx: (md) and `reStructuredText`:idx: (RST) are markup
languages whose goal is to typeset texts with complex structure,
formatting and references using simple plaintext representation.
@ -110,6 +112,8 @@ Supported standard RST features:
Additional Nim-specific features
--------------------------------
* referencing to definitions in external files, see
[Markup external referencing] section
* directives: ``code-block`` \[cmp:Sphinx], ``title``,
``index`` \[cmp:Sphinx]
* predefined roles
@ -170,6 +174,86 @@ Optional additional features, by default turned on:
.. warning:: Using Nim-specific features can cause other RST implementations
to fail on your document.
Referencing
===========
To be able to copy and share links Nim generates anchors for all
main document elements:
* headlines (including document title)
* footnotes
* explicitly set anchors: RST internal cross-references and
inline internal targets
* Nim symbols (external referencing), see [Nim DocGen Tools Guide] for details.
But direct use of those anchors have 2 problems:
1. the anchors are usually mangled (e.g. spaces substituted to minus
signs, etc).
2. manual usage of anchors is not checked, so it's easy to get broken
links inside your project if e.g. spelling has changed for a heading
or you use a wrong relative path to your document.
That's why Nim implementation has syntax for using
*original* labels for referencing.
Such referencing can be either local/internal or external:
* Local referencing (inside any given file) is defined by
RST standard or Pandoc Markdown User guide.
* External (cross-document) referencing is a Nim-specific feature,
though it's not really different from local referencing by its syntax.
Markup local referencing
------------------------
There are 2 syntax option available for referencing to objects
inside any given file, e.g. for headlines:
Markdown RST
Some headline Some headline
============= =============
Ref. [Some headline] Ref. `Some headline`_
Markup external referencing
---------------------------
The syntax is the same as for local referencing, but the anchors are
saved in ``.idx`` files, so one needs to generate them beforehand,
and they should be loaded by an `.. importdoc::` directive.
E.g. if we want to reference section "Some headline" in ``file1.md``
from ``file2.md``, then ``file2.md`` may look like:
```
.. importdoc:: file1.md
Ref. [Some headline]
```
```cmd
nim md2html --index:only file1.md # creates ``htmldocs/file1.idx``
nim md2html file2.md # creates ``htmldocs/file2.html``
```
To allow cross-references between any files in any order (especially, if
circular references are present), it's strongly reccommended
to make a run for creating all the indexes first:
```cmd
nim md2html --index:only file1.md # creates ``htmldocs/file1.idx``
nim md2html --index:only file2.md # creates ``htmldocs/file2.idx``
nim md2html file1.md # creates ``htmldocs/file1.html``
nim md2html file2.md # creates ``htmldocs/file2.html``
```
and then one can freely reference any objects as if these 2 documents
are actually 1 file.
Other
=====
Idiosyncrasies
--------------