manual split up into multiple files; documented the new concurrency system
This commit is contained in:
Araq
parent
e9dec2feed
commit
2c1f3f75f5
31 changed files with 6198 additions and 5901 deletions
356
doc/manual/lexing.txt
Normal file
356
doc/manual/lexing.txt
Normal file
|
|
@ -0,0 +1,356 @@
|
|||
Lexical Analysis
|
||||
================
|
||||
|
||||
Encoding
|
||||
--------
|
||||
|
||||
All Nim source files are in the UTF-8 encoding (or its ASCII subset). Other
|
||||
encodings are not supported. Any of the standard platform line termination
|
||||
sequences can be used - the Unix form using ASCII LF (linefeed), the Windows
|
||||
form using the ASCII sequence CR LF (return followed by linefeed), or the old
|
||||
Macintosh form using the ASCII CR (return) character. All of these forms can be
|
||||
used equally, regardless of platform.
|
||||
|
||||
|
||||
Indentation
|
||||
-----------
|
||||
|
||||
Nim's standard grammar describes an `indentation sensitive`:idx: language.
|
||||
This means that all the control structures are recognized by indentation.
|
||||
Indentation consists only of spaces; tabulators are not allowed.
|
||||
|
||||
The indentation handling is implemented as follows: The lexer annotates the
|
||||
following token with the preceding number of spaces; indentation is not
|
||||
a separate token. This trick allows parsing of Nim with only 1 token of
|
||||
lookahead.
|
||||
|
||||
The parser uses a stack of indentation levels: the stack consists of integers
|
||||
counting the spaces. The indentation information is queried at strategic
|
||||
places in the parser but ignored otherwise: The pseudo terminal ``IND{>}``
|
||||
denotes an indentation that consists of more spaces than the entry at the top
|
||||
of the stack; IND{=} an indentation that has the same number of spaces. ``DED``
|
||||
is another pseudo terminal that describes the *action* of popping a value
|
||||
from the stack, ``IND{>}`` then implies to push onto the stack.
|
||||
|
||||
With this notation we can now easily define the core of the grammar: A block of
|
||||
statements (simplified example)::
|
||||
|
||||
ifStmt = 'if' expr ':' stmt
|
||||
(IND{=} 'elif' expr ':' stmt)*
|
||||
(IND{=} 'else' ':' stmt)?
|
||||
|
||||
simpleStmt = ifStmt / ...
|
||||
|
||||
stmt = IND{>} stmt ^+ IND{=} DED # list of statements
|
||||
/ simpleStmt # or a simple statement
|
||||
|
||||
|
||||
|
||||
Comments
|
||||
--------
|
||||
|
||||
Comments start anywhere outside a string or character literal with the
|
||||
hash character ``#``.
|
||||
Comments consist of a concatenation of `comment pieces`:idx:. A comment piece
|
||||
starts with ``#`` and runs until the end of the line. The end of line characters
|
||||
belong to the piece. If the next line only consists of a comment piece with
|
||||
no other tokens between it and the preceding one, it does not start a new
|
||||
comment:
|
||||
|
||||
|
||||
.. code-block:: nim
|
||||
i = 0 # This is a single comment over multiple lines.
|
||||
# The scanner merges these two pieces.
|
||||
# The comment continues here.
|
||||
|
||||
|
||||
`Documentation comments`:idx: are comments that start with two ``##``.
|
||||
Documentation comments are tokens; they are only allowed at certain places in
|
||||
the input file as they belong to the syntax tree!
|
||||
|
||||
|
||||
Identifiers & Keywords
|
||||
----------------------
|
||||
|
||||
Identifiers in Nim can be any string of letters, digits
|
||||
and underscores, beginning with a letter. Two immediate following
|
||||
underscores ``__`` are not allowed::
|
||||
|
||||
letter ::= 'A'..'Z' | 'a'..'z' | '\x80'..'\xff'
|
||||
digit ::= '0'..'9'
|
||||
IDENTIFIER ::= letter ( ['_'] (letter | digit) )*
|
||||
|
||||
Currently any unicode character with an ordinal value > 127 (non ASCII) is
|
||||
classified as a ``letter`` and may thus be part of an identifier but later
|
||||
versions of the language may assign some Unicode characters to belong to the
|
||||
operator characters instead.
|
||||
|
||||
The following keywords are reserved and cannot be used as identifiers:
|
||||
|
||||
.. code-block:: nim
|
||||
:file: keywords.txt
|
||||
|
||||
Some keywords are unused; they are reserved for future developments of the
|
||||
language.
|
||||
|
||||
Nim is a `style-insensitive`:idx: language. This means that it is not
|
||||
case-sensitive and even underscores are ignored:
|
||||
**type** is a reserved word, and so is **TYPE** or **T_Y_P_E**. The idea behind
|
||||
this is that this allows programmers to use their own preferred spelling style
|
||||
and libraries written by different programmers cannot use incompatible
|
||||
conventions. A Nim-aware editor or IDE can show the identifiers as
|
||||
preferred. Another advantage is that it frees the programmer from remembering
|
||||
the exact spelling of an identifier.
|
||||
|
||||
|
||||
String literals
|
||||
---------------
|
||||
|
||||
Terminal symbol in the grammar: ``STR_LIT``.
|
||||
|
||||
String literals can be delimited by matching double quotes, and can
|
||||
contain the following `escape sequences`:idx:\ :
|
||||
|
||||
================== ===================================================
|
||||
Escape sequence Meaning
|
||||
================== ===================================================
|
||||
``\n`` `newline`:idx:
|
||||
``\r``, ``\c`` `carriage return`:idx:
|
||||
``\l`` `line feed`:idx:
|
||||
``\f`` `form feed`:idx:
|
||||
``\t`` `tabulator`:idx:
|
||||
``\v`` `vertical tabulator`:idx:
|
||||
``\\`` `backslash`:idx:
|
||||
``\"`` `quotation mark`:idx:
|
||||
``\'`` `apostrophe`:idx:
|
||||
``\`` '0'..'9'+ `character with decimal value d`:idx:;
|
||||
all decimal digits directly
|
||||
following are used for the character
|
||||
``\a`` `alert`:idx:
|
||||
``\b`` `backspace`:idx:
|
||||
``\e`` `escape`:idx: `[ESC]`:idx:
|
||||
``\x`` HH `character with hex value HH`:idx:;
|
||||
exactly two hex digits are allowed
|
||||
================== ===================================================
|
||||
|
||||
|
||||
Strings in Nim may contain any 8-bit value, even embedded zeros. However
|
||||
some operations may interpret the first binary zero as a terminator.
|
||||
|
||||
|
||||
Triple quoted string literals
|
||||
-----------------------------
|
||||
|
||||
Terminal symbol in the grammar: ``TRIPLESTR_LIT``.
|
||||
|
||||
String literals can also be delimited by three double quotes
|
||||
``"""`` ... ``"""``.
|
||||
Literals in this form may run for several lines, may contain ``"`` and do not
|
||||
interpret any escape sequences.
|
||||
For convenience, when the opening ``"""`` is followed by a newline (there may
|
||||
be whitespace between the opening ``"""`` and the newline),
|
||||
the newline (and the preceding whitespace) is not included in the string. The
|
||||
ending of the string literal is defined by the pattern ``"""[^"]``, so this:
|
||||
|
||||
.. code-block:: nim
|
||||
""""long string within quotes""""
|
||||
|
||||
Produces::
|
||||
|
||||
"long string within quotes"
|
||||
|
||||
|
||||
Raw string literals
|
||||
-------------------
|
||||
|
||||
Terminal symbol in the grammar: ``RSTR_LIT``.
|
||||
|
||||
There are also raw string literals that are preceded with the
|
||||
letter ``r`` (or ``R``) and are delimited by matching double quotes (just
|
||||
like ordinary string literals) and do not interpret the escape sequences.
|
||||
This is especially convenient for regular expressions or Windows paths:
|
||||
|
||||
.. code-block:: nim
|
||||
|
||||
var f = openFile(r"C:\texts\text.txt") # a raw string, so ``\t`` is no tab
|
||||
|
||||
To produce a single ``"`` within a raw string literal, it has to be doubled:
|
||||
|
||||
.. code-block:: nim
|
||||
|
||||
r"a""b"
|
||||
|
||||
Produces::
|
||||
|
||||
a"b
|
||||
|
||||
``r""""`` is not possible with this notation, because the three leading
|
||||
quotes introduce a triple quoted string literal. ``r"""`` is the same
|
||||
as ``"""`` since triple quoted string literals do not interpret escape
|
||||
sequences either.
|
||||
|
||||
|
||||
Generalized raw string literals
|
||||
-------------------------------
|
||||
|
||||
Terminal symbols in the grammar: ``GENERALIZED_STR_LIT``,
|
||||
``GENERALIZED_TRIPLESTR_LIT``.
|
||||
|
||||
The construct ``identifier"string literal"`` (without whitespace between the
|
||||
identifier and the opening quotation mark) is a
|
||||
generalized raw string literal. It is a shortcut for the construct
|
||||
``identifier(r"string literal")``, so it denotes a procedure call with a
|
||||
raw string literal as its only argument. Generalized raw string literals
|
||||
are especially convenient for embedding mini languages directly into Nim
|
||||
(for example regular expressions).
|
||||
|
||||
The construct ``identifier"""string literal"""`` exists too. It is a shortcut
|
||||
for ``identifier("""string literal""")``.
|
||||
|
||||
|
||||
Character literals
|
||||
------------------
|
||||
|
||||
Character literals are enclosed in single quotes ``''`` and can contain the
|
||||
same escape sequences as strings - with one exception: `newline`:idx: (``\n``)
|
||||
is not allowed as it may be wider than one character (often it is the pair
|
||||
CR/LF for example). Here are the valid `escape sequences`:idx: for character
|
||||
literals:
|
||||
|
||||
================== ===================================================
|
||||
Escape sequence Meaning
|
||||
================== ===================================================
|
||||
``\r``, ``\c`` `carriage return`:idx:
|
||||
``\l`` `line feed`:idx:
|
||||
``\f`` `form feed`:idx:
|
||||
``\t`` `tabulator`:idx:
|
||||
``\v`` `vertical tabulator`:idx:
|
||||
``\\`` `backslash`:idx:
|
||||
``\"`` `quotation mark`:idx:
|
||||
``\'`` `apostrophe`:idx:
|
||||
``\`` '0'..'9'+ `character with decimal value d`:idx:;
|
||||
all decimal digits directly
|
||||
following are used for the character
|
||||
``\a`` `alert`:idx:
|
||||
``\b`` `backspace`:idx:
|
||||
``\e`` `escape`:idx: `[ESC]`:idx:
|
||||
``\x`` HH `character with hex value HH`:idx:;
|
||||
exactly two hex digits are allowed
|
||||
================== ===================================================
|
||||
|
||||
A character is not an Unicode character but a single byte. The reason for this
|
||||
is efficiency: for the overwhelming majority of use-cases, the resulting
|
||||
programs will still handle UTF-8 properly as UTF-8 was specially designed for
|
||||
this. Another reason is that Nim can thus support ``array[char, int]`` or
|
||||
``set[char]`` efficiently as many algorithms rely on this feature. The `TRune`
|
||||
type is used for Unicode characters, it can represent any Unicode character.
|
||||
``TRune`` is declared in the `unicode module <unicode.html>`_.
|
||||
|
||||
|
||||
Numerical constants
|
||||
-------------------
|
||||
|
||||
Numerical constants are of a single type and have the form::
|
||||
|
||||
hexdigit = digit | 'A'..'F' | 'a'..'f'
|
||||
octdigit = '0'..'7'
|
||||
bindigit = '0'..'1'
|
||||
HEX_LIT = '0' ('x' | 'X' ) hexdigit ( ['_'] hexdigit )*
|
||||
DEC_LIT = digit ( ['_'] digit )*
|
||||
OCT_LIT = '0o' octdigit ( ['_'] octdigit )*
|
||||
BIN_LIT = '0' ('b' | 'B' ) bindigit ( ['_'] bindigit )*
|
||||
|
||||
INT_LIT = HEX_LIT
|
||||
| DEC_LIT
|
||||
| OCT_LIT
|
||||
| BIN_LIT
|
||||
|
||||
INT8_LIT = INT_LIT ['\''] ('i' | 'I') '8'
|
||||
INT16_LIT = INT_LIT ['\''] ('i' | 'I') '16'
|
||||
INT32_LIT = INT_LIT ['\''] ('i' | 'I') '32'
|
||||
INT64_LIT = INT_LIT ['\''] ('i' | 'I') '64'
|
||||
|
||||
UINT8_LIT = INT_LIT ['\''] ('u' | 'U')
|
||||
UINT8_LIT = INT_LIT ['\''] ('u' | 'U') '8'
|
||||
UINT16_LIT = INT_LIT ['\''] ('u' | 'U') '16'
|
||||
UINT32_LIT = INT_LIT ['\''] ('u' | 'U') '32'
|
||||
UINT64_LIT = INT_LIT ['\''] ('u' | 'U') '64'
|
||||
|
||||
exponent = ('e' | 'E' ) ['+' | '-'] digit ( ['_'] digit )*
|
||||
FLOAT_LIT = digit (['_'] digit)* (('.' (['_'] digit)* [exponent]) |exponent)
|
||||
FLOAT32_LIT = HEX_LIT '\'' ('f'|'F') '32'
|
||||
| (FLOAT_LIT | DEC_LIT | OCT_LIT | BIN_LIT) ['\''] ('f'|'F') '32'
|
||||
FLOAT64_LIT = HEX_LIT '\'' ('f'|'F') '64'
|
||||
| (FLOAT_LIT | DEC_LIT | OCT_LIT | BIN_LIT) ['\''] ('f'|'F') '64'
|
||||
|
||||
|
||||
As can be seen in the productions, numerical constants can contain underscores
|
||||
for readability. Integer and floating point literals may be given in decimal (no
|
||||
prefix), binary (prefix ``0b``), octal (prefix ``0o``) and hexadecimal
|
||||
(prefix ``0x``) notation.
|
||||
|
||||
There exists a literal for each numerical type that is
|
||||
defined. The suffix starting with an apostrophe ('\'') is called a
|
||||
`type suffix`:idx:. Literals without a type suffix are of the type ``int``,
|
||||
unless the literal contains a dot or ``E|e`` in which case it is of
|
||||
type ``float``. For notational convenience the apostrophe of a type suffix
|
||||
is optional if it is not ambiguous (only hexadecimal floating point literals
|
||||
with a type suffix can be ambiguous).
|
||||
|
||||
|
||||
The type suffixes are:
|
||||
|
||||
================= =========================
|
||||
Type Suffix Resulting type of literal
|
||||
================= =========================
|
||||
``'i8`` int8
|
||||
``'i16`` int16
|
||||
``'i32`` int32
|
||||
``'i64`` int64
|
||||
``'u`` uint
|
||||
``'u8`` uint8
|
||||
``'u16`` uint16
|
||||
``'u32`` uint32
|
||||
``'u64`` uint64
|
||||
``'f32`` float32
|
||||
``'f64`` float64
|
||||
================= =========================
|
||||
|
||||
Floating point literals may also be in binary, octal or hexadecimal
|
||||
notation:
|
||||
``0B0_10001110100_0000101001000111101011101111111011000101001101001001'f64``
|
||||
is approximately 1.72826e35 according to the IEEE floating point standard.
|
||||
|
||||
|
||||
Operators
|
||||
---------
|
||||
|
||||
In Nim one can define his own operators. An operator is any
|
||||
combination of the following characters::
|
||||
|
||||
= + - * / < >
|
||||
@ $ ~ & % |
|
||||
! ? ^ . : \
|
||||
|
||||
These keywords are also operators:
|
||||
``and or not xor shl shr div mod in notin is isnot of``.
|
||||
|
||||
`=`:tok:, `:`:tok:, `::`:tok: are not available as general operators; they
|
||||
are used for other notational purposes.
|
||||
|
||||
``*:`` is as a special case the two tokens `*`:tok: and `:`:tok:
|
||||
(to support ``var v*: T``).
|
||||
|
||||
|
||||
Other tokens
|
||||
------------
|
||||
|
||||
The following strings denote other tokens::
|
||||
|
||||
` ( ) { } [ ] , ; [. .] {. .} (. .)
|
||||
|
||||
|
||||
The `slice`:idx: operator `..`:tok: takes precedence over other tokens that
|
||||
contain a dot: `{..}`:tok: are the three tokens `{`:tok:, `..`:tok:, `}`:tok:
|
||||
and not the two tokens `{.`:tok:, `.}`:tok:.
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue