caseStmtMacros no longer experimental, experimental manual refactor (#19173)

* `caseStmtMacros` no longer experimental, experimental manual refactor

* Update doc/manual.rst

* apply review suggestions

* apply review

Co-authored-by: Andreas Rumpf <rumpf_a@web.de>

changelog.md

@@ -43,6 +43,8 @@
       x, y, z: int
     Baz = object
   ```
+- [Case statement macros](manual.html#macros-case-statement-macros) are no longer experimental,
+  meaning you no longer need to enable the experimental switch `caseStmtMacros` to use them. 
 
 ## Compiler changes
 

compiler/options.nim

@@ -195,7 +195,7 @@ type
     notnil,
     dynamicBindSym,
     forLoopMacros, # not experimental anymore; remains here for backwards compatibility
-    caseStmtMacros,
+    caseStmtMacros, # ditto
     codeReordering,
     compiletimeFFI,
       ## This requires building nim with `-d:nimHasLibFFI`

compiler/semstmts.nim

@@ -987,7 +987,7 @@ proc semCase(c: PContext, n: PNode; flags: TExprFlags): PNode =
   else:
     popCaseContext(c)
     closeScope(c)
-    if caseStmtMacros in c.features:
+    #if caseStmtMacros in c.features:
     result = handleCaseStmtMacro(c, n, flags)
     if result != nil:
       return result

doc/manual.rst

@@ -497,10 +497,10 @@ backtick token and the character literal. This special case ensures that a decla
 like ``proc `'customLiteral`(s: string)`` is valid. ``proc `'customLiteral`(s: string)``
 is the same as ``proc `'\''customLiteral`(s: string)``.
 
-See also `Custom Numeric Literals <#custom-numeric-literals>`_.
+See also `custom numeric literals <#custom-numeric-literals>`_.
 
 
-Numeric Literals
+Numeric literals
 ----------------
 
 Numeric literals have the form::
@@ -625,7 +625,7 @@ Hence: 0b10000000'u8 == 0x80'u8 == 128, but, 0b10000000'i8 == 0x80'i8 == -1
 instead of causing an overflow error.
 
 
-Custom Numeric Literals
+Custom numeric literals
 ~~~~~~~~~~~~~~~~~~~~~~~
 
 If the suffix is not predefined, then the suffix is assumed to be a call
@@ -700,26 +700,6 @@ contain a dot: `{..}` are the three tokens `{`:tok:, `..`:tok:, `}`:tok:
 and not the two tokens `{.`:tok:, `.}`:tok:.
 
 
-Unicode Operators
------------------
-
-Under the `--experimental:unicodeOperators` switch these Unicode operators are
-also parsed as operators::
-
-  ∙ ∘ × ★ ⊗ ⊘ ⊙ ⊛ ⊠ ⊡ ∩ ∧ ⊓   # same priority as * (multiplication)
-  ± ⊕ ⊖ ⊞ ⊟ ∪ ∨ ⊔             # same priority as + (addition)
-
-
-If enabled, Unicode operators can be combined with non-Unicode operator
-symbols. The usual precedence extensions then apply, for example, `⊠=` is an
-assignment like operator just like `*=` is.
-
-No Unicode normalization step is performed.
-
-**Note**: Due to parser limitations one **cannot** enable this feature via a
-pragma `{.experimental: "unicodeOperators".}` reliably.
-
-
 Syntax
 ======
 
@@ -1323,46 +1303,6 @@ as `MyEnum.value`:
 
 To implement bit fields with enums see `Bit fields <#set-type-bit-fields>`_
 
-Overloadable enum field names
------------------------------
-
-To be enabled via `{.experimental: "overloadableEnums".}`.
-
-Enum field names are overloadable much like routines. When an overloaded
-enum field is used, it produces a closed sym choice construct, here
-written as `(E|E)`.
-During overload resolution the right `E` is picked, if possible.
-For (array/object...) constructors the right `E` is picked, comparable to
-how `[byte(1), 2, 3]` works, one needs to use `[T.E, E2, E3]`. Ambiguous
-enum fields produce a static error:
-
-.. code-block:: nim
-    :test: "nim c $1"
-
-  {.experimental: "overloadableEnums".}
-
-  type
-    E1 = enum
-      value1,
-      value2
-    E2 = enum
-      value1,
-      value2 = 4
-
-  const
-    Lookuptable = [
-      E1.value1: "1",
-      value2: "2"
-    ]
-
-  proc p(e: E1) =
-    # disambiguation in 'case' statements:
-    case e
-    of value1: echo "A"
-    of value2: echo "B"
-
-  p value2
-
 
 String type
 -----------
@@ -1684,11 +1624,6 @@ must match the order of the tuple's definition. Different tuple-types are
 *equivalent* if they specify the same fields of the same type in the same
 order. The *names* of the fields also have to be the same.
 
-The assignment operator for tuples copies each component.
-The default assignment operator for objects copies each component. Overloading
-of the assignment operator is described `here
-<manual_experimental.html#type-bound-operations>`_.
-
 .. code-block:: nim
 
   type
@@ -1765,6 +1700,10 @@ introduce new object roots apart from `system.RootObj`.
     Student = ref object of Person # Error: inheritance only works with non-final objects
       id: int
 
+The assignment operator for tuples and objects copies each component.
+The methods to override this copying behavior are described `here
+<manual.html#procedures-type-bound-operations>`_.
+
 
 Object construction
 -------------------
@@ -2685,6 +2624,7 @@ Varargs matching
 
 See `Varargs <#types-varargs>`_.
 
+
 iterable
 --------
 
@@ -2725,6 +2665,24 @@ available. Let `p` be an overloaded symbol. These contexts are:
 
 As usual, ambiguous matches produce a compile-time error.
 
+Named argument overloading
+--------------------------
+
+Routines with the same type signature can be called individually if
+a parameter has different names between them.
+
+.. code-block:: Nim
+  proc foo(x: int) =
+    echo "Using x: ", x
+  proc foo(y: int) =
+    echo "Using y: ", y
+
+  foo(x = 2) # Using x: 2
+  foo(y = 2) # Using y: 2
+
+Not supplying the parameter name in such cases results in an
+ambiguity error.
+
 
 Statements and expressions
 ==========================
@@ -3836,8 +3794,8 @@ behavior inside loop bodies. See `closureScope
 <system.html#closureScope.t,untyped>`_ and `capture
 <sugar.html#capture.m,varargs[typed],untyped>`_ for details on how to change this behavior.
 
-Anonymous Procs
+Anonymous procedures
----------------
+--------------------
 
 Unnamed procedures can be used as lambda expressions to pass into other
 procedures:
@@ -3854,6 +3812,42 @@ executable code. The  `sugar <sugar.html>`_ module contains the `=>` macro
 which enables a more succinct syntax for anonymous procedures resembling
 lambdas as they are in languages like JavaScript, C#, etc.
 
+Do notation
+-----------
+
+As a special convenience notation that keeps most elements of a
+regular proc expression, the `do` keyword can be used to pass
+anonymous procedures to routines:
+
+.. code-block:: nim
+  var cities = @["Frankfurt", "Tokyo", "New York", "Kyiv"]
+
+  sort(cities) do (x, y: string) -> int:
+    cmp(x.len, y.len)
+
+  # Less parentheses using the method plus command syntax:
+  cities = cities.map do (x: string) -> string:
+    "City of " & x
+
+`do` is written after the parentheses enclosing the regular proc params.
+The proc expression represented by the `do` block is appended to the routine
+call as the last argument. In calls using the command syntax, the `do` block
+will bind to the immediately preceding expression rather than the command call.
+
+`do` with a parameter list corresponds to an anonymous `proc`, however
+`do` without parameters is treated as a normal statement list. This allows
+macros to receive both indented statement lists as an argument in inline
+calls, as well as a direct mirror of Nim's routine syntax.
+
+.. code-block:: nim
+  # Passing a statement list to an inline macro:
+  macroResults.add quote do:
+    if not `ex`:
+      echo `info`, ": Check failed: ", `expString`
+  
+  # Processing a routine definition in a macro:
+  rpc(router, "add") do (a, b: int) -> int:
+    result = a + b
 
 Func
 ----
@@ -5133,7 +5127,7 @@ code:
         deletedKeys: seq[bool]
 
 
-Type Classes
+Type classes
 ------------
 
 A type class is a special pseudo-type that can be used to match against
@@ -5863,7 +5857,15 @@ twice:
 While macros enable advanced compile-time code transformations, they
 cannot change Nim's syntax.
 
-Debug Example
+**Style note:** For code readability, it is best to use the least powerful
+programming construct that remains expressive. So the "check list" is:
+
+(1) Use an ordinary proc/iterator, if possible.
+(2) Else: Use a generic proc/iterator, if possible.
+(3) Else: Use a template, if possible.
+(4) Else: Use a macro.
+
+Debug example
 -------------
 
 The following example implements a powerful `debug` command that accepts a
@@ -5921,7 +5923,7 @@ constructor expression. This is why `debug` iterates over all of `args`'s
 children.
 
 
-BindSym
+bindSym
 -------
 
 The above `debug` macro relies on the fact that `write`, `writeLine` and
@@ -5970,43 +5972,38 @@ However, the symbols `write`, `writeLine` and `stdout` are already bound
 and are not looked up again. As the example shows, `bindSym` does work with
 overloaded symbols implicitly.
 
-Case-Of Macro
+Note that the symbol names passed to `bindSym` have to be constant. The
--------------
+experimental feature `dynamicBindSym` (`experimental manual
+<manual_experimental.html#dynamic-arguments-for-bindsym>`_)
+allows this value to be computed dynamically.
 
-In Nim, it is possible to have a macro with the syntax of a *case-of*
+Post-statement blocks
-expression just with the difference that all *of-branches* are passed to
+---------------------
-and processed by the macro implementation. It is then up the macro
+
-implementation to transform the *of-branches* into a valid Nim
+Macros can receive `of`, `elif`, `else`, `except`, `finally` and `do`
-statement. The following example should show how this feature could be
+blocks (including their different forms such as `do` with routine parameters)
-used for a lexical analyzer.
+as arguments if called in statement form.
 
 .. code-block:: nim
-  import std/macros
+  macro performWithUndo(task, undo: untyped) = ...
 
-  macro case_token(args: varargs[untyped]): untyped =
+  performWithUndo do:
-    echo args.treeRepr
+    # multiple-line block of code
-    # creates a lexical analyzer from regular expressions
+    # to perform the task
-    # ... (implementation is an exercise for the reader ;-)
+  do:
-    discard
+    # code to undo it
   
-  case_token: # this colon tells the parser it is a macro statement
+  let num = 12
-  of r"[A-Za-z_]+[A-Za-z_0-9]*":
+  # a single colon may be used if there is no initial block
-    return tkIdentifier
+  match (num mod 3, num mod 5):
-  of r"0-9+":
+  of (0, 0):
-    return tkInteger
+    echo "FizzBuzz"
-  of r"[\+\-\*\?]+":
+  of (0, _):
-    return tkOperator
+    echo "Fizz"
+  of (_, 0):
+    echo "Buzz"
   else:
-    return tkUnknown
+    echo num
-
-
-**Style note**: For code readability, it is best to use the least powerful
-programming construct that still suffices. So the "check list" is:
-
-(1) Use an ordinary proc/iterator, if possible.
-(2) Else: Use a generic proc/iterator, if possible.
-(3) Else: Use a template, if possible.
-(4) Else: Use a macro.
 
 
 For loop macro
@@ -6072,6 +6069,48 @@ Another example:
     echo a, " ", b
 
 
+Case statement macros
+---------------------
+
+Macros named `` `case` `` can provide implementations of `case` statements
+for certain types. The following is an example of such an implementation
+for tuples, leveraging the existing equality operator for tuples
+(as provided in `system.==`):
+
+.. code-block:: nim
+    :test: "nim c $1"
+  import std/macros
+
+  macro `case`(n: tuple): untyped =
+    result = newTree(nnkIfStmt)
+    let selector = n[0]
+    for i in 1 ..< n.len:
+      let it = n[i]
+      case it.kind
+      of nnkElse, nnkElifBranch, nnkElifExpr, nnkElseExpr:
+        result.add it
+      of nnkOfBranch:
+        for j in 0..it.len-2:
+          let cond = newCall("==", selector, it[j])
+          result.add newTree(nnkElifBranch, cond, it[^1])
+      else:
+        error "custom 'case' for tuple cannot handle this node", it
+
+  case ("foo", 78)
+  of ("foo", 78): echo "yes"
+  of ("bar", 88): echo "no"
+  else: discard
+
+`case` macros are subject to overload resolution. The type of the
+`case` statement's selector expression is matched against the type
+of the first argument of the `case` macro. Then the complete `case`
+statement is passed in place of the argument and the macro is evaluated.
+
+In other words, the macro needs to transform the full `case` statement
+but only the statement's selector expression is used to determine which
+macro to call.
+
+
 Special Types
 =============
 
@@ -6959,7 +6998,8 @@ experimental pragma
 The `experimental` pragma enables experimental language features. Depending
 on the concrete feature, this means that the feature is either considered
 too unstable for an otherwise stable release or that the future of the feature
-is uncertain (it may be removed at any time).
+is uncertain (it may be removed at any time). See the
+`experimental manual <manual_experimental.html>`_ for more details.
 
 Example:
 
@@ -7060,6 +7100,21 @@ alignment requirement of the type are ignored.
 This pragma has no effect on the JS backend.
 
 
+Noalias pragma
+==============
+
+Since version 1.4 of the Nim compiler, there is a `.noalias` annotation for variables
+and parameters. It is mapped directly to C/C++'s `restrict`:c: keyword and means that
+the underlying pointer is pointing to a unique location in memory, no other aliases to
+this location exist. It is *unchecked* that this alias restriction is followed. If the
+restriction is violated, the backend optimizer is free to miscompile the code.
+This is an **unsafe** language feature.
+
+Ideally in later versions of the language, the restriction will be enforced at
+compile time. (This is also why the name `noalias` was choosen instead of a more
+verbose name like `unsafeAssumeNoAlias`.)
+
+
 Volatile pragma
 ---------------
 The `volatile` pragma is for variables only. It declares the variable as
@@ -8032,3 +8087,137 @@ Threads and exceptions
 The interaction between threads and exceptions is simple: A *handled* exception
 in one thread cannot affect any other thread. However, an *unhandled* exception
 in one thread terminates the whole *process*.
+
+
+Guards and locks
+================
+
+Nim provides common low level concurrency mechanisms like locks, atomic
+intrinsics or condition variables.
+
+Nim significantly improves on the safety of these features via additional
+pragmas:
+
+1) A `guard`:idx: annotation is introduced to prevent data races.
+2) Every access of a guarded memory location needs to happen in an
+   appropriate `locks`:idx: statement.
+
+
+Guards and locks sections
+-------------------------
+
+Protecting global variables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Object fields and global variables can be annotated via a `guard` pragma:
+
+.. code-block:: nim
+
+  var glock: TLock
+  var gdata {.guard: glock.}: int
+
+The compiler then ensures that every access of `gdata` is within a `locks`
+section:
+
+.. code-block:: nim
+
+  proc invalid =
+    # invalid: unguarded access:
+    echo gdata
+
+  proc valid =
+    # valid access:
+    {.locks: [glock].}:
+      echo gdata
+
+Top level accesses to `gdata` are always allowed so that it can be initialized
+conveniently. It is *assumed* (but not enforced) that every top level statement
+is executed before any concurrent action happens.
+
+The `locks` section deliberately looks ugly because it has no runtime
+semantics and should not be used directly! It should only be used in templates
+that also implement some form of locking at runtime:
+
+.. code-block:: nim
+
+  template lock(a: TLock; body: untyped) =
+    pthread_mutex_lock(a)
+    {.locks: [a].}:
+      try:
+        body
+      finally:
+        pthread_mutex_unlock(a)
+
+
+The guard does not need to be of any particular type. It is flexible enough to
+model low level lockfree mechanisms:
+
+.. code-block:: nim
+
+  var dummyLock {.compileTime.}: int
+  var atomicCounter {.guard: dummyLock.}: int
+
+  template atomicRead(x): untyped =
+    {.locks: [dummyLock].}:
+      memoryReadBarrier()
+      x
+
+  echo atomicRead(atomicCounter)
+
+
+The `locks` pragma takes a list of lock expressions `locks: [a, b, ...]`
+in order to support *multi lock* statements. Why these are essential is
+explained in the `lock levels <#guards-and-locks-lock-levels>`_ section.
+
+
+Protecting general locations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The `guard` annotation can also be used to protect fields within an object.
+The guard then needs to be another field within the same object or a
+global variable.
+
+Since objects can reside on the heap or on the stack, this greatly enhances
+the expressivity of the language:
+
+.. code-block:: nim
+
+  type
+    ProtectedCounter = object
+      v {.guard: L.}: int
+      L: TLock
+
+  proc incCounters(counters: var openArray[ProtectedCounter]) =
+    for i in 0..counters.high:
+      lock counters[i].L:
+        inc counters[i].v
+
+The access to field `x.v` is allowed since its guard `x.L`  is active.
+After template expansion, this amounts to:
+
+.. code-block:: nim
+
+  proc incCounters(counters: var openArray[ProtectedCounter]) =
+    for i in 0..counters.high:
+      pthread_mutex_lock(counters[i].L)
+      {.locks: [counters[i].L].}:
+        try:
+          inc counters[i].v
+        finally:
+          pthread_mutex_unlock(counters[i].L)
+
+There is an analysis that checks that `counters[i].L` is the lock that
+corresponds to the protected location `counters[i].v`. This analysis is called
+`path analysis`:idx: because it deals with paths to locations
+like `obj.field[i].fieldB[j]`.
+
+The path analysis is **currently unsound**, but that doesn't make it useless.
+Two paths are considered equivalent if they are syntactically the same.
+
+This means the following compiles (for now) even though it really should not:
+
+.. code-block:: nim
+
+  {.locks: [a[i].L].}:
+    inc i
+    access a[i].v

lib/core/macros.nim

@@ -465,12 +465,7 @@ proc bindSym*(ident: string | NimNode, rule: BindSymRule = brClosed): NimNode {.
   ## If `rule == brForceOpen` always an `nnkOpenSymChoice` tree is
   ## returned even if the symbol is not ambiguous.
   ##
-  ## Experimental feature:
+  ## See the `manual <manual.html#macros-bindsym>`_ for more details.
-  ## use {.experimental: "dynamicBindSym".} to activate it.
-  ## If called from template / regular code, `ident` and `rule` must be
-  ## constant expression / literal value.
-  ## If called from macros / compile time procs / static blocks,
-  ## `ident` and `rule` can be VM computed value.
 
 proc genSym*(kind: NimSymKind = nskLet; ident = ""): NimNode {.
   magic: "NGenSym", noSideEffect.}

tests/macros/tcasestmtmacro.nim

@@ -4,8 +4,6 @@ yes
 '''
 """
 
-{.experimental: "caseStmtMacros".}
-
 import macros
 
 macro `case`(n: tuple): untyped =