Nim/lib/pure/sugar.nim
c-blake a0b33f9408
Add hashWangYi1 (#13823)
* Unwind just the "pseudorandom probing" (whole hash-code-keyed variable
stride double hashing) part of recent sets & tables changes (which has
still been causing bugs over a month later (e.g., two days ago
https://github.com/nim-lang/Nim/issues/13794) as well as still having
several "figure this out" implementation question comments in them (see
just diffs of this PR).

This topic has been discussed in many places:
  https://github.com/nim-lang/Nim/issues/13393
  https://github.com/nim-lang/Nim/pull/13418
  https://github.com/nim-lang/Nim/pull/13440
  https://github.com/nim-lang/Nim/issues/13794

Alternative/non-mandatory stronger integer hashes (or vice-versa opt-in
identity hashes) are a better solution that is more general (no illusion
of one hard-coded sequence solving all problems) while retaining the
virtues of linear probing such as cache obliviousness and age-less tables
under delete-heavy workloads (still untested after a month of this change).

The only real solution for truly adversarial keys is a hash keyed off of
data unobservable to attackers.  That all fits better with a few families
of user-pluggable/define-switchable hashes which can be provided in a
separate PR more about `hashes.nim`.

This PR carefully preserves the better (but still hard coded!) probing
of the  `intsets` and other recent fixes like `move` annotations, hash
order invariant tests, `intsets.missingOrExcl` fixing, and the move of
`rightSize` into `hashcommon.nim`.

* Fix `data.len` -> `dataLen` problem.

* This is an alternate resolution to https://github.com/nim-lang/Nim/issues/13393
(which arguably could be resolved outside the stdlib).

Add version1 of Wang Yi's hash specialized to 8 byte integers.  This gives
simple help to users having trouble with overly colliding hash(key)s.  I.e.,
  A) `import hashes; proc hash(x: myInt): Hash = hashWangYi1(int(x))`
      in the instantiation context of a `HashSet` or `Table`
or
  B) more globally, compile with `nim c -d:hashWangYi1`.

No hash can be all things to all use cases, but this one is A) vetted to
scramble well by the SMHasher test suite (a necessarily limited but far
more thorough test than prior proposals here), B) only a few ALU ops on
many common CPUs, and C) possesses an easy via "grade school multi-digit
multiplication" fall back for weaker deployment contexts.

Some people might want to stampede ahead unbridled, but my view is that a
good plan is to
  A) include this in the stdlib for a release or three to let people try it
     on various key sets nim-core could realistically never access/test
     (maybe mentioning it in the changelog so people actually try it out),
  B) have them report problems (if any),
  C) if all seems good, make the stdlib more novice friendly by adding
     `hashIdentity(x)=x` and changing the default `hash() = hashWangYi1`
     with some `when defined` rearranging so users can `-d:hashIdentity`
     if they want the old behavior back.
This plan is compatible with any number of competing integer hashes if
people want to add them.  I would strongly recommend they all *at least*
pass the SMHasher suite since the idea here is to become more friendly to
novices who do not generally understand hashing failure modes.

* Re-organize to work around `when nimvm` limitations; Add some tests; Add
a changelog.md entry.

* Add less than 64-bit CPU when fork.

* Fix decl instead of call typo.

* First attempt at fixing range error on 32-bit platforms; Still do the
arithmetic in doubled up 64-bit, but truncate the hash to the lower
32-bits, but then still return `uint64` to be the same.  So, type
correct but truncated hash value.  Update `thashes.nim` as well.

* A second try at making 32-bit mode CI work.

* Use a more systematic identifier convention than Wang Yi's code.

* Fix test that was wrong for as long as `toHashSet` used `rightSize` (a
very long time, I think).  `$a`/`$b` depend on iteration order which
varies with table range reduced hash order which varies with range for
some `hash()`.  With 3 elements, 3!=6 is small and we've just gotten
lucky with past experimental `hash()` changes.  An alternate fix here
would be to not stringify but use the HashSet operators, but it is not
clear that doesn't alter the "spirit" of the test.

* Fix another stringified test depending upon hash order.

* Oops - revert the string-keyed test.

* Fix another stringify test depending on hash order.

* Add a better than always zero `defined(js)` branch.

* It turns out to be easy to just work all in `BigInt` inside JS and thus
guarantee the same low order bits of output hashes (for `isSafeInteger`
input numbers).  Since `hashWangYi1` output bits are equally random in
all their bits, this means that tables will be safely scrambled for table
sizes up to 2**32 or 4 gigaentries which is probably fine, as long as the
integer keys are all < 2**53 (also likely fine).  (I'm unsure why the
infidelity with C/C++ back ends cut off is 32, not 53 bits.)

Since HashSet & Table only use the low order bits, a quick corollary of
this is that `$` on most int-keyed sets/tables will be the same in all
the various back ends which seems a nice-to-have trait.

* These string hash tests fail for me locally.  Maybe this is what causes
the CI hang for testament pcat collections?

* Oops. That failure was from me manually patching string hash in hashes.  Revert.

* Import more test improvements from https://github.com/nim-lang/Nim/pull/13410

* Fix bug where I swapped order when reverting the test.  Ack.

* Oh, just accept either order like more and more hash tests.

* Iterate in the same order.

* `return` inside `emit` made us skip `popFrame` causing weird troubles.

* Oops - do Windows branch also.

* `nimV1hash` -> multiply-mnemonic, type-scoped `nimIntHash1` (mnemonic
resolutions are "1 == identity", 1 for Nim Version 1, 1 for
first/simplest/fastest in a series of possibilities.  Should be very
easy to remember.)

* Re-organize `when nimvm` logic to be a strict `when`-`else`.

* Merge other changes.

* Lift constants to a common area.

* Fall back to identity hash when `BigInt` is unavailable.

* Increase timeout slightly (probably just real-time perturbation of CI
system performance).
2020-04-15 20:11:18 +02:00

371 lines
11 KiB
Nim

#
#
# Nim's Runtime Library
# (c) Copyright 2015 Dominik Picheta
#
# See the file "copying.txt", included in this
# distribution, for details about the copyright.
#
## This module implements nice syntactic sugar based on Nim's
## macro system.
include system/inclrtl
import macros
import typetraits
proc createProcType(p, b: NimNode): NimNode {.compileTime.} =
#echo treeRepr(p)
#echo treeRepr(b)
result = newNimNode(nnkProcTy)
var formalParams = newNimNode(nnkFormalParams)
formalParams.add b
case p.kind
of nnkPar, nnkTupleConstr:
for i in 0 ..< p.len:
let ident = p[i]
var identDefs = newNimNode(nnkIdentDefs)
case ident.kind
of nnkExprColonExpr:
identDefs.add ident[0]
identDefs.add ident[1]
else:
identDefs.add newIdentNode("i" & $i)
identDefs.add(ident)
identDefs.add newEmptyNode()
formalParams.add identDefs
else:
var identDefs = newNimNode(nnkIdentDefs)
identDefs.add newIdentNode("i0")
identDefs.add(p)
identDefs.add newEmptyNode()
formalParams.add identDefs
result.add formalParams
result.add newEmptyNode()
#echo(treeRepr(result))
#echo(result.toStrLit())
macro `=>`*(p, b: untyped): untyped =
## Syntax sugar for anonymous procedures.
##
## .. code-block:: nim
##
## proc passTwoAndTwo(f: (int, int) -> int): int =
## f(2, 2)
##
## passTwoAndTwo((x, y) => x + y) # 4
#echo treeRepr(p)
#echo(treeRepr(b))
var params: seq[NimNode] = @[newIdentNode("auto")]
case p.kind
of nnkPar, nnkTupleConstr:
for c in children(p):
var identDefs = newNimNode(nnkIdentDefs)
case c.kind
of nnkExprColonExpr:
identDefs.add(c[0])
identDefs.add(c[1])
identDefs.add(newEmptyNode())
of nnkIdent:
identDefs.add(c)
identDefs.add(newIdentNode("auto"))
identDefs.add(newEmptyNode())
of nnkInfix:
if c[0].kind == nnkIdent and c[0].ident == !"->":
var procTy = createProcType(c[1], c[2])
params[0] = procTy[0][0]
for i in 1 ..< procTy[0].len:
params.add(procTy[0][i])
else:
error("Expected proc type (->) got (" & $c[0].ident & ").")
break
else:
echo treeRepr c
error("Incorrect procedure parameter list.")
params.add(identDefs)
of nnkIdent:
var identDefs = newNimNode(nnkIdentDefs)
identDefs.add(p)
identDefs.add(newIdentNode("auto"))
identDefs.add(newEmptyNode())
params.add(identDefs)
of nnkInfix:
if p[0].kind == nnkIdent and p[0].ident == !"->":
var procTy = createProcType(p[1], p[2])
params[0] = procTy[0][0]
for i in 1 ..< procTy[0].len:
params.add(procTy[0][i])
else:
error("Expected proc type (->) got (" & $p[0].ident & ").")
else:
error("Incorrect procedure parameter list.")
result = newProc(params = params, body = b, procType = nnkLambda)
#echo(result.treeRepr)
#echo(result.toStrLit())
#return result # TODO: Bug?
macro `->`*(p, b: untyped): untyped =
## Syntax sugar for procedure types.
##
## .. code-block:: nim
##
## proc pass2(f: (float, float) -> float): float =
## f(2, 2)
##
## # is the same as:
##
## proc pass2(f: proc (x, y: float): float): float =
## f(2, 2)
result = createProcType(p, b)
macro dump*(x: typed): untyped =
## Dumps the content of an expression, useful for debugging.
## It accepts any expression and prints a textual representation
## of the tree representing the expression - as it would appear in
## source code - together with the value of the expression.
##
## As an example,
##
## .. code-block:: nim
## let
## x = 10
## y = 20
## dump(x + y)
##
## will print ``x + y = 30``.
let s = x.toStrLit
let r = quote do:
debugEcho `s`, " = ", `x`
return r
# TODO: consider exporting this in macros.nim
proc freshIdentNodes(ast: NimNode): NimNode =
# Replace NimIdent and NimSym by a fresh ident node
# see also https://github.com/nim-lang/Nim/pull/8531#issuecomment-410436458
proc inspect(node: NimNode): NimNode =
case node.kind:
of nnkIdent, nnkSym:
result = ident($node)
of nnkEmpty, nnkLiterals:
result = node
else:
result = node.kind.newTree()
for child in node:
result.add inspect(child)
result = inspect(ast)
template distinctBase*(T: typedesc): typedesc {.deprecated: "use distinctBase from typetraits instead".} =
## reverses ``type T = distinct A``; works recursively.
typetraits.distinctBase(T)
macro capture*(locals: varargs[typed], body: untyped): untyped {.since: (1, 1).} =
## Useful when creating a closure in a loop to capture some local loop variables
## by their current iteration values. Example:
##
## .. code-block:: Nim
## import strformat, sequtils, sugar
## var myClosure : proc()
## for i in 5..7:
## for j in 7..9:
## if i * j == 42:
## capture i, j:
## myClosure = proc () = echo fmt"{i} * {j} = 42"
## myClosure() # output: 6 * 7 == 42
## let m = @[proc (s: string): string = "to " & s, proc (s: string): string = "not to " & s]
## var l = m.mapIt(capture(it, proc (s: string): string = it(s)))
## let r = l.mapIt(it("be"))
## echo r[0] & ", or " & r[1] # output: to be, or not to be
var params = @[newIdentNode("auto")]
let locals = if locals.len == 1 and locals[0].kind == nnkBracket: locals[0]
else: locals
for arg in locals:
params.add(newIdentDefs(ident(arg.strVal), freshIdentNodes getTypeInst arg))
result = newNimNode(nnkCall)
result.add(newProc(newEmptyNode(), params, body, nnkProcDef))
for arg in locals: result.add(arg)
when (NimMajor, NimMinor) >= (1, 1):
import std / private / underscored_calls
macro dup*[T](arg: T, calls: varargs[untyped]): T =
## Turns an `in-place`:idx: algorithm into one that works on
## a copy and returns this copy.
## **Since**: Version 1.2.
runnableExamples:
import algorithm
var a = @[1, 2, 3, 4, 5, 6, 7, 8, 9]
doAssert a.dup(sort) == sorted(a)
# Chaining:
var aCopy = a
aCopy.insert(10)
doAssert a.dup(insert(10), sort) == sorted(aCopy)
var s1 = "abc"
var s2 = "xyz"
doAssert s1 & s2 == s1.dup(&= s2)
result = newNimNode(nnkStmtListExpr, arg)
let tmp = genSym(nskVar, "dupResult")
result.add newVarStmt(tmp, arg)
underscoredCalls(result, calls, tmp)
result.add tmp
proc transLastStmt(n, res, bracketExpr: NimNode): (NimNode, NimNode, NimNode) {.since: (1, 1).} =
# Looks for the last statement of the last statement, etc...
case n.kind
of nnkIfExpr, nnkIfStmt, nnkTryStmt, nnkCaseStmt:
result[0] = copyNimTree(n)
result[1] = copyNimTree(n)
result[2] = copyNimTree(n)
for i in ord(n.kind == nnkCaseStmt)..<n.len:
(result[0][i], result[1][^1], result[2][^1]) = transLastStmt(n[i], res, bracketExpr)
of nnkStmtList, nnkStmtListExpr, nnkBlockStmt, nnkBlockExpr, nnkWhileStmt,
nnkForStmt, nnkElifBranch, nnkElse, nnkElifExpr, nnkOfBranch, nnkExceptBranch:
result[0] = copyNimTree(n)
result[1] = copyNimTree(n)
result[2] = copyNimTree(n)
if n.len >= 1:
(result[0][^1], result[1][^1], result[2][^1]) = transLastStmt(n[^1], res, bracketExpr)
of nnkTableConstr:
result[1] = n[0][0]
result[2] = n[0][1]
if bracketExpr.len == 1:
bracketExpr.add([newCall(bindSym"typeof", newEmptyNode()), newCall(
bindSym"typeof", newEmptyNode())])
template adder(res, k, v) = res[k] = v
result[0] = getAst(adder(res, n[0][0], n[0][1]))
of nnkCurly:
result[2] = n[0]
if bracketExpr.len == 1:
bracketExpr.add(newCall(bindSym"typeof", newEmptyNode()))
template adder(res, v) = res.incl(v)
result[0] = getAst(adder(res, n[0]))
else:
result[2] = n
if bracketExpr.len == 1:
bracketExpr.add(newCall(bindSym"typeof", newEmptyNode()))
template adder(res, v) = res.add(v)
result[0] = getAst(adder(res, n))
macro collect*(init, body: untyped): untyped {.since: (1, 1).} =
## Comprehension for seq/set/table collections. ``init`` is
## the init call, and so custom collections are supported.
##
## The last statement of ``body`` has special syntax that specifies
## the collection's add operation. Use ``{e}`` for set's ``incl``,
## ``{k: v}`` for table's ``[]=`` and ``e`` for seq's ``add``.
##
## The ``init`` proc can be called with any number of arguments,
## i.e. ``initTable(initialSize)``.
runnableExamples:
import sets, tables
let data = @["bird", "word"]
## seq:
let k = collect(newSeq):
for i, d in data.pairs:
if i mod 2 == 0: d
assert k == @["bird"]
## seq with initialSize:
let x = collect(newSeqOfCap(4)):
for i, d in data.pairs:
if i mod 2 == 0: d
assert x == @["bird"]
## HashSet:
let y = initHashSet.collect:
for d in data.items: {d}
assert y == data.toHashSet
## Table:
let z = collect(initTable(2)):
for i, d in data.pairs: {i: d}
assert z == {0: "bird", 1: "word"}.toTable
# analyse the body, find the deepest expression 'it' and replace it via
# 'result.add it'
let res = genSym(nskVar, "collectResult")
expectKind init, {nnkCall, nnkIdent, nnkSym}
let bracketExpr = newTree(nnkBracketExpr,
if init.kind == nnkCall: init[0] else: init)
let (resBody, keyType, valueType) = transLastStmt(body, res, bracketExpr)
if bracketExpr.len == 3:
bracketExpr[1][1] = keyType
bracketExpr[2][1] = valueType
else:
bracketExpr[1][1] = valueType
let call = newTree(nnkCall, bracketExpr)
if init.kind == nnkCall:
for i in 1 ..< init.len:
call.add init[i]
result = newTree(nnkStmtListExpr, newVarStmt(res, call), resBody, res)
when isMainModule:
since (1, 1):
import algorithm
var a = @[1, 2, 3, 4, 5, 6, 7, 8, 9]
doAssert dup(a, sort(_)) == sorted(a)
doAssert a.dup(sort) == sorted(a)
#Chaining:
var aCopy = a
aCopy.insert(10)
doAssert a.dup(insert(10)).dup(sort()) == sorted(aCopy)
import random
const b = @[0, 1, 2]
let c = b.dup shuffle()
doAssert c[0] == 1
doAssert c[1] == 0
#test collect
import sets, tables
let data = @["bird", "word"] # if this gets stuck in your head, its not my fault
assert collect(newSeq, for (i, d) in data.pairs: (if i mod 2 == 0: d)) == @["bird"]
assert collect(initTable(2), for (i, d) in data.pairs: {i: d}) == {0: "bird",
1: "word"}.toTable
assert initHashSet.collect(for d in data.items: {d}) == data.toHashSet
let x = collect(newSeqOfCap(4)):
for (i, d) in data.pairs:
if i mod 2 == 0: d
assert x == @["bird"]
# bug #12874
let bug1 = collect(
newSeq,
for (i, d) in data.pairs:(
block:
if i mod 2 == 0:
d
else:
d & d
)
)
assert bug1 == @["bird", "wordword"]
import strutils
let y = collect(newSeq):
for (i, d) in data.pairs:
try: parseInt(d) except: 0
assert y == @[0, 0]
let z = collect(newSeq):
for (i, d) in data.pairs:
case d
of "bird": "word"
else: d
assert z == @["word", "word"]