jyapayne/llvmpy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

misc doc editing

Browse source
This commit is contained in:
Maggie Mari 2012-08-20 13:26:37 -05:00
commit 1b254f48b5
1 changed files with 285 additions and 182 deletions
Download patch file Download diff file Expand all files Collapse all files

467
docs/source/doc/kaleidoscope/PythonLangImpl6.rst
View file

@ -24,7 +24,7 @@ is good or bad. In this tutorial we'll assume that it is okay to use
this as a way to show some interesting parsing techniques. this as a way to show some interesting parsing techniques.
At the end of this tutorial, we'll run through an example Kaleidoscope At the end of this tutorial, we'll run through an example Kaleidoscope
application that `renders the Mandelbrot set <#example>`_. This gives an application that :ref:`renders the Mandelbrot set <example>`. This gives an
example of what you can build with Kaleidoscope and its feature set. example of what you can build with Kaleidoscope and its feature set.
User-defined Operators: the Idea User-defined Operators: the Idea
@ -53,17 +53,29 @@ binary operators. An example of this is:
.. code-block:: python .. code-block:: python
# Logical unary not. def unary!(v) if v then 0 # Logical unary not.
else 1 def unary!(v)
if v then
0
else
1
# Define > with the same precedence as <. # Define > with the same precedence as <.
def binary> 10 (LHS RHS) RHS < LHS def binary> 10 (LHS RHS)
RHS < LHS
# Binary "logical or", (note that it does not "short circuit"). # Binary "logical or", (note that it does not "short circuit").
def binary| 5 (LHS RHS) if LHS then 1 else if RHS then 1 else 0 def binary| 5 (LHS RHS)
if LHS then
1
else if RHS then
1
else
0
# Define = with slightly lower precedence than relationals. # Define = with slightly lower precedence than relationals.
def binary= 9 (LHS RHS) !(LHS < RHS | LHS > RHS) def binary= 9 (LHS RHS)
!(LHS < RHS | LHS > RHS)
@ -87,40 +99,60 @@ keywords:
.. code-block:: python .. code-block:: python
class InToken(object): pass class class InToken(object):
BinaryToken(object): pass class UnaryToken(object): pass ... def pass
Tokenize(string): ... elif identifier == 'in': yield InToken() elif class BinaryToken(object):
identifier == 'binary': yield BinaryToken() elif identifier == 'unary': pass
yield UnaryToken() else: yield IdentifierToken(identifier) class UnaryToken(object):
pass
...
def Tokenize(string):
...
elif identifier == 'in':
yield InToken()
elif identifier == 'binary':
yield BinaryToken()
elif identifier == 'unary':
yield UnaryToken()
else:
yield IdentifierToken(identifier)
This just adds lexer support for the unary and binary keywords, like we This just adds lexer support for the unary and binary keywords, like we
did in `previous chapters <PythonLangImpl5.html#iflexer>`_. One nice did in `previous chapters <PythonLangImpl5.html#iflexer>`_. One nice
thing about our current AST, is that we represent binary operators with thing about our current AST, is that we represent binary operators with
full generalisation by using their ASCII code as the opcode. For our full generalisation by using their ASCII code as the opcode. For our
extended operators, we'll use this same representation, so we don't need extended operators, we'll use this same representation, so we don't need
any new AST or parser support. any new AST or parser support.
On the other hand, we have to be able to represent the definitions of
these new operators, in the "def binary| 5" part of the function
definition. In our grammar so far, the "name" for the function
definition is parsed as the "prototype" production and into the
``PrototypeNode``. To represent our new user-defined operators as
prototypes, we have to extend the ``PrototypeNode`` like this:
.. code-block:: python
# This class represents the "prototype" for a function, which captures its name,
# and its argument names (thus implicitly the number of arguments the function
# takes), as well as if it is an operator.
class PrototypeNode(object):
On the other hand, we have to be able to represent the definitions of def __init__(self, name, args, is_operator=False, precedence=0):
these new operators, in the "def binary| 5" part of the function self.name = name
definition. In our grammar so far, the "name" for the function self.args = args
definition is parsed as the "prototype" production and into the self.is_operator = is_operator
``PrototypeNode``. To represent our new user-defined operators as self.precedence = precedence
prototypes, we have to extend the ``PrototypeNode`` like this:
# This class represents the "prototype" for a def IsBinaryOp(self):
function, which captures its name, # and its argument names (thus return self.is_operator and len(self.args) == 2
implicitly the number of arguments the function # takes), as well as if
it is an operator. class PrototypeNode(object): def GetOperatorName(self):
assert self.is_operator
def __init__(self, name, args, is_operator=False, precedence=0): return self.name[-1]
self.name = name self.args = args self.is_operator = is_operator
self.precedence = precedence def CodeGen(self):
...
def IsBinaryOp(self): return self.is_operator and len(self.args) == 2
def GetOperatorName(self): assert self.is_operator return self.name[-1]
def CodeGen(self): ...
@ -134,42 +166,51 @@ user-defined operator, we need to parse it:
.. code-block:: python .. code-block:: python
# prototype # ::= id '(' id* ')' # ::= binary # prototype
LETTER number? (id, id) # ::= unary LETTER (id) def # ::= id '(' id* ')'
ParsePrototype(self): precedence = None if isinstance(self.current, # ::= binary LETTER number? (id, id)
IdentifierToken): kind = 'normal' function_name = self.current.name # ::= unary LETTER (id)
self.Next() # eat function name. elif isinstance(self.current, def ParsePrototype(self):
BinaryToken): kind = 'binary' self.Next() # eat 'binary'. if not precedence = None
isinstance(self.current, CharacterToken): raise RuntimeError('Expected if isinstance(self.current, IdentifierToken):
an operator after "binary".') function_name = 'binary' + kind = 'normal'
self.current.char self.Next() # eat the operator. if function_name = self.current.name
isinstance(self.current, NumberToken): if not 1 <= self.current.value <= self.Next() # eat function name.
100: raise RuntimeError('Invalid precedence: must be in range [1, elif isinstance(self.current, BinaryToken):
100].') precedence = self.current.value self.Next() # eat the kind = 'binary'
precedence. else: raise RuntimeError('Expected function name, "unary" or self.Next() # eat 'binary'.
"binary" in ' 'prototype.') if not isinstance(self.current, CharacterToken):
raise RuntimeError('Expected an operator after "binary".')
:: function_name = 'binary' + self.current.char
self.Next() # eat the operator.
if self.current != CharacterToken('('): if isinstance(self.current, NumberToken):
raise RuntimeError('Expected "(" in prototype.') if not 1 <= self.current.value <= 100:
self.Next() # eat '('. raise RuntimeError('Invalid precedence: must be in range [1, 100].')
precedence = self.current.value
arg_names = [] self.Next() # eat the precedence.
while isinstance(self.current, IdentifierToken): else:
arg_names.append(self.current.name) raise RuntimeError('Expected function name, "unary" or "binary" in '
self.Next() 'prototype.')
if self.current != CharacterToken(')'): if self.current != CharacterToken('('):
raise RuntimeError('Expected ")" in prototype.') raise RuntimeError('Expected "(" in prototype.')
self.Next() # eat '('.
# Success.
self.Next() # eat ')'. arg_names = []
while isinstance(self.current, IdentifierToken):
if kind == 'binary' and len(arg_names) != 2: arg_names.append(self.current.name)
raise RuntimeError('Invalid number of arguments for a binary operator.') self.Next()
return PrototypeNode(function_name, arg_names, kind != 'normal', precedence) if self.current != CharacterToken(')'):
raise RuntimeError('Expected ")" in prototype.')
# Success.
self.Next() # eat ')'.
if kind == 'binary' and len(arg_names) != 2:
raise RuntimeError('Invalid number of arguments for a binary operator.')
return PrototypeNode(function_name, arg_names, kind != 'normal', precedence)
@ -189,24 +230,23 @@ default case for our existing binary operator node:
.. code-block:: python .. code-block:: python
def CodeGen(self): left = self.left.CodeGen() def CodeGen(self):
right = self.right.CodeGen() left = self.left.CodeGen()
right = self.right.CodeGen()
::
if self.operator == '+':
if self.operator == '+': return g_llvm_builder.fadd(left, right, 'addtmp')
return g_llvm_builder.fadd(left, right, 'addtmp') elif self.operator == '-':
elif self.operator == '-': return g_llvm_builder.fsub(left, right, 'subtmp')
return g_llvm_builder.fsub(left, right, 'subtmp') elif self.operator == '*':
elif self.operator == '*': return g_llvm_builder.fmul(left, right, 'multmp')
return g_llvm_builder.fmul(left, right, 'multmp') elif self.operator == '<':
elif self.operator == '<': result = g_llvm_builder.fcmp(FCMP_ULT, left, right, 'cmptmp')
result = g_llvm_builder.fcmp(FCMP_ULT, left, right, 'cmptmp') # Convert bool 0 or 1 to double 0.0 or 1.0.
# Convert bool 0 or 1 to double 0.0 or 1.0. return g_llvm_builder.uitofp(result, Type.double(), 'booltmp')
return g_llvm_builder.uitofp(result, Type.double(), 'booltmp') else:
else: function = g_llvm_module.get_function_named('binary' + self.operator)
function = g_llvm_module.get_function_named('binary' + self.operator) return g_llvm_builder.call(function, [left, right], 'binop')
return g_llvm_builder.call(function, [left, right], 'binop')
@ -226,31 +266,38 @@ whenever we define a new binary operator:
.. code-block:: python .. code-block:: python
# The binary operator precedence chart. # The binary operator precedence chart.
g_binop_precedence = {} ... class FunctionNode(object): ... def g_binop_precedence = {}
CodeGen(self): ... # Create a function object. function = ...
self.prototype.CodeGen() class FunctionNode(object):
...
def CodeGen(self):
...
# Create a function object.
function = self.prototype.CodeGen()
# If this is a binary operator, install its precedence.
if self.prototype.IsBinaryOp():
operator = self.prototype.GetOperatorName()
g_binop_precedence[operator] = self.prototype.precedence
...
# Finish off the function.
try:
...
except:
function.delete()
if self.prototype.IsBinaryOp():
del g_binop_precedence[self.prototype.GetOperatorName()]
raise
return function
:: ...
def main():
# If this is a binary operator, install its precedence. ...
if self.prototype.IsBinaryOp(): g_binop_precedence['<'] = 10
operator = self.prototype.GetOperatorName() g_binop_precedence['+'] = 20 g_binop_precedence['-'] = 20
g_binop_precedence[operator] = self.prototype.precedence g_binop_precedence['*'] = 40
... ...
# Finish off the function.
try:
...
except:
function.delete()
if self.prototype.IsBinaryOp():
del g_binop_precedence[self.prototype.GetOperatorName()]
raise
return function
... def main(): ... g_binop_precedence['<'] = 10
g_binop_precedence['+'] = 20 g_binop_precedence['-'] = 20
g_binop_precedence['*'] = 40 ...
@ -276,13 +323,15 @@ that, we need an AST node:
.. code-block:: python .. code-block:: python
# Expression class for a unary operator. class # Expression class for a unary operator.
UnaryExpressionNode(ExpressionNode): class UnaryExpressionNode(ExpressionNode):
def __init__(self, operator, operand): self.operator = operator def __init__(self, operator, operand):
self.operand = operand self.operator = operator
self.operand = operand
def CodeGen(self): ...
def CodeGen(self):
...
@ -294,18 +343,17 @@ simple: we'll add a new function to do it:
.. code-block:: python .. code-block:: python
# unary ::= primary | unary_operator unary def # unary ::= primary | unary_operator unary
ParseUnary(self): # If the current token is not an operator, it must be def ParseUnary(self):
a primary expression. if (not isinstance(self.current, CharacterToken) # If the current token is not an operator, it must be a primary expression.
or self.current in [CharacterToken('('), CharacterToken(',')]): return if (not isinstance(self.current, CharacterToken) or
self.ParsePrimary() self.current in [CharacterToken('('), CharacterToken(',')]):
return self.ParsePrimary()
::
# If this is a unary operator, read it.
# If this is a unary operator, read it. operator = self.current.chara
operator = self.current.char self.Next() # eat the operator.
self.Next() # eat the operator. return UnaryExpressionNode(operator, self.ParseUnary())
return UnaryExpressionNode(operator, self.ParseUnary())
@ -325,12 +373,17 @@ call ParseUnary instead:
.. code-block:: python .. code-block:: python
# binoprhs ::= (binary_operator unary)* def # binoprhs ::= (binary_operator unary)*
ParseBinOpRHS(self, left, left_precedence): ... # Parse the unary def ParseBinOpRHS(self, left, left_precedence):
expression after the binary operator. right = self.ParseUnary() ... ...
# Parse the unary expression after the binary operator.
right = self.ParseUnary()
...
# expression ::= unary binoprhs def ParseExpression(self): left = # expression ::= unary binoprhs
self.ParseUnary() return self.ParseBinOpRHS(left, 0) def ParseExpression(self):
left = self.ParseUnary()
return self.ParseBinOpRHS(left, 0)
@ -342,23 +395,33 @@ operator code above with:
.. code-block:: python .. code-block:: python
# prototype # ::= id '(' id* ')' # ::= binary # prototype
LETTER number? (id, id) # ::= unary LETTER (id) def # ::= id '(' id* ')'
ParsePrototype(self): precedence = None if isinstance(self.current, # ::= binary LETTER number? (id, id)
IdentifierToken): ... elif isinstance(self.current, UnaryToken): kind = # ::= unary LETTER (id)
'unary' self.Next() # eat 'unary'. if not isinstance(self.current, def ParsePrototype(self):
CharacterToken): raise RuntimeError('Expected an operator after precedence = None
"unary".') function_name = 'unary' + self.current.char self.Next() # if isinstance(self.current, IdentifierToken):
eat the operator. elif isinstance(self.current, BinaryToken): ... else: ...
raise RuntimeError('Expected function name, "unary" or "binary" in ' elif isinstance(self.current, UnaryToken):
'prototype.') ... if kind == 'unary' and len(arg_names) != 1: raise kind = 'unary'
RuntimeError('Invalid number of arguments for a unary operator.') elif self.Next() # eat 'unary'.
kind == 'binary' and len(arg_names) != 2: raise RuntimeError('Invalid if not isinstance(self.current, CharacterToken):
number of arguments for a binary operator.') raise RuntimeError('Expected an operator after "unary".')
function_name = 'unary' + self.current.char
:: self.Next() #eat the operator.
elif isinstance(self.current, BinaryToken):
return PrototypeNode(function_name, arg_names, kind != 'normal', precedence) ...
else:
raise RuntimeError('Expected function name, "unary" or "binary" in '
'prototype.')
...
if kind == 'unary' and len(arg_names) != 1:
raise RuntimeError('Invalid number of arguments for a unary operator.')
elif kind == 'binary' and len(arg_names) != 2:
raise RuntimeError('Invalid number of arguments for a binary operator.')
return PrototypeNode(function_name, arg_names, kind != 'normal', precedence)
@ -372,10 +435,12 @@ unary operators. It looks like this:
.. code-block:: python .. code-block:: python
class UnaryExpressionNode(ExpressionNode): ... class UnaryExpressionNode(ExpressionNode):
def CodeGen(self): operand = self.operand.CodeGen() function = ...
g_llvm_module.get_function_named('unary' + self.operator) return def CodeGen(self):
g_llvm_builder.call(function, [operand], 'unop') operand = self.operand.CodeGen()
function = g_llvm_module.get_function_named('unary' + self.operator)
return g_llvm_builder.call(function, [operand], 'unop')
@ -397,10 +462,17 @@ operator (assuming we import ``putchard`` as described in Chapter 4):
.. code-block:: python .. code-block:: python
ready> def binary : 1 (x y) 0 # Low-precedence ready> def binary : 1 (x y) 0 # Low-precedence operator that ignores operands.
operator that ignores operands. ... ready> extern putchard(x) ... ready> ...
def printd(x) putchard(x) : putchard(10) .. ready> printd(65) : ready> extern putchard(x)
printd(66) : printd(67) A B C Evaluated to: 0.0 ...
ready> def printd(x) putchard(x) : putchard(10)
..
ready> printd(65) : printd(66) : printd(67)
A
B
C
Evaluated to: 0.0
@ -409,23 +481,40 @@ We can also define a bunch of other "primitive" operations, such as:
.. code-block:: python .. code-block:: python
# Logical unary not. def unary!(v) if v then 0 # Logical unary not.
else 1 def unary!(v)
if v then
0
else
1
# Unary negate. # Unary negate.
def unary-(v) 0-v def unary-(v)
0-v
# Define > with the same precedence as <. # Define > with the same precedence as <.
def binary> 10 (LHS RHS) RHS < LHS def binary> 10 (LHS RHS)
RHS < LHS
# Binary logical or, which does not short circuit. # Binary logical or, which does not short circuit.
def binary| 5 (LHS RHS) if LHS then 1 else if RHS then 1 else 0 def binary| 5 (LHS RHS)
if LHS then
1
else if RHS then
1
else
0
# Binary logical and, which does not short circuit. # Binary logical and, which does not short circuit.
def binary& 6 (LHS RHS) if !LHS then 0 else !!RHS def binary& 6 (LHS RHS)
if !LHS then
0
else
!!RHS
# Define = with slightly lower precedence than relationals. # Define = with slightly lower precedence than relationals.
def binary = 9 (LHS RHS) !(LHS < RHS | LHS > RHS) def binary = 9 (LHS RHS)
!(LHS < RHS | LHS > RHS)
@ -441,20 +530,33 @@ denser the character:
ready> ready>
extern putchard(char) def printdensity(d) if d > 8 then putchard(32) # ' extern putchard(char)
' else if d > 4 then putchard(46) # '.' else if d > 2 then putchard(43) def printdensity(d)
# '+' else putchard(42); # '*' ... ready> printdensity(1): if d > 8 then
printdensity(2): printdensity(3) : printdensity(4): printdensity(5): putchard(32) # ' '
printdensity(9): putchard(10)*\ ++.. Evaluated to 0.000000 else if d > 4 then
putchard(46) # '.'
else if d > 2 then
putchard(43) # '+'
else
putchard(42); # '*'
...
ready> printdensity(1): printdensity(2): printdensity(3) :
printdensity(4): printdensity(5): printdensity(9): putchard(10)
*++..
Evaluated to 0.000000
Based on these simple primitive operations, we can start to define more Based on these simple primitive operations, we can start to define more
interesting things. For example, here's a little function that solves interesting things. For example, here's a little function that solves
for the number of iterations it takes a function in the complex plane to for the number of iterations it takes a function in the complex plane to
converge: converge:
.. code-block:: python
# determine whether the specific location # determine whether the specific location diverges.
diverges. # Solve for z = z^2 + c in the complex plane. def # Solve for z = z^2 + c in the complex plane.
mandelconverger(real imag iters creal cimag) if iters > 255 | def mandelconverger(real imag iters creal cimag)
if iters > 255 |
(real\ *real + imag*\ imag > 4) then iters else (real\ *real + imag*\ imag > 4) then iters else
mandelconverger(real\ *real - imag*\ imag + creal, 2\ *real*\ imag + mandelconverger(real\ *real - imag*\ imag + creal, 2\ *real*\ imag +
cimag, iters+1, creal, cimag) cimag, iters+1, creal, cimag)
@ -464,6 +566,7 @@ denser the character:
.. _example:
This "z = z2 + c" function is a beautiful little creature that is the This "z = z2 + c" function is a beautiful little creature that is the
basis for computation of the `Mandelbrot basis for computation of the `Mandelbrot