jyapayne/llvmpy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Finished editing PythonLangImpl2.rst doc

Browse source
This commit is contained in:
Maggie Mari 2012-08-17 12:03:26 -05:00
commit 258efb0518
2 changed files with 327 additions and 282 deletions
Download patch file Download diff file Expand all files Collapse all files

471
docs/source/doc/kaleidoscope/PythonLangImpl2.rst
View file

@ -39,8 +39,8 @@ We'll start with expressions first:
.. code-block:: python
# Base class for all expression nodes. class
ExpressionNode(object): pass
# Base class for all expression nodes.
class ExpressionNode(object): pass
# Expression class for numeric literals like "1.0".
class NumberExpressionNode(ExpressionNode):
@ -65,8 +65,7 @@ that we'll use in the basic form of the Kaleidoscope language:
.. code-block:: python
# Expression class for referencing a variable,
like "a".
# Expression class for referencing a variable, like "a".
class VariableExpressionNode(ExpressionNode):
def __init__(self, name):
self.name = name
@ -80,7 +79,7 @@ that we'll use in the basic form of the Kaleidoscope language:
# Expression class for function calls.
class CallExpressionNode(ExpressionNode):
def __init__self, callee, args):
def __init__(self, callee, args):
self.callee = callee
self.args = args
@ -103,9 +102,9 @@ way to talk about functions themselves:
.. code-block:: python
# This class represents the "prototype" for a
function, which captures its name, # and its argument names (thus
implicitly the number of arguments the function # takes).
# This class represents the "prototype" for a function, which captures its name,
# and its argument names (thus implicitly the number of arguments the function
# takes).
class PrototypeNode(object):
def __init__(self, name, args):
self.name = name
@ -161,8 +160,8 @@ class with some basic helper routines:
self.Next()
# Provide a simple token buffer. Parser.current is the current token the
# parser is looking at. Parser.Next() reads another token from the lexer
and # updates Parser.current with its results.
# parser is looking at. Parser.Next() reads another token from the lexer and
# updates Parser.current with its results.
def Next(self):
self.current = self.tokens.next()
@ -173,8 +172,8 @@ to look one token ahead at what the lexer is returning. Every function
in our parser will assume that ``self.current`` is the current token
that needs to be parsed. Note that the first token is read as soon as
the parser is instantiated. Let us ignore the ``binop_precedence``
parameter for now. It will be explained when we start `parsing binary
operators <#parserbinops>`_.
parameter for now. It will be explained when we start parsing binary
operators.
With these basic helper functions, we can implement the first piece of
our grammar: numeric literals.
@ -247,30 +246,29 @@ function calls:
.. code-block:: python
# identifierexpr ::= identifier \| identifier '('
expression\* ')'
# identifierexpr ::= identifier | identifier '(' expression* ')'
def ParseIdentifierExpr(self):
identifier_name = self.current.name
self.Next() # eat identifier.
if self.current != CharacterToken('('): # Simple variable reference.
return VariableExpressionNode(identifier_name);
if self.current != CharacterToken('('): # Simple variable reference.
return VariableExpressionNode(identifier_name);
# Call.
self.Next() # eat '('.
args = []
if self.current != CharacterToken(')'):
while True:
args.append(self.ParseExpression())
if self.current == CharacterToken(')'):
break
elif self.current != CharacterToken(','):
raise RuntimeError('Expected ")" or "," in argument list.')
self.Next()
# Call.
self.Next() # eat '('.
args = []
if self.current != CharacterToken(')'):
while True:
args.append(self.ParseExpression())
if self.current == CharacterToken(')'):
break
elif self.current != CharacterToken(','):
raise RuntimeError('Expected ")" or "," in argument list.')
self.Next()
self.Next() # eat ')'.
return CallExpressionNode(identifier_name, args)
self.Next() # eat ')'.
return CallExpressionNode(identifier_name, args)
@ -295,17 +293,16 @@ primary expression, we need to determine what sort of expression it is:
.. code-block:: python
# primary ::= identifierexpr | numberexpr |
parenexpr
# primary ::= identifierexpr | numberexpr | parenexpr
def ParsePrimary(self):
if isinstance(self.current, IdentifierToken):
return self.ParseIdentifierExpr()
elif isinstance(self.current, NumberToken):
return self.ParseNumberExpr();
elif self.current == CharacterToken('('):
return self.ParseParenExpr()
else:
raise RuntimeError('Unknown token when expecting an expression.')
elif isinstance(self.current, NumberToken):
return self.ParseNumberExpr();
elif self.current == CharacterToken('('):
return self.ParseParenExpr()
else:
raise RuntimeError('Unknown token when expecting an expression.')
@ -340,21 +337,22 @@ Now is the time to use it:
.. code-block:: python
def main(): # Install standard binary operators.
# 1 is lowest possible precedence. 40 is the highest.
operator_precedence = {
'<': 10,
'+': 20,
'-': 20,
'*': 40
}
def main():
# Install standard binary operators.
# 1 is lowest possible precedence. 40 is the highest.
operator_precedence = {
'<': 10,
'+': 20,
'-': 20,
'*': 40
}
# Run the main ``interpreter loop``.
while True:
# Run the main ``interpreter loop``.
while True:
...
...
parser = Parser(Tokenize(raw), operator_precedence)
parser = Parser(Tokenize(raw), operator_precedence)
@ -373,8 +371,8 @@ token, or -1 if the token is not a binary operator:
.. code-block:: python
# Gets the precedence of the current token, or -1
if the token is not a binary # operator.
# Gets the precedence of the current token, or -1 if the token is not a binary
# operator.
def GetCurrentTokenPrecedence(self):
if isinstance(self.current, CharacterToken):
return self.binop_precedence.get(self.current.char, -1)
@ -416,7 +414,7 @@ the code passes the expression for ``a`` into ``ParseBinOpRHS`` and the
current token is ``+``.
The precedence value passed into ``ParseBinOpRHS`` indicates the \*
minimal operator precedence\* that the function is allowed to eat. For
minimal operator precedence \* that the function is allowed to eat. For
example, if the current pair stream is ``[+, x]`` and ``ParseBinOpRHS``
is passed in a precedence of 40, it will not consume any tokens (because
the precedence of '+' is only 20). With this in mind, ``ParseBinOpRHS``
@ -451,7 +449,7 @@ expression:
.. code-block:: python
binary_operator = self.current.char
self.Next() # eat the operator.
self.Next() # eat the operator.
# Parse the primary expression after the binary operator.
@ -475,11 +473,11 @@ precedence (which is '+' in this case):
.. code-block:: python
# If binary_operator binds less tightly with
right than the operator after # right, let the pending operator take
right as its left.
# If binary_operator binds less tightly with right than the operator after
# right, let the pending operator take right as its left.
next_precedence = self.GetCurrentTokenPrecedence()
if precedence < next_precedence:
...
@ -521,8 +519,8 @@ duplicated for context):
.. code-block:: python
# If binary_operator binds less tightly with
right than the operator after # right, let the pending operator take right as its left.
# If binary_operator binds less tightly with right than the operator after
# right, let the pending operator take right as its left.
next_precedence = self.GetCurrentTokenPrecedence()
if precedence < next_precedence:
right = self.ParseBinOpRHS(right, precedence + 1)
@ -654,7 +652,7 @@ The Driver
The driver for this simply invokes all of the parsing pieces with a
top-level dispatch loop. There isn't much interesting here, so I'll just
include the top-level loop. See `below <#code>`_ for full code.
include the top-level loop. See :ref:`below <code>` for full code.
.. code-block:: python
@ -725,6 +723,8 @@ LLVM Intermediate Representation (IR) from the AST.
--------------
.. _code:
Full Code Listing
===========================
@ -739,8 +739,10 @@ external libraries at all for this.
import re
Lexer
-----
Lexer
-----
.. code-block:: python
# The lexer yields one of these types for each token.
class EOFToken(object):
@ -769,45 +771,54 @@ external libraries at all for this.
return not self == other
# Regular expressions that tokens and comments of our language.
REGEX_NUMBER = re.compile('[0-9]+(?:.[0-9]+)?') REGEX_IDENTIFIER =
re.compile('[a-zA-Z][a-zA-Z0-9]\ *') REGEX_COMMENT = re.compile('#.*')
REGEX_NUMBER = re.compile('[0-9]+(?:\.[0-9]+)?')
REGEX_IDENTIFIER = re.compile('[a-zA-Z][a-zA-Z0-9] *')
REGEX_COMMENT = re.compile('#.*')
def Tokenize(string): while string: # Skip whitespace. if
string[0].isspace(): string = string[1:] continue
def Tokenize(string):
while string:
# Skip whitespace.
if string[0].isspace():
string = string[1:]
continue
# Run regexes.
comment_match = REGEX_COMMENT.match(string)
number_match = REGEX_NUMBER.match(string)
identifier_match = REGEX_IDENTIFIER.match(string)
# Run regexes.
comment_match = REGEX_COMMENT.match(string)
number_match = REGEX_NUMBER.match(string)
identifier_match = REGEX_IDENTIFIER.match(string)
# Check if any of the regexes matched and yield the appropriate result.
if comment_match:
comment = comment_match.group(0)
string = string[len(comment):]
elif number_match:
number = number_match.group(0)
yield NumberToken(float(number))
string = string[len(number):]
elif identifier_match:
identifier = identifier_match.group(0)
# Check if we matched a keyword.
if identifier == 'def':
yield DefToken()
elif identifier == 'extern':
yield ExternToken()
else:
yield IdentifierToken(identifier)
string = string[len(identifier):]
else:
# Yield the ASCII value of the unknown character.
yield CharacterToken(string[0])
string = string[1:]
# Check if any of the regexes matched and yield the appropriate result.
if comment_match:
comment = comment_match.group(0)
string = string[len(comment):]
elif number_match:
number = number_match.group(0)
yield NumberToken(float(number))
string = string[len(number):]
elif identifier_match:
identifier = identifier_match.group(0)
# Check if we matched a keyword.
if identifier == 'def':
yield DefToken()
elif identifier == 'extern':
yield ExternToken()
else:
yield IdentifierToken(identifier)
string = string[len(identifier):]
else:
# Yield the ASCII value of the unknown character.
yield CharacterToken(string[0])
string = string[1:]
yield EOFToken()
Abstract Syntax Tree (aka Parse Tree)
-------------------------------------
Abstract Syntax Tree (aka Parse Tree)
-------------------------------------
.. code-block:: python
# Base class for all expression nodes.
class ExpressionNode(object):
@ -845,37 +856,43 @@ external libraries at all for this.
self.args = args
# This class represents a function definition itself.
class FunctionNode(object): def __init__(self, prototype, body):
self.prototype = prototype self.body = body
class FunctionNode(object):
def __init__(self, prototype, body):
self.prototype = prototype
self.body = body
Parser
------
Parser
------
.. code-block:: python
class Parser(object):
def __init__(self, tokens, binop_precedence):
self.tokens = tokens
self.binop_precedence = binop_precedence
self.Next()
self.Next()
# Provide a simple token buffer. Parser.current is the current token the
# parser is looking at. Parser.Next() reads another token from the lexer
and # updates Parser.current with its results.
# parser is looking at. Parser.Next() reads another token from the lexer and
# updates Parser.current with its results.
def Next(self):
self.current = self.tokens.next()
# Gets the precedence of the current token, or -1 if the token is not a
binary # operator.
# Gets the precedence of the current token, or -1 if the token is not a binary
# operator.
def GetCurrentTokenPrecedence(self):
if isinstance(self.current, CharacterToken):
return self.binop_precedence.get(self.current.char, -1)
else:
return -1
# identifierexpr ::= identifier \| identifier '(' expression\* ')'
# identifierexpr ::= identifier | identifier '(' expression* ')'
def ParseIdentifierExpr(self):
identifier_name = self.current.name
self.Next() # eat identifier.
self.Next() # eat identifier.
if self.current != CharacterToken('('): # Simple variable reference.
@ -896,163 +913,167 @@ external libraries at all for this.
self.Next() # eat ')'.
return CallExpressionNode(identifier_name, args)
# numberexpr ::= number
def ParseNumberExpr(self):
result = NumberExpressionNode(self.current.value)
self.Next() # consume the number.
return result
# numberexpr ::= number
def ParseNumberExpr(self):
result = NumberExpressionNode(self.current.value)
self.Next() # consume the number.
return result
# parenexpr ::= '(' expression ')'
def ParseParenExpr(self):
self.Next() # eat '('.
# parenexpr ::= '(' expression ')'
def ParseParenExpr(self):
self.Next() # eat '('.
contents = self.ParseExpression()
contents = self.ParseExpression()
if self.current != CharacterToken(')'):
raise RuntimeError('Expected ")".')
self.Next() # eat ')'.
if self.current != CharacterToken(')'):
raise RuntimeError('Expected ")".')
self.Next() # eat ')'.
return contents
return contents
# primary ::= identifierexpr | numberexpr | parenexpr
def ParsePrimary(self):
if isinstance(self.current, IdentifierToken):
return self.ParseIdentifierExpr()
elif isinstance(self.current, NumberToken):
return self.ParseNumberExpr()
elif self.current == CharacterToken('('):
return self.ParseParenExpr()
else:
raise RuntimeError('Unknown token when expecting an expression.')
# primary ::= identifierexpr | numberexpr | parenexpr
def ParsePrimary(self):
if isinstance(self.current, IdentifierToken):
return self.ParseIdentifierExpr()
elif isinstance(self.current, NumberToken):
return self.ParseNumberExpr()
elif self.current == CharacterToken('('):
return self.ParseParenExpr()
else:
raise RuntimeError('Unknown token when expecting an expression.')
# binoprhs ::= (operator primary)*
def ParseBinOpRHS(self, left, left_precedence):
# If this is a binary operator, find its precedence.
while True:
precedence = self.GetCurrentTokenPrecedence()
# binoprhs ::= (operator primary)*
def ParseBinOpRHS(self, left, left_precedence):
# If this is a binary operator, find its precedence.
while True:
precedence = self.GetCurrentTokenPrecedence()
# If this is a binary operator that binds at least as tightly as the
# current one, consume it; otherwise we are done.
if precedence < left_precedence:
return left
# If this is a binary operator that binds at least as tightly as the
# current one, consume it; otherwise we are done.
if precedence < left_precedence:
return left
binary_operator = self.current.char
self.Next() # eat the operator.
binary_operator = self.current.char
self.Next() # eat the operator.
# Parse the primary expression after the binary operator.
right = self.ParsePrimary()
# Parse the primary expression after the binary operator.
right = self.ParsePrimary()
# If binary_operator binds less tightly with right than the operator after
# right, let the pending operator take right as its left.
next_precedence = self.GetCurrentTokenPrecedence()
if precedence < next_precedence:
right = self.ParseBinOpRHS(right, precedence + 1)
# If binary_operator binds less tightly with right than the operator after
# right, let the pending operator take right as its left.
next_precedence = self.GetCurrentTokenPrecedence()
if precedence < next_precedence:
right = self.ParseBinOpRHS(right, precedence + 1)
# Merge left/right.
left = BinaryOperatorExpressionNode(binary_operator, left, right)
# Merge left/right.
left = BinaryOperatorExpressionNode(binary_operator, left, right)
# expression ::= primary binoprhs
def ParseExpression(self):
left = self.ParsePrimary()
return self.ParseBinOpRHS(left, 0)
# expression ::= primary binoprhs
def ParseExpression(self):
left = self.ParsePrimary()
return self.ParseBinOpRHS(left, 0)
# prototype ::= id '(' id\* ')'
def ParsePrototype(self):
if not isinstance(self.current, IdentifierToken):
raise RuntimeError('Expected function name in prototype.')
# prototype ::= id '(' id* ')'
def ParsePrototype(self):
if not isinstance(self.current, IdentifierToken):
raise RuntimeError('Expected function name in prototype.')
function_name = self.current.name
self.Next() # eat function name.
function_name = self.current.name
self.Next() # eat function name.
if self.current != CharacterToken('('):
raise RuntimeError('Expected "(" in prototype.')
self.Next() # eat '('.
if self.current != CharacterToken('('):
raise RuntimeError('Expected "(" in prototype.')
self.Next() # eat '('.
arg_names = []
while isinstance(self.current, IdentifierToken):
arg_names.append(self.current.name)
self.Next()
arg_names = []
while isinstance(self.current, IdentifierToken):
arg_names.append(self.current.name)
self.Next()
if self.current != CharacterToken(')'):
raise RuntimeError('Expected ")" in prototype.')
if self.current != CharacterToken(')'):
raise RuntimeError('Expected ")" in prototype.')
# Success.
self.Next() # eat ')'.
# Success.
self.Next() # eat ')'.
return PrototypeNode(function_name, arg_names)
return PrototypeNode(function_name, arg_names)
# definition ::= 'def' prototype expression
def ParseDefinition(self):
self.Next() # eat def.
# definition ::= 'def' prototype expression
def ParseDefinition(self):
self.Next() # eat def.
proto = self.ParsePrototype()
body = self.ParseExpression()
return FunctionNode(proto, body)
# toplevelexpr ::= expression
def ParseTopLevelExpr(self):
proto = PrototypeNode('', [])
return FunctionNode(proto, self.ParseExpression())
# toplevelexpr ::= expression
def ParseTopLevelExpr(self):
proto = PrototypeNode('', [])
return FunctionNode(proto, self.ParseExpression())
# external ::= 'extern' prototype
def ParseExtern(self):
self.Next() #eat extern.
return self.ParsePrototype()
# external ::= 'extern' prototype
def ParseExtern(self):
self.Next() # eat extern.
return self.ParsePrototype()
# Top-Level parsing
def HandleDefinition(self):
self.Handle(self.ParseDefinition, 'Parsed a function definition.')
# Top-Level parsing
def HandleDefinition(self):
self.Handle(self.ParseDefinition, 'Parsed a function definition.')
def HandleExtern(self):
self.Handle(self.ParseExtern, 'Parsed an extern.')
def HandleExtern(self):
self.Handle(self.ParseExtern, 'Parsed an extern.')
def HandleTopLevelExpression(self):
self.Handle(self.ParseTopLevelExpr, 'Parsed a top-level expression.')
def HandleTopLevelExpression(self):
self.Handle(self.ParseTopLevelExpr, 'Parsed a top-level expression.')
def Handle(self, function, message):
try:
function()
print message
except Exception, e:
print 'Error:', e
def Handle(self, function, message):
try:
self.Next() # Skip for error recovery.
except:
pass
function()
print message
except Exception, e:
print 'Error:', e
try:
self.Next() # Skip for error recovery.
except:
pass
Main driver code.
-----------------
Main driver code.
-----------------
.. code-block:: python
def main():
# Install standard binary operators.
# 1 is lowest possible precedence. 40 is the highest.
operator_precedence = {
'<': 10,
'+':20,
'-': 20,
'*': 40
}
# Install standard binary operators.
# 1 is lowest possible precedence. 40 is the highest.
operator_precedence = {
'<': 10,
'+': 20,
'-': 20,
'*': 40
}
# Run the main "interpreter loop".
while True:
print 'ready>',
try:
raw = raw_input()
except KeyboardInterrupt:
return
parser = Parser(Tokenize(raw), operator_precedence)
# Run the main "interpreter loop".
while True:
# top ::= definition | external | expression | EOF
if isinstance(parser.current, EOFToken):
break
if isinstance(parser.current, DefToken):
parser.HandleDefinition()
elif isinstance(parser.current, ExternToken):
parser.HandleExtern()
else:
parser.HandleTopLevelExpression()
print 'ready>',
try:
raw = raw_input()
except KeyboardInterrupt:
return
if ==name__ == '__main__':
parser = Parser(Tokenize(raw), operator_precedence)
while True:
# top ::= definition | external | expression | EOF
if isinstance(parser.current, EOFToken):
break
if isinstance(parser.current, DefToken):
parser.HandleDefinition()
elif isinstance(parser.current, ExternToken):
parser.HandleExtern()
else:
parser.HandleTopLevelExpression()
if __name__ == '__main__':
main()

78
docs/source/doc/kaleidoscope/PythonLangImpl4.rst
View file

@ -287,15 +287,21 @@ this:
try:
self.Next() # Skip for error recovery.
except:
pass {% endhighlight %}
pass
Recall that we compile top-level expressions into a self-contained LLVM
function that takes no arguments and returns the computed double.
Recall that we compile top-level expressions into a self-contained LLVM
function that takes no arguments and returns the computed double.
With just these two changes, lets see how Kaleidoscope works now!
With just these two changes, lets see how Kaleidoscope works now!
ready> 4+5 Read a top level expression: define
double @0() { entry: ret double 9.000000e+00 }
.. code-block:: bash
ready> 4+5
Read a top level expression:
define double @0() {
entry:
ret double 9.000000e+00
}
Evaluated to: 9.0
@ -307,16 +313,24 @@ synthesize for each top-level expression that is typed in. This
demonstrates very basic functionality, but can we do more?
.. code-block:: python
.. code-block:: bash
ready> def testfunc(x y) x + y\*2 Read a function
definition: define double @testfunc(double %x, double %y) { entry:
%multmp = fmul double %y, 2.000000e+00 ; <double> [#uses=1] %addtmp = fadd double
%multmp, %x ; <double> [#uses=1] ret double %addtmp }
ready> def testfunc(x y) x + y*2
Read a function definition:
define double @testfunc(double %x, double %y) {
entry:
%multmp = fmul double %y, 2.000000e+00 ; <double> [#uses=1]
%addtmp = fadd double %multmp, %x ; <double> [#uses=1]
ret double %addtmp
}
ready> testfunc(4, 10) Read a top level expression: define double @0() {
entry: %calltmp = call double @testfunc(double 4.000000e+00, double
1.000000e+01) ; <double> [#uses=1] ret double %calltmp }
ready> testfunc(4, 10)
Read a top level expression:
define double @0() {
entry:
%calltmp = call double @testfunc(double 4.000000e+00, double 1.000000e+01) ; <double> [#uses=1]
ret double %calltmp
}
*Evaluated to: 24.0*
@ -339,23 +353,33 @@ anonymous functions, you should get the idea by now :) :
.. code-block:: bash
ready> extern sin(x) Read an extern: declare double
@sin(double)
ready> extern sin(x)
Read an extern:
declare double @sin(double)
ready> extern cos(x) Read an extern: declare double @cos(double)
ready> extern cos(x)
Read an extern:
declare double @cos(double)
ready> sin(1.0) *Evaluated to: 0.841470984808*
ready> sin(1.0)
*Evaluated to: 0.841470984808*
ready> def foo(x) sin(x)\ *sin(x) + cos(x)*\ cos(x) Read a function
definition: define double @foo(double %x) { entry: %calltmp = call
double @sin(double %x) ; <double> [#uses=1] %calltmp1 = call double @sin(double
%x) ; <double> [#uses=1] %multmp = fmul double %calltmp, %calltmp1 ; <double> [#uses=1]
%calltmp2 = call double @cos(double %x) ; <double> [#uses=1] %calltmp3 = call
double @cos(double %x) ; <double> [#uses=1] %multmp4 = fmul double %calltmp2,
%calltmp3 ; <double> [#uses=1] %addtmp = fadd double %multmp, %multmp4 ;
<double> [#uses=1] ret double %addtmp }
ready> def foo(x) sin(x) *sin(x) + cos(x)* cos(x)
Read a function definition:
define double @foo(double %x) {
entry:
%calltmp = call double @sin(double %x) ; <double> [#uses=1]
%calltmp1 = call double @sin(double %x) ; <double> [#uses=1]
%multmp = fmul double %calltmp, %calltmp1 ; <double> [#uses=1]
%calltmp2 = call double @cos(double %x) ; <double> [#uses=1]
%calltmp3 = call double @cos(double %x) ; <double> [#uses=1]
%multmp4 = fmul double %calltmp2, %calltmp3 ; <double> [#uses=1]
%addtmp = fadd double %multmp, %multmp4 ; <double> [#uses=1]
ret double %addtmp
}
ready> foo(4.0) *Evaluated to: 1.000000*
ready> foo(4.0)
*Evaluated to: 1.000000*