jyapayne/llvmpy
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Misc doc cleanup

Browse source
This commit is contained in:
Maggie Mari 2012-08-17 14:34:51 -05:00
commit 8e00cd7e66
3 changed files with 154 additions and 145 deletions
Download patch file Download diff file Expand all files Collapse all files

52
docs/source/doc/kaleidoscope/PythonLangImpl1.rst
View file

@ -160,7 +160,7 @@ The Lexer
When it comes to implementing a language, the first thing needed is the When it comes to implementing a language, the first thing needed is the
ability to process a text file and recognize what it says. The ability to process a text file and recognize what it says. The
traditional way to do this is to use a traditional way to do this is to use a
`lexer <http://en.wikipedia.org/wiki/Lexical_analysis>`_" (aka `lexer <http://en.wikipedia.org/wiki/Lexical_analysis>`_ (aka
'scanner') to break the input up into "tokens". Each token returned by 'scanner') to break the input up into "tokens". Each token returned by
the lexer includes a token type and potentially some metadata (e.g. the the lexer includes a token type and potentially some metadata (e.g. the
numeric value of a number). First, we define the possibilities: numeric value of a number). First, we define the possibilities:
@ -238,15 +238,11 @@ ignoring whitespace between tokens:
.. code-block:: python .. code-block:: python
def Tokenize(string): def Tokenize(string):
while string: # Skip whitespace. while string: # Skip whitespace.
if string[0].isspace(): if string[0].isspace():
string = string[1:] string = string[1:]
continue continue
::
...
@ -271,9 +267,9 @@ ignore the captured match:
.. code-block:: python .. code-block:: python
# Check if any of the regexes matched and yield # Check if any of the regexes matched and yield
# the appropriate result. # the appropriate result.
if comment_match: if comment_match:
comment = comment_match.group(0) comment = comment_match.group(0)
string = string[len(comment):] string = string[len(comment):]
@ -281,27 +277,27 @@ For numbers, we yield the captured match, converted to a float and
tagged with the appropriate token type: tagged with the appropriate token type:
.. code-block:: python .. code-block:: python
elif number_match: elif number_match:
number = number_match.group(0) number = number_match.group(0)
yield NumberToken(float(number)) yield NumberToken(float(number))
string = string[len(number):] string = string[len(number):]
The identifier case is a little more complex. We have to check for The identifier case is a little more complex. We have to check for
keywords to decide whether we have captured an identifier or a keyword: keywords to decide whether we have captured an identifier or a keyword:
.. code-block:: python .. code-block:: python
elif identifier_match: elif identifier_match:
identifier = identifier_match.group(0) identifier = identifier_match.group(0)
# Check if we matched a keyword. # Check if we matched a keyword.
if identifier == 'def': if identifier == 'def':
yield DefToken() yield DefToken()
elif identifier == 'extern': elif identifier == 'extern':
yield ExternToken() yield ExternToken()
else: else:
yield IdentifierToken(identifier) yield IdentifierToken(identifier)
string = string[len(identifier):] string = string[len(identifier):]
Finally, if we haven't recognized a comment, a number of an identifier, Finally, if we haven't recognized a comment, a number of an identifier,
@ -311,9 +307,9 @@ used, for example, for operators like ``+`` or ``*``:
.. code-block:: python .. code-block:: python
else: # Yield the unknown character. else: # Yield the unknown character.
yield CharacterToken(string[0]) yield CharacterToken(string[0])
string = string[1:] string = string[1:]
Once we're done with the loop, we return a final end-of-file token: Once we're done with the loop, we return a final end-of-file token:
@ -321,5 +317,5 @@ Once we're done with the loop, we return a final end-of-file token:
.. code-block:: python .. code-block:: python
yield EOFToken() yield EOFToken()

234
docs/source/doc/kaleidoscope/PythonLangImpl3.rst
View file

@ -439,7 +439,6 @@ at this point. We'll fix this in `Chapter 5 <PythonLangImpl5.html>`_ :).
# Validate the generated code, checking for consistency. # Validate the generated code, checking for consistency.
function.verify() function.verify()
@ -642,8 +641,10 @@ need to `download <../download.html>`_ and
import re import re
from llvm.core import Module, Constant, Type, Function, Builder, FCMP_ULT from llvm.core import Module, Constant, Type, Function, Builder, FCMP_ULT
Globals Globals
------- -------
.. code-block:: python
# The LLVM module, which holds all the IR code. # The LLVM module, which holds all the IR code.
g_llvm_module = Module.new('my cool jit') g_llvm_module = Module.new('my cool jit')
@ -655,8 +656,10 @@ need to `download <../download.html>`_ and
# and what their LLVM representation is. # and what their LLVM representation is.
g_named_values = {} g_named_values = {}
Lexer Lexer
----- -----
.. code-block:: python
# The lexer yields one of these types for each token. # The lexer yields one of these types for each token.
class EOFToken(object): class EOFToken(object):
@ -678,55 +681,58 @@ need to `download <../download.html>`_ and
class CharacterToken(object): class CharacterToken(object):
def __init__(self, char): def __init__(self, char):
self.char = char def __eq__(self, other): self.char = char
def __eq__(self, other):
return isinstance(other, CharacterToken)and self.char == other.char return isinstance(other, CharacterToken)and self.char == other.char
def __ne__(self, other): def __ne__(self, other):
return not self == other return not self == other
# Regular expressions that tokens and comments of our language. # Regular expressions that tokens and comments of our language.
REGEX_NUMBER = re.compile('[0-9]+(?:.[0-9]+)?') REGEX_NUMBER = re.compile('[0-9]+(?:.[0-9]+)?')
REGEX_IDENTIFIER = re.compile('[a-zA-Z][a-zA-Z0-9]\ *') REGEX_IDENTIFIER = re.compile('[a-zA-Z][a-zA-Z0-9]\ *')
REGEX_COMMENT = re.compile('#.*') REGEX_COMMENT = re.compile('#.*')
def Tokenize(string):
while string:
# Skip whitespace.
if string[0].isspace():
string = string[1:]
continue
# Run regexes.
comment_match = REGEX_COMMENT.match(string)
number_match = REGEX_NUMBER.match(string)
identifier_match = REGEX_IDENTIFIER.match(string)
# Check if any of the regexes matched and yield the appropriate result. def Tokenize(string):
if comment_match: while string:
comment = comment_match.group(0) # Skip whitespace.
string = string[len(comment):] if string[0].isspace():
elif number_match: string = string[1:]
number = number_match.group(0) continue
yield NumberToken(float(number))
string = string[len(number):] # Run regexes.
elif identifier_match: comment_match = REGEX_COMMENT.match(string)
identifier = identifier_match.group(0) number_match = REGEX_NUMBER.match(string)
# Check if we matched a keyword. identifier_match = REGEX_IDENTIFIER.match(string)
if identifier == 'def':
yield DefToken() # Check if any of the regexes matched and yield the appropriate result.
elif identifier == 'extern': if comment_match:
yield ExternToken() comment = comment_match.group(0)
else: string = string[len(comment):]
yield IdentifierToken(identifier) elif number_match:
string = string[len(identifier):] number = number_match.group(0)
else: yield NumberToken(float(number))
# Yield the ASCII value of the unknown character. string = string[len(number):]
yield CharacterToken(string[0]) elif identifier_match:
string = string[1:] identifier = identifier_match.group(0)
# Check if we matched a keyword.
if identifier == 'def':
yield DefToken()
elif identifier == 'extern':
yield ExternToken()
else:
yield IdentifierToken(identifier)
string = string[len(identifier):]
else:
# Yield the ASCII value of the unknown character.
yield CharacterToken(string[0])
string = string[1:]
yield EOFToken()
yield EOFToken() Abstract Syntax Tree (aka Parse Tree)
-------------------------------------
Abstract Syntax Tree (aka Parse Tree)
------------------------------------- .. code-block:: python
# Base class for all expression nodes. # Base class for all expression nodes.
class ExpressionNode(object): class ExpressionNode(object):
@ -756,25 +762,27 @@ need to `download <../download.html>`_ and
# Expression class for a binary operator. # Expression class for a binary operator.
class BinaryOperatorExpressionNode(ExpressionNode): class BinaryOperatorExpressionNode(ExpressionNode):
def __init__(self, operator, left, right): self.operator = operator def __init__(self, operator, left, right):
self.left = left self.right = right self.operator = operator
self.left = left
self.right = right
def CodeGen(self): def CodeGen(self):
left = self.left.CodeGen() left = self.left.CodeGen()
right = self.right.CodeGen() right = self.right.CodeGen()
if self.operator == '+': if self.operator == '+':
return g_llvm_builder.fadd(left, right, 'addtmp') return g_llvm_builder.fadd(left, right, 'addtmp')
elif self.operator == '-': elif self.operator == '-':
return g_llvm_builder.fsub(left, right, 'subtmp') return g_llvm_builder.fsub(left, right, 'subtmp')
elif self.operator == '*': elif self.operator == '*':
return g_llvm_builder.fmul(left, right, 'multmp') return g_llvm_builder.fmul(left, right, 'multmp')
elif self.operator == '<': elif self.operator == '<':
result = g_llvm_builder.fcmp(FCMP_ULT, left, right, 'cmptmp') result = g_llvm_builder.fcmp(FCMP_ULT, left, right, 'cmptmp')
# Convert bool 0 or 1 to double 0.0 or 1.0. # Convert bool 0 or 1 to double 0.0 or 1.0.
return g_llvm_builder.uitofp(result, Type.double(), 'booltmp') return g_llvm_builder.uitofp(result, Type.double(), 'booltmp')
else: else:
raise RuntimeError('Unknown binary operator.') raise RuntimeError('Unknown binary operator.')
# Expression class for function calls. # Expression class for function calls.
class CallExpressionNode(ExpressionNode): class CallExpressionNode(ExpressionNode):
@ -787,13 +795,13 @@ need to `download <../download.html>`_ and
# Look up the name in the global module table. # Look up the name in the global module table.
callee = g_llvm_module.get_function_named(self.callee) callee = g_llvm_module.get_function_named(self.callee)
# Check for argument mismatch error. # Check for argument mismatch error.
if len(callee.args) != len(self.args): if len(callee.args) != len(self.args):
raise RuntimeError('Incorrect number of arguments passed.') raise RuntimeError('Incorrect number of arguments passed.')
arg_values = [i.CodeGen() for i in self.args] arg_values = [i.CodeGen() for i in self.args]
return g_llvm_builder.call(callee, arg_values, 'calltmp') return g_llvm_builder.call(callee, arg_values, 'calltmp')
# This class represents the "prototype" for a function, which captures its name, # This class represents the "prototype" for a function, which captures its name,
# and its argument names (thus implicitly the number of arguments the function # and its argument names (thus implicitly the number of arguments the function
@ -805,9 +813,9 @@ need to `download <../download.html>`_ and
self.args = args self.args = args
def CodeGen(self): def CodeGen(self):
# Make the function type, eg. double(double,double). # Make the function type, eg. double(double,double).
funct_type = Type.function( funct_type = Type.function(
Type.double(), [Type.double()] * len(self.args), False) Type.double(), [Type.double()] * len(self.args), False)
function = Function.new(g_llvm_module, funct_type, self.name) function = Function.new(g_llvm_module, funct_type, self.name)
@ -866,8 +874,10 @@ need to `download <../download.html>`_ and
return function return function
Parser Parser
------ ------
.. code-block:: python
class Parser(object): class Parser(object):
@ -877,13 +887,13 @@ need to `download <../download.html>`_ and
self.Next() self.Next()
# Provide a simple token buffer. Parser.current is the current token the # Provide a simple token buffer. Parser.current is the current token the
# parser is looking at. Parser.Next() reads another token from the lexer # parser is looking at. Parser.Next() reads another token from the lexer and
and # updates Parser.current with its results. # updates Parser.current with its results.
def Next(self): def Next(self):
self.current = self.tokens.next() self.current = self.tokens.next()
# Gets the precedence of the current token, or -1 if the token is not a # Gets the precedence of the current token, or -1 if the token is not a binary
binary # operator. # operator.
def GetCurrentTokenPrecedence(self): def GetCurrentTokenPrecedence(self):
if isinstance(self.current, CharacterToken): if isinstance(self.current, CharacterToken):
return self.binop_precedence.get(self.current.char, -1) return self.binop_precedence.get(self.current.char, -1)
@ -893,7 +903,7 @@ need to `download <../download.html>`_ and
# identifierexpr ::= identifier | identifier '(' expression* ')' # identifierexpr ::= identifier | identifier '(' expression* ')'
def ParseIdentifierExpr(self): def ParseIdentifierExpr(self):
identifier_name = self.current.name identifier_name = self.current.name
self.Next() # eat identifier. self.Next() # eat identifier.
if self.current != CharacterToken('('): # Simple variable reference. if self.current != CharacterToken('('): # Simple variable reference.
return VariableExpressionNode(identifier_name) return VariableExpressionNode(identifier_name)
@ -916,7 +926,7 @@ need to `download <../download.html>`_ and
# numberexpr ::= number # numberexpr ::= number
def ParseNumberExpr(self): def ParseNumberExpr(self):
result = NumberExpressionNode(self.current.value) result = NumberExpressionNode(self.current.value)
self.Next() # consume the number. self.Next() # consume the number.
return result return result
# parenexpr ::= '(' expression ')' # parenexpr ::= '(' expression ')'
@ -948,25 +958,25 @@ need to `download <../download.html>`_ and
while True: while True:
precedence = self.GetCurrentTokenPrecedence() precedence = self.GetCurrentTokenPrecedence()
# If this is a binary operator that binds at least as tightly as the # If this is a binary operator that binds at least as tightly as the
# current one, consume it; otherwise we are done. # current one, consume it; otherwise we are done.
if precedence < left_precedence: if precedence < left_precedence:
return left return left
binary_operator = self.current.char binary_operator = self.current.char
self.Next() # eat the operator. self.Next() # eat the operator.
# Parse the primary expression after the binary operator. # Parse the primary expression after the binary operator.
right = self.ParsePrimary() right = self.ParsePrimary()
# If binary_operator binds less tightly with right than the operator after # If binary_operator binds less tightly with right than the operator after
# right, let the pending operator take right as its left. # right, let the pending operator take right as its left.
next_precedence = self.GetCurrentTokenPrecedence() next_precedence = self.GetCurrentTokenPrecedence()
if precedence < next_precedence: if precedence < next_precedence:
right = self.ParseBinOpRHS(right, precedence + 1) right = self.ParseBinOpRHS(right, precedence + 1)
# Merge left/right. # Merge left/right.
left = BinaryOperatorExpressionNode(binary_operator, left, right) left = BinaryOperatorExpressionNode(binary_operator, left, right)
# expression ::= primary binoprhs # expression ::= primary binoprhs
def ParseExpression(self): def ParseExpression(self):
@ -1000,7 +1010,7 @@ need to `download <../download.html>`_ and
# definition ::= 'def' prototype expression # definition ::= 'def' prototype expression
def ParseDefinition(self): def ParseDefinition(self):
self.Next() # eat def. self.Next() # eat def.
proto = self.ParsePrototype() proto = self.ParsePrototype()
body = self.ParseExpression() body = self.ParseExpression()
return FunctionNode(proto, body) return FunctionNode(proto, body)
@ -1011,7 +1021,7 @@ need to `download <../download.html>`_ and
return FunctionNode(proto, self.ParseExpression()) return FunctionNode(proto, self.ParseExpression())
# external ::= 'extern' prototype # external ::= 'extern' prototype
def ParseExtern(self): def ParseExtern(self):
self.Next() # eat extern. self.Next() # eat extern.
return self.ParsePrototype() return self.ParsePrototype()
@ -1026,17 +1036,19 @@ need to `download <../download.html>`_ and
self.Handle(self.ParseTopLevelExpr, 'Read a top-level expression:') self.Handle(self.ParseTopLevelExpr, 'Read a top-level expression:')
def Handle(self, function, message): def Handle(self, function, message):
try: try:
print message, function().CodeGen() print message, function().CodeGen()
except Exception, e: except Exception, e:
print 'Error:', e print 'Error:', e
try: try:
self.Next() # Skip for error recovery. self.Next() # Skip for error recovery.
except: except:
pass pass
Main driver code. Main driver code.
----------------- -----------------
.. code-block:: python
def main(): def main():
# Install standard binary operators. # Install standard binary operators.

13
docs/source/doc/kaleidoscope/PythonLangImpl4.rst
View file

@ -284,10 +284,10 @@ this:
print 'Evaluated to:', result.as_real(Type.double()) print 'Evaluated to:', result.as_real(Type.double())
except Exception, e: except Exception, e:
print 'Error:', e print 'Error:', e
try: try:
self.Next() # Skip for error recovery. self.Next() # Skip for error recovery.
except: except:
pass pass
Recall that we compile top-level expressions into a self-contained LLVM Recall that we compile top-level expressions into a self-contained LLVM
function that takes no arguments and returns the computed double. function that takes no arguments and returns the computed double.
@ -400,9 +400,10 @@ example, we can create a C file with the following simple function:
.. code-block:: c .. code-block:: c
#include #include <stdio.h>
double putchard(double x) { putchar((char)x); return 0; } {% double putchard(double x) {
putchar((char)x); return 0; } {%
endhighlight %} endhighlight %}
We can then compile this into a shared library with GCC: We can then compile this into a shared library with GCC: