Remove .md files and add .rst files
This commit is contained in:
Travis E. Oliphant
parent
c1c9b6e724
commit
0e223a61c2
86 changed files with 11275 additions and 11943 deletions
|
|
@ -1,330 +0,0 @@
|
|||
---
|
||||
layout: page
|
||||
title: "Kaleidoscope: Chapter 1"
|
||||
---
|
||||
|
||||
# Tutorial Introduction and the Lexer
|
||||
|
||||
Written by [Chris Lattner](mailto:sabre@nondot.org)
|
||||
and [Max Shawabkeh](http://max99x.com)
|
||||
|
||||
**Chapter 1**
|
||||
|
||||
|
||||
* This will become a table of contents (this text will be scraped).
|
||||
{:toc}
|
||||
|
||||
[**Chapter 2: Implementing a Parser and AST**](PythonLangImpl2.html)
|
||||
|
||||
# Introduction
|
||||
|
||||
Welcome to the "Implementing a language with LLVM" tutorial. This tutorial
|
||||
runs through the implementation of a simple language, showing how fun and
|
||||
easy it can be. This tutorial will get you up and started as well as help to
|
||||
build a framework you can extend to other languages. The code in this
|
||||
tutorial can also be used as a playground to hack on other LLVM specific
|
||||
things.
|
||||
|
||||
It is useful to point out ahead of time that this tutorial is really about
|
||||
teaching compiler techniques and LLVM specifically, *not* about teaching
|
||||
modern and sane software engineering principles. In practice, this means that
|
||||
we'll take a number of shortcuts to simplify the exposition. If you dig in and
|
||||
use the code as a basis for future projects, fixing its deficiencies shouldn't
|
||||
be hard.
|
||||
|
||||
|
||||
We've tried to put this tutorial together in a way that makes chapters easy
|
||||
to skip over if you are already familiar with or are uninterested in the
|
||||
various pieces. The structure of the tutorial is:
|
||||
|
||||
* **[Chapter 1](#language): Introduction to the Kaleidoscope language,
|
||||
and the definition of its Lexer** -- This shows where we are going
|
||||
and the basic functionality that we want it to do. In order to make this
|
||||
tutorial maximally understandable and hackable, we choose to implement
|
||||
everything in Python instead of using lexer and parser generators. LLVM
|
||||
obviously works just fine with such tools, feel free to use one if you prefer.
|
||||
|
||||
* **[Chapter 2](PythonLangImpl2.html): Implementing a Parser and AST** --
|
||||
With the lexer in place, we can talk about parsing techniques and
|
||||
basic AST construction. This tutorial describes recursive descent parsing and
|
||||
operator precedence parsing. Nothing in Chapters 1 or 2 is LLVM-specific,
|
||||
the code doesn't even import the LLVM modules at this point. :)
|
||||
|
||||
* **[Chapter 3](PythonLangImpl3.html): Code generation to LLVM IR** -- With
|
||||
the AST ready, we can show off how easy generation of LLVM IR really is.
|
||||
|
||||
* **[Chapter 4](PythonLangImpl4.html): Adding JIT and Optimizer support** --
|
||||
Because a lot of people are interested in using LLVM as a JIT,
|
||||
we'll dive right into it and show you the 3 lines it takes to add JIT support.
|
||||
LLVM is also useful in many other ways, but this is one simple and "sexy" way
|
||||
to shows off its power. :)
|
||||
|
||||
* **[Chapter 5](PythonLangImpl5.html): Extending the Language: Control Flow**
|
||||
-- With the language up and running, we show how to extend it
|
||||
with control flow operations (if/then/else and a 'for' loop). This gives us a
|
||||
chance to talk about simple SSA construction and control flow.
|
||||
|
||||
* **[Chapter 6](PythonLangImpl6.html): Extending the Language:
|
||||
User-defined Operators** -- This is a silly but fun chapter that talks about
|
||||
extending the language to let the user program define their own arbitrary
|
||||
unary and binary operators (with assignable precedence!). This lets us build
|
||||
a significant piece of the "language" as library routines.
|
||||
|
||||
* **[Chapter 7](PythonLangImpl7.html): Extending the Language:
|
||||
Mutable Variables** -- This chapter talks about adding user-defined local
|
||||
variables along with an assignment operator. The interesting part about this
|
||||
is how easy and trivial it is to construct SSA form in LLVM: no, LLVM does
|
||||
*not* require your front-end to construct SSA form!
|
||||
|
||||
* **[Chapter 8](PythonLangImpl8.html): Conclusion and other
|
||||
useful LLVM tidbits** -- This chapter wraps up the series by talking about
|
||||
potential ways to extend the language, but also includes a bunch of pointers to
|
||||
info about "special topics" like adding garbage collection support, exceptions,
|
||||
debugging, support for "spaghetti stacks", and a bunch of other tips and
|
||||
tricks.
|
||||
|
||||
By the end of the tutorial, we'll have written a bit less than 540 lines of
|
||||
non-comment, non-blank, lines of code. With this small amount of code, we'll
|
||||
have built up a very reasonable compiler for a non-trivial language including
|
||||
a hand-written lexer, parser, AST, as well as code generation support with a JIT
|
||||
compiler. While other systems may have interesting "hello world" tutorials,
|
||||
I think the breadth of this tutorial is a great testament to the strengths of
|
||||
LLVM and why you should consider it if you're interested in language or compiler
|
||||
design.
|
||||
|
||||
A note about this tutorial: we expect you to extend the language and play
|
||||
with it on your own. Take the code and go crazy hacking away at it, compilers
|
||||
don't need to be scary creatures - it can be a lot of fun to play with
|
||||
languages!
|
||||
|
||||
* * *
|
||||
|
||||
# The Basic Language # {#language}
|
||||
|
||||
This tutorial will be illustrated with a toy language that we'll call
|
||||
"[Kaleidoscope](http://en.wikipedia.org/wiki/Kaleidoscope)" (derived
|
||||
from "meaning beautiful, form, and view").
|
||||
Kaleidoscope is a procedural language that allows you to define functions, use
|
||||
conditionals, math, etc. Over the course of the tutorial, we'll extend
|
||||
Kaleidoscope to support the if/then/else construct, a for loop, user defined
|
||||
operators, JIT compilation with a simple command line interface, etc.
|
||||
|
||||
Because we want to keep things simple, the only datatype in Kaleidoscope is a
|
||||
64-bit floating point type. As such, all values are implicitly double precision
|
||||
and the language doesn't require type declarations. This gives the language a
|
||||
very nice and simple syntax. For example, the following simple example computes
|
||||
[Fibonacci numbers](http://en.wikipedia.org/wiki/Fibonacci_number):
|
||||
|
||||
{% highlight python %}
|
||||
# Compute the x'th fibonacci number.
|
||||
def fib(x)
|
||||
if x < 3 then
|
||||
1
|
||||
else
|
||||
fib(x-1)+fib(x-2)
|
||||
|
||||
# This expression will compute the 40th number.
|
||||
fib(40)
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
We also allow Kaleidoscope to call into standard library functions (the LLVM
|
||||
JIT makes this completely trivial). This means that you can use the 'extern'
|
||||
keyword to define a function before you use it (this is also useful for mutually
|
||||
recursive functions). For example:
|
||||
|
||||
{% highlight python %}
|
||||
extern sin(arg);
|
||||
extern cos(arg);
|
||||
extern atan2(arg1 arg2);
|
||||
|
||||
atan2(sin(0.4), cos(42))
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
A more interesting example is included in Chapter 6 where we write a little
|
||||
Kaleidoscope application that [displays](PythonLangImpl6.html#example)
|
||||
a Mandelbrot Set</a> at various levels of magnification.
|
||||
|
||||
Lets dive into the implementation of this language!
|
||||
|
||||
* * *
|
||||
|
||||
# The Lexer # {#lexer}
|
||||
|
||||
|
||||
When it comes to implementing a language, the first thing needed is
|
||||
the ability to process a text file and recognize what it says.
|
||||
The traditional way to do this is to use a
|
||||
[lexer](http://en.wikipedia.org/wiki/Lexical_analysis)" (aka 'scanner')
|
||||
to break the input up into "tokens". Each token returned by the lexer includes
|
||||
a token type and potentially some metadata (e.g. the numeric value of a number).
|
||||
First, we define the possibilities:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
# The lexer yields one of these types for each token.
|
||||
class EOFToken(object):
|
||||
pass
|
||||
|
||||
class DefToken(object):
|
||||
pass
|
||||
|
||||
class ExternToken(object):
|
||||
pass
|
||||
|
||||
class IdentifierToken(object):
|
||||
def __init__(self, name): self.name = name
|
||||
|
||||
class NumberToken(object):
|
||||
def __init__(self, value): self.value = value
|
||||
|
||||
class CharacterToken(object):
|
||||
def __init__(self, char): self.char = char
|
||||
def __eq__(self, other):
|
||||
return isinstance(other, CharacterToken) and self.char == other.char
|
||||
def __ne__(self, other): return not self == other
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
Each token yielded by our lexer will be of one of the above types. For simple
|
||||
tokens that are always the same, like the "def" keyword, the lexer will yield
|
||||
`DefToken()`>. Identifiers, numbers and characters, on the other
|
||||
hand, have extra data, so when the lexer encounteres the number 123.45, it will
|
||||
emit it as `NumberToken(123.45)`. An identifier `foo` will be
|
||||
emitted as `IdentifierToken('foo')`. And finally, an unknown character
|
||||
like '+' will be returned as `CharacterToken('+')`. You may notice that
|
||||
we overload the equality and inequality operators for the characters; this will
|
||||
later simplify character comparisons in the parser code.
|
||||
|
||||
The actual implementation of the lexer is a single function called `Tokenize`,
|
||||
which takes a string and
|
||||
[yields](http://docs.python.org/reference/simple_stmts.html#the-yield-statement)
|
||||
tokens. For simplicity, we will use
|
||||
[regular expressions](http://docs.python.org/library/re.html)
|
||||
to parse out the tokens. This is terribly inefficient, but
|
||||
perfectly sufficient for our needs.
|
||||
|
||||
First, we define the regular expressions for our tokens. Numbers and strings
|
||||
of digits, optionally followed by a period and another string of digits.
|
||||
Identifiers (and keywords) are alphanumeric string starting with a letter and
|
||||
comments are anything between a hash (`#`) and the end of the line.
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
import re
|
||||
|
||||
...
|
||||
|
||||
# Regular expressions that tokens and comments of our language.
|
||||
REGEX_NUMBER = re.compile('[0-9]+(?:\.[0-9]+)?')
|
||||
REGEX_IDENTIFIER = re.compile('[a-zA-Z][a-zA-Z0-9]*')
|
||||
REGEX_COMMENT = re.compile('#.*')
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
Next, let's start defining the `Tokenize` function itself. The first
|
||||
thing we need to do is set up a loop that scans the string, while ignoring
|
||||
whitespace between tokens:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
def Tokenize(string):
|
||||
while string:
|
||||
# Skip whitespace.
|
||||
if string[0].isspace():
|
||||
string = string[1:]
|
||||
continue
|
||||
|
||||
...
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
Next we want to find out what the next token is. For this we run the regexes
|
||||
we defined above on the remainder of the string. To simplify the rest of the
|
||||
code, we run all three regexes each time. As mentioned above, inefficiencies are
|
||||
ignored for the purpose of this tutorial:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
# Run regexes.
|
||||
comment_match = REGEX_COMMENT.match(string)
|
||||
number_match = REGEX_NUMBER.match(string)
|
||||
identifier_match = REGEX_IDENTIFIER.match(string)
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
Now se check if any of the regexes matched. For comments, we simply
|
||||
ignore the captured match:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
# Check if any of the regexes matched and yield the appropriate result.
|
||||
if comment_match:
|
||||
comment = comment_match.group(0)
|
||||
string = string[len(comment):]
|
||||
{% endhighlight python %}
|
||||
|
||||
For numbers, we yield the captured match, converted to a float and tagged
|
||||
with the appropriate token type:
|
||||
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
elif number_match:
|
||||
number = number_match.group(0)
|
||||
yield NumberToken(float(number))
|
||||
string = string[len(number):]
|
||||
{% endhighlight %}
|
||||
|
||||
The identifier case is a little more complex. We have to check for keywords
|
||||
to decide whether we have captured an identifier or a keyword:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
elif identifier_match:
|
||||
identifier = identifier_match.group(0)
|
||||
# Check if we matched a keyword.
|
||||
if identifier == 'def':
|
||||
yield DefToken()
|
||||
elif identifier == 'extern':
|
||||
yield ExternToken()
|
||||
else:
|
||||
yield IdentifierToken(identifier)
|
||||
string = string[len(identifier):]
|
||||
{% endhighlight %}
|
||||
|
||||
Finally, if we haven't recognized a comment, a number of an identifier, we
|
||||
yield the current character as an "unknown character" token. This is used, for
|
||||
example, for operators like `+` or `*`:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
else:
|
||||
# Yield the unknown character.
|
||||
yield CharacterToken(string[0])
|
||||
string = string[1:]
|
||||
{% endhighlight %}
|
||||
|
||||
Once we're done with the loop, we return a final end-of-file token:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
yield EOFToken()
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
With this, we have the complete lexer for the basic Kaleidoscope language
|
||||
(the [full code listing](PythonLangImpl2.html#code) for the Lexer is
|
||||
available in the [next chapter](PythonLangImpl2.html) of the
|
||||
tutorial). Next we'll [build a simple parser that
|
||||
uses this to build an Abstract Syntax Tree](PythonLangImpl2.html).
|
||||
When we have that, we'll
|
||||
include a driver so that you can use the lexer and parser together.
|
||||
|
||||
* * *
|
||||
|
||||
**[Next: Implementing a Parser and AST](PythonLangImpl2.html)**
|
||||
|
||||
|
||||
|
||||
286
docs/source/doc/kaleidoscope/PythonLangImpl1.rst
Normal file
286
docs/source/doc/kaleidoscope/PythonLangImpl1.rst
Normal file
|
|
@ -0,0 +1,286 @@
|
|||
+------------------------------------+
|
||||
| layout: page |
|
||||
+------------------------------------+
|
||||
| title: "Kaleidoscope: Chapter 1" |
|
||||
+------------------------------------+
|
||||
|
||||
Tutorial Introduction and the Lexer
|
||||
===================================
|
||||
|
||||
Written by `Chris Lattner <mailto:sabre@nondot.org>`_ and `Max
|
||||
Shawabkeh <http://max99x.com>`_
|
||||
|
||||
**Chapter 1**
|
||||
|
||||
- This will become a table of contents (this text will be scraped).
|
||||
{:toc}
|
||||
|
||||
`**Chapter 2: Implementing a Parser and AST** <PythonLangImpl2.html>`_
|
||||
|
||||
Introduction
|
||||
============
|
||||
|
||||
Welcome to the "Implementing a language with LLVM" tutorial. This
|
||||
tutorial runs through the implementation of a simple language, showing
|
||||
how fun and easy it can be. This tutorial will get you up and started as
|
||||
well as help to build a framework you can extend to other languages. The
|
||||
code in this tutorial can also be used as a playground to hack on other
|
||||
LLVM specific things.
|
||||
|
||||
It is useful to point out ahead of time that this tutorial is really
|
||||
about teaching compiler techniques and LLVM specifically, *not* about
|
||||
teaching modern and sane software engineering principles. In practice,
|
||||
this means that we'll take a number of shortcuts to simplify the
|
||||
exposition. If you dig in and use the code as a basis for future
|
||||
projects, fixing its deficiencies shouldn't be hard.
|
||||
|
||||
We've tried to put this tutorial together in a way that makes chapters
|
||||
easy to skip over if you are already familiar with or are uninterested
|
||||
in the various pieces. The structure of the tutorial is:
|
||||
|
||||
- **`Chapter 1 <#language>`_: Introduction to the Kaleidoscope
|
||||
language, and the definition of its Lexer** -- This shows where we
|
||||
are going and the basic functionality that we want it to do. In order
|
||||
to make this tutorial maximally understandable and hackable, we
|
||||
choose to implement everything in Python instead of using lexer and
|
||||
parser generators. LLVM obviously works just fine with such tools,
|
||||
feel free to use one if you prefer.
|
||||
|
||||
- **`Chapter 2 <PythonLangImpl2.html>`_: Implementing a Parser and
|
||||
AST** -- With the lexer in place, we can talk about parsing
|
||||
techniques and basic AST construction. This tutorial describes
|
||||
recursive descent parsing and operator precedence parsing. Nothing in
|
||||
Chapters 1 or 2 is LLVM-specific, the code doesn't even import the
|
||||
LLVM modules at this point. :)
|
||||
|
||||
- **`Chapter 3 <PythonLangImpl3.html>`_: Code generation to LLVM IR**
|
||||
-- With the AST ready, we can show off how easy generation of LLVM IR
|
||||
really is.
|
||||
|
||||
- **`Chapter 4 <PythonLangImpl4.html>`_: Adding JIT and Optimizer
|
||||
support** -- Because a lot of people are interested in using LLVM as
|
||||
a JIT, we'll dive right into it and show you the 3 lines it takes to
|
||||
add JIT support. LLVM is also useful in many other ways, but this is
|
||||
one simple and "sexy" way to shows off its power. :)
|
||||
|
||||
- **`Chapter 5 <PythonLangImpl5.html>`_: Extending the Language:
|
||||
Control Flow** -- With the language up and running, we show how to
|
||||
extend it with control flow operations (if/then/else and a 'for'
|
||||
loop). This gives us a chance to talk about simple SSA construction
|
||||
and control flow.
|
||||
|
||||
- **`Chapter 6 <PythonLangImpl6.html>`_: Extending the Language:
|
||||
User-defined Operators** -- This is a silly but fun chapter that
|
||||
talks about extending the language to let the user program define
|
||||
their own arbitrary unary and binary operators (with assignable
|
||||
precedence!). This lets us build a significant piece of the
|
||||
"language" as library routines.
|
||||
|
||||
- **`Chapter 7 <PythonLangImpl7.html>`_: Extending the Language:
|
||||
Mutable Variables** -- This chapter talks about adding user-defined
|
||||
local variables along with an assignment operator. The interesting
|
||||
part about this is how easy and trivial it is to construct SSA form
|
||||
in LLVM: no, LLVM does *not* require your front-end to construct SSA
|
||||
form!
|
||||
|
||||
- **`Chapter 8 <PythonLangImpl8.html>`_: Conclusion and other useful
|
||||
LLVM tidbits** -- This chapter wraps up the series by talking about
|
||||
potential ways to extend the language, but also includes a bunch of
|
||||
pointers to info about "special topics" like adding garbage
|
||||
collection support, exceptions, debugging, support for "spaghetti
|
||||
stacks", and a bunch of other tips and tricks.
|
||||
|
||||
By the end of the tutorial, we'll have written a bit less than 540 lines
|
||||
of non-comment, non-blank, lines of code. With this small amount of
|
||||
code, we'll have built up a very reasonable compiler for a non-trivial
|
||||
language including a hand-written lexer, parser, AST, as well as code
|
||||
generation support with a JIT compiler. While other systems may have
|
||||
interesting "hello world" tutorials, I think the breadth of this
|
||||
tutorial is a great testament to the strengths of LLVM and why you
|
||||
should consider it if you're interested in language or compiler design.
|
||||
|
||||
A note about this tutorial: we expect you to extend the language and
|
||||
play with it on your own. Take the code and go crazy hacking away at it,
|
||||
compilers don't need to be scary creatures - it can be a lot of fun to
|
||||
play with languages!
|
||||
|
||||
--------------
|
||||
|
||||
The Basic Language # {#language}
|
||||
================================
|
||||
|
||||
This tutorial will be illustrated with a toy language that we'll call
|
||||
"`Kaleidoscope <http://en.wikipedia.org/wiki/Kaleidoscope>`_\ " (derived
|
||||
from "meaning beautiful, form, and view"). Kaleidoscope is a procedural
|
||||
language that allows you to define functions, use conditionals, math,
|
||||
etc. Over the course of the tutorial, we'll extend Kaleidoscope to
|
||||
support the if/then/else construct, a for loop, user defined operators,
|
||||
JIT compilation with a simple command line interface, etc.
|
||||
|
||||
Because we want to keep things simple, the only datatype in Kaleidoscope
|
||||
is a 64-bit floating point type. As such, all values are implicitly
|
||||
double precision and the language doesn't require type declarations.
|
||||
This gives the language a very nice and simple syntax. For example, the
|
||||
following simple example computes `Fibonacci
|
||||
numbers <http://en.wikipedia.org/wiki/Fibonacci_number>`_:
|
||||
|
||||
{% highlight python %} # Compute the x'th fibonacci number. def fib(x)
|
||||
if x < 3 then 1 else fib(x-1)+fib(x-2)
|
||||
|
||||
This expression will compute the 40th number.
|
||||
=============================================
|
||||
|
||||
fib(40) {% endhighlight %}
|
||||
|
||||
We also allow Kaleidoscope to call into standard library functions (the
|
||||
LLVM JIT makes this completely trivial). This means that you can use the
|
||||
'extern' keyword to define a function before you use it (this is also
|
||||
useful for mutually recursive functions). For example:
|
||||
|
||||
{% highlight python %} extern sin(arg); extern cos(arg); extern
|
||||
atan2(arg1 arg2);
|
||||
|
||||
atan2(sin(0.4), cos(42)) {% endhighlight %}
|
||||
|
||||
A more interesting example is included in Chapter 6 where we write a
|
||||
little Kaleidoscope application that
|
||||
`displays <PythonLangImpl6.html#example>`_ a Mandelbrot Set at various
|
||||
levels of magnification.
|
||||
|
||||
Lets dive into the implementation of this language!
|
||||
|
||||
--------------
|
||||
|
||||
The Lexer # {#lexer}
|
||||
====================
|
||||
|
||||
When it comes to implementing a language, the first thing needed is the
|
||||
ability to process a text file and recognize what it says. The
|
||||
traditional way to do this is to use a
|
||||
`lexer <http://en.wikipedia.org/wiki/Lexical_analysis>`_" (aka
|
||||
'scanner') to break the input up into "tokens". Each token returned by
|
||||
the lexer includes a token type and potentially some metadata (e.g. the
|
||||
numeric value of a number). First, we define the possibilities:
|
||||
|
||||
{% highlight python %} # The lexer yields one of these types for each
|
||||
token. class EOFToken(object): pass
|
||||
|
||||
class DefToken(object): pass
|
||||
|
||||
class ExternToken(object): pass
|
||||
|
||||
class IdentifierToken(object): def **init**\ (self, name): self.name =
|
||||
name
|
||||
|
||||
class NumberToken(object): def **init**\ (self, value): self.value =
|
||||
value
|
||||
|
||||
class CharacterToken(object): def **init**\ (self, char): self.char =
|
||||
char def **eq**\ (self, other): return isinstance(other, CharacterToken)
|
||||
and self.char == other.char def **ne**\ (self, other): return not self
|
||||
== other {% endhighlight %}
|
||||
|
||||
Each token yielded by our lexer will be of one of the above types. For
|
||||
simple tokens that are always the same, like the "def" keyword, the
|
||||
lexer will yield ``DefToken()``>. Identifiers, numbers and characters,
|
||||
on the other hand, have extra data, so when the lexer encounteres the
|
||||
number 123.45, it will emit it as ``NumberToken(123.45)``. An identifier
|
||||
``foo`` will be emitted as ``IdentifierToken('foo')``. And finally, an
|
||||
unknown character like '+' will be returned as ``CharacterToken('+')``.
|
||||
You may notice that we overload the equality and inequality operators
|
||||
for the characters; this will later simplify character comparisons in
|
||||
the parser code.
|
||||
|
||||
The actual implementation of the lexer is a single function called
|
||||
``Tokenize``, which takes a string and
|
||||
`yields <http://docs.python.org/reference/simple_stmts.html#the-yield-statement>`_
|
||||
tokens. For simplicity, we will use `regular
|
||||
expressions <http://docs.python.org/library/re.html>`_ to parse out the
|
||||
tokens. This is terribly inefficient, but perfectly sufficient for our
|
||||
needs.
|
||||
|
||||
First, we define the regular expressions for our tokens. Numbers and
|
||||
strings of digits, optionally followed by a period and another string of
|
||||
digits. Identifiers (and keywords) are alphanumeric string starting with
|
||||
a letter and comments are anything between a hash (``#``) and the end of
|
||||
the line.
|
||||
|
||||
{% highlight python %} import re
|
||||
|
||||
...
|
||||
|
||||
Regular expressions that tokens and comments of our language.
|
||||
=============================================================
|
||||
|
||||
REGEX\_NUMBER = re.compile('[0-9]+(?:.[0-9]+)?') REGEX\_IDENTIFIER =
|
||||
re.compile('[a-zA-Z][a-zA-Z0-9]\ *') REGEX\_COMMENT = re.compile('#.*')
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
Next, let's start defining the ``Tokenize`` function itself. The first
|
||||
thing we need to do is set up a loop that scans the string, while
|
||||
ignoring whitespace between tokens:
|
||||
|
||||
{% highlight python %} def Tokenize(string): while string: # Skip
|
||||
whitespace. if string[0].isspace(): string = string[1:] continue
|
||||
|
||||
::
|
||||
|
||||
...
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
Next we want to find out what the next token is. For this we run the
|
||||
regexes we defined above on the remainder of the string. To simplify the
|
||||
rest of the code, we run all three regexes each time. As mentioned
|
||||
above, inefficiencies are ignored for the purpose of this tutorial:
|
||||
|
||||
{% highlight python %} # Run regexes. comment\_match =
|
||||
REGEX\_COMMENT.match(string) number\_match = REGEX\_NUMBER.match(string)
|
||||
identifier\_match = REGEX\_IDENTIFIER.match(string) {% endhighlight %}
|
||||
|
||||
Now se check if any of the regexes matched. For comments, we simply
|
||||
ignore the captured match:
|
||||
|
||||
{% highlight python %} # Check if any of the regexes matched and yield
|
||||
the appropriate result. if comment\_match: comment =
|
||||
comment\_match.group(0) string = string[len(comment):] {% endhighlight
|
||||
python %}
|
||||
|
||||
For numbers, we yield the captured match, converted to a float and
|
||||
tagged with the appropriate token type:
|
||||
|
||||
{% highlight python %} elif number\_match: number =
|
||||
number\_match.group(0) yield NumberToken(float(number)) string =
|
||||
string[len(number):] {% endhighlight %}
|
||||
|
||||
The identifier case is a little more complex. We have to check for
|
||||
keywords to decide whether we have captured an identifier or a keyword:
|
||||
|
||||
{% highlight python %} elif identifier\_match: identifier =
|
||||
identifier\_match.group(0) # Check if we matched a keyword. if
|
||||
identifier == 'def': yield DefToken() elif identifier == 'extern': yield
|
||||
ExternToken() else: yield IdentifierToken(identifier) string =
|
||||
string[len(identifier):] {% endhighlight %}
|
||||
|
||||
Finally, if we haven't recognized a comment, a number of an identifier,
|
||||
we yield the current character as an "unknown character" token. This is
|
||||
used, for example, for operators like ``+`` or ``*``:
|
||||
|
||||
{% highlight python %} else: # Yield the unknown character. yield
|
||||
CharacterToken(string[0]) string = string[1:] {% endhighlight %}
|
||||
|
||||
Once we're done with the loop, we return a final end-of-file token:
|
||||
|
||||
{% highlight python %} yield EOFToken() {% endhighlight %}
|
||||
|
||||
With this, we have the complete lexer for the basic Kaleidoscope
|
||||
language (the `full code listing <PythonLangImpl2.html#code>`_ for the
|
||||
Lexer is available in the `next chapter <PythonLangImpl2.html>`_ of the
|
||||
tutorial). Next we'll `build a simple parser that uses this to build an
|
||||
Abstract Syntax Tree <PythonLangImpl2.html>`_. When we have that, we'll
|
||||
include a driver so that you can use the lexer and parser together.
|
||||
|
||||
--------------
|
||||
|
||||
**`Next: Implementing a Parser and AST <PythonLangImpl2.html>`_**
|
||||
|
|
@ -1,998 +0,0 @@
|
|||
---
|
||||
layout: page
|
||||
title: "Kaleidoscope: Chapter 2"
|
||||
---
|
||||
|
||||
# Implementing a Parser and AST
|
||||
|
||||
Written by [Chris Lattner](mailto:sabre@nondot.org)
|
||||
and [Max Shawabkeh](http://max99x.com)
|
||||
|
||||
|
||||
**Chapter 2**
|
||||
|
||||
|
||||
* This will become a table of contents (this text will be scraped).
|
||||
{:toc}
|
||||
|
||||
|
||||
**[Chapter 3: Code generation to LLVM IR](PythonLangImpl3.html)**
|
||||
|
||||
|
||||
# Introduction # {#intro}
|
||||
|
||||
Welcome to Chapter 2 of the
|
||||
[Implementing a language with LLVM](http://www.llvm.org/docs/tutorial/index.html)
|
||||
tutorial.
|
||||
This chapter shows you how to use the lexer, built in
|
||||
[Chapter 1](PythonLangImpl1.html), to build a full
|
||||
[parser](http://en.wikipedia.org/wiki/Parsing) for
|
||||
our Kaleidoscope language. Once we have a parser, we'll define and build an
|
||||
[Abstract Syntax Tree](http://en.wikipedia.org/wiki/Abstract_syntax_tree)
|
||||
(AST).
|
||||
|
||||
The parser we will build uses a combination of [Recursive Descent
|
||||
Parsing](http://en.wikipedia.org/wiki/Recursive_descent_parser) and
|
||||
[Operator-Precedence Parsing](http://en.wikipedia.org/wiki/Operator-precedence_parser)
|
||||
to parse the Kaleidoscope language (the latter for
|
||||
binary expressions and the former for everything else). Before we get to
|
||||
parsing though, lets talk about the output of the parser: the Abstract Syntax
|
||||
Tree.
|
||||
|
||||
* * *
|
||||
|
||||
|
||||
# The Abstract Syntax Tree (AST) # {#ast}
|
||||
|
||||
The AST for a program captures its behavior in such a way that it is easy for
|
||||
later stages of the compiler (e.g. code generation) to interpret. We basically
|
||||
want one object for each construct in the language, and the AST should closely
|
||||
model the language. In Kaleidoscope, we have expressions, a prototype, and a
|
||||
function object. We'll start with expressions first:
|
||||
|
||||
{% highlight python %}
|
||||
# Base class for all expression nodes.
|
||||
class ExpressionNode(object):
|
||||
pass
|
||||
|
||||
# Expression class for numeric literals like "1.0".
|
||||
class NumberExpressionNode(ExpressionNode):
|
||||
def __init__(self, value):
|
||||
self.value = value
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
The code above shows the definition of the base ExpressionNode class and one
|
||||
subclass which we use for numeric literals. The important thing to note about
|
||||
this code is that the NumberExpressionNode class captures the numeric value of
|
||||
the literal as an instance variable. This allows later phases of the compiler to
|
||||
know what the stored numeric value is.
|
||||
|
||||
Right now we only create the AST, so there are no useful methods on them.
|
||||
It would be very easy to add a virtual method to pretty print the code, for
|
||||
example. Here are the other expression AST node definitions that we'll use
|
||||
in the basic form of the Kaleidoscope language:
|
||||
|
||||
{% highlight python %}
|
||||
# Expression class for referencing a variable, like "a".
|
||||
class VariableExpressionNode(ExpressionNode):
|
||||
def __init__(self, name):
|
||||
self.name = name
|
||||
|
||||
# Expression class for a binary operator.
|
||||
class BinaryOperatorExpressionNode(ExpressionNode):
|
||||
def __init__(self, operator, left, right):
|
||||
self.operator = operator
|
||||
self.left = left
|
||||
self.right = right
|
||||
|
||||
# Expression class for function calls.
|
||||
class CallExpressionNode(ExpressionNode):
|
||||
def __init__(self, callee, args):
|
||||
self.callee = callee
|
||||
self.args = args
|
||||
{% endhighlight %}
|
||||
|
||||
This is all (intentionally) rather straight-forward: variables capture the
|
||||
variable name, binary operators capture their opcode (e.g. '+'), and calls
|
||||
capture a function name as well as a list of any argument expressions. One thing
|
||||
that is nice about our AST is that it captures the language features without
|
||||
talking about the syntax of the language. Note that there is no discussion about
|
||||
precedence of binary operators, lexical structure, etc.
|
||||
|
||||
For our basic language, these are all of the expression nodes we'll define.
|
||||
Because it doesn't have conditional control flow, it isn't Turing-complete;
|
||||
we'll fix that in a later installment. The two things we need next are a way
|
||||
to talk about the interface to a function, and a way to talk about functions
|
||||
themselves:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
# This class represents the "prototype" for a function, which captures its name,
|
||||
# and its argument names (thus implicitly the number of arguments the function
|
||||
# takes).
|
||||
class PrototypeNode(object):
|
||||
def __init__(self, name, args):
|
||||
self.name = name
|
||||
self.args = args
|
||||
|
||||
# This class represents a function definition itself.
|
||||
class FunctionNode(object):
|
||||
def __init__(self, prototype, body):
|
||||
self.prototype = prototype
|
||||
self.body = body
|
||||
{% endhighlight %}
|
||||
|
||||
In Kaleidoscope, functions are typed with just a count of their arguments.
|
||||
Since all values are double precision floating point, the type of each argument
|
||||
doesn't need to be stored anywhere. In a more aggressive and realistic
|
||||
language, the `ExpressionNode` class would probably have a type field.
|
||||
|
||||
With this scaffolding, we can now talk about parsing expressions and function
|
||||
bodies in Kaleidoscope.
|
||||
|
||||
* * *
|
||||
|
||||
|
||||
# Parser Basics # {#parserbasics}
|
||||
|
||||
Now that we have an AST to build, we need to define the parser code to build
|
||||
it. The idea here is that we want to parse something like `x + y` (which
|
||||
is returned as three tokens by the lexer) into an AST that could be generated
|
||||
with calls like this:
|
||||
|
||||
{% highlight python %}
|
||||
x = VariableExpressionNode('x')
|
||||
y = VariableExpressionNode('y')
|
||||
result = BinaryOperatorExpressionNode('+', x, y)
|
||||
{% endhighlight %}
|
||||
|
||||
In order to do this, we'll start by defining a lightweight `Parser`
|
||||
class with some basic helper routines:
|
||||
|
||||
{% highlight python %}
|
||||
class Parser(object):
|
||||
|
||||
def __init__(self, tokens, binop_precedence):
|
||||
self.tokens = tokens
|
||||
self.binop_precedence = binop_precedence
|
||||
self.Next()
|
||||
|
||||
# Provide a simple token buffer. Parser.current is the current token the
|
||||
# parser is looking at. Parser.Next() reads another token from the lexer and
|
||||
# updates Parser.current with its results.
|
||||
def Next(self):
|
||||
self.current = self.tokens.next()
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
This implements a simple token buffer around the lexer. This allows
|
||||
us to look one token ahead at what the lexer is returning. Every function in
|
||||
our parser will assume that `self.current` is the current token that
|
||||
needs to be parsed. Note that the first token is read as soon as the parser is
|
||||
instantiated. Let us ignore the `binop_precedence` parameter for now. It
|
||||
will be explained when we start [parsing binary operators](#parserbinops).
|
||||
|
||||
With these basic helper functions, we can implement the first
|
||||
piece of our grammar: numeric literals.
|
||||
|
||||
* * *
|
||||
|
||||
# Basic Expression Parsing # {#parserprimexprs}
|
||||
|
||||
We start with numeric literals, because they are the simplest to process.
|
||||
For each production in our grammar, we'll define a function which parses that
|
||||
production. For numeric literals, we have:
|
||||
|
||||
{% highlight python %}
|
||||
# numberexpr ::= number
|
||||
def ParseNumberExpr(self):
|
||||
result = NumberExpressionNode(self.current.value)
|
||||
self.Next() # consume the number.
|
||||
return result
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
This method is very simple: it expects to be called when the current token
|
||||
is a `NumberToken`. It takes the current number value, creates a
|
||||
`NumberExpressionNode`, advances to the next token, and finally returns.
|
||||
|
||||
|
||||
There are some interesting aspects to this. The most important one is that
|
||||
this routine eats all of the tokens that correspond to the production and
|
||||
returns the lexer buffer with the next token (which is not part of the grammar
|
||||
production) ready to go. This is a fairly standard way to go for recursive
|
||||
descent parsers. For a better example, the parenthesis operator is defined like
|
||||
this:
|
||||
|
||||
{% highlight python %}
|
||||
# parenexpr ::= '(' expression ')'
|
||||
def ParseParenExpr(self):
|
||||
self.Next() # eat '('.
|
||||
|
||||
contents = self.ParseExpression()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")".')
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return contents
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
This function illustrates an interesting aspect of the parser. The function
|
||||
uses recursion by calling `ParseExpression` (we will soon see that
|
||||
`ParseExpression` can call `ParseParenExpr`). This is powerful
|
||||
because it allows us to handle recursive grammars, and keeps each production
|
||||
very simple. Note that parentheses do not cause construction of AST nodes
|
||||
themselves. While we could do it this way, the most important role of
|
||||
parentheses are to guide the parser and provide grouping. Once the parser
|
||||
constructs the AST, parentheses are not needed.
|
||||
|
||||
The next simple production is for handling variable references and function
|
||||
calls:
|
||||
|
||||
{% highlight python %}
|
||||
# identifierexpr ::= identifier | identifier '(' expression* ')'
|
||||
def ParseIdentifierExpr(self):
|
||||
identifier_name = self.current.name
|
||||
self.Next() # eat identifier.
|
||||
|
||||
if self.current != CharacterToken('('): # Simple variable reference.
|
||||
return VariableExpressionNode(identifier_name);
|
||||
|
||||
# Call.
|
||||
self.Next() # eat '('.
|
||||
args = []
|
||||
if self.current != CharacterToken(')'):
|
||||
while True:
|
||||
args.append(self.ParseExpression())
|
||||
if self.current == CharacterToken(')'):
|
||||
break
|
||||
elif self.current != CharacterToken(','):
|
||||
raise RuntimeError('Expected ")" or "," in argument list.')
|
||||
self.Next()
|
||||
|
||||
self.Next() # eat ')'.
|
||||
return CallExpressionNode(identifier_name, args)
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
This routine follows the same style as the other routines. It expects to be
|
||||
called if the current token is an `IdentifierToken`. It also has
|
||||
recursion and error handling. One interesting aspect of this is that it uses
|
||||
*look-ahead* to determine if the current identifier is a stand alone
|
||||
variable reference or if it is a function call expression. It handles this by
|
||||
checking to see if the token after the identifier is a '(' token, constructing
|
||||
either a `VariableExpressionNode` or `CallExpressionNode` as
|
||||
appropriate.
|
||||
|
||||
Now that we have all of our simple expression-parsing logic in place, we can
|
||||
define a helper function to wrap it together into one entry point. We call this
|
||||
class of expressions "primary" expressions, for reasons that will become more
|
||||
clear [later in the tutorial](PythonLangImpl6.html#unary). In order
|
||||
to parse an arbitrary primary expression, we need to determine what sort of
|
||||
expression it is:
|
||||
|
||||
{% highlight python %}
|
||||
# primary ::= identifierexpr | numberexpr | parenexpr
|
||||
def ParsePrimary(self):
|
||||
if isinstance(self.current, IdentifierToken):
|
||||
return self.ParseIdentifierExpr()
|
||||
elif isinstance(self.current, NumberToken):
|
||||
return self.ParseNumberExpr();
|
||||
elif self.current == CharacterToken('('):
|
||||
return self.ParseParenExpr()
|
||||
else:
|
||||
raise RuntimeError('Unknown token when expecting an expression.')
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
Now that you see the definition of this function, it is more obvious why we
|
||||
can assume the state of `Parser.current` in the various functions. This
|
||||
uses look-ahead to determine which sort of expression is being inspected, and
|
||||
then parses it with a function call.
|
||||
|
||||
Now that basic expressions are handled, we need to handle binary expressions.
|
||||
They are a bit more complex.
|
||||
|
||||
|
||||
|
||||
|
||||
* * *
|
||||
|
||||
# Binary Expression Parsing # {#parserbinops}
|
||||
|
||||
Binary expressions are significantly harder to parse because they are often
|
||||
ambiguous. For example, when given the string `x+y*z`, the parser can choose
|
||||
to parse it as either `(x+y)*z` or `x+(y*z)`. With common definitions from
|
||||
mathematics, we expect the later parse, because `*` (multiplication) has
|
||||
higher *precedence* than `+` (addition).
|
||||
|
||||
There are many ways to handle this, but an elegant and efficient way is
|
||||
to use [Operator-Precedence Parsing](http://en.wikipedia.org/wiki/Operator-precedence_parser).
|
||||
This parsing technique uses the precedence of binary operators to
|
||||
guide recursion. To start with, we need a table of precedences. Remember the
|
||||
`binop_precedence` parameter we passed to the `Parser`
|
||||
constructor? Now is the time to use it:
|
||||
|
||||
{% highlight python %}
|
||||
def main():
|
||||
# Install standard binary operators.
|
||||
# 1 is lowest possible precedence. 40 is the highest.
|
||||
operator_precedence = {
|
||||
'<': 10,
|
||||
'+': 20,
|
||||
'-': 20,
|
||||
'*': 40
|
||||
}
|
||||
|
||||
# Run the main `interpreter loop`.
|
||||
while True:
|
||||
|
||||
...
|
||||
|
||||
parser = Parser(Tokenize(raw), operator_precedence)
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
For the basic form of Kaleidoscope, we will only support 4 binary operators
|
||||
(this can obviously be extended by you, our brave and intrepid reader). Having a
|
||||
dictionary makes it easy to add new operators and makes it clear that the
|
||||
algorithm doesn't depend on the specific operators involved, but it would be
|
||||
easy enough to eliminate the map and hardcode the comparisons.
|
||||
|
||||
We also define a helper function to get the precedence of the current token,
|
||||
or -1 if the token is not a binary operator:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
# Gets the precedence of the current token, or -1 if the token is not a binary
|
||||
# operator.
|
||||
def GetCurrentTokenPrecedence(self):
|
||||
if isinstance(self.current, CharacterToken):
|
||||
return self.binop_precedence.get(self.current.char, -1)
|
||||
else:
|
||||
return -1
|
||||
{% endhighlight %}
|
||||
|
||||
With the helper above defined, we can now start parsing binary expressions.
|
||||
The basic idea of operator precedence parsing is to break down an expression
|
||||
with potentially ambiguous binary operators into pieces. Consider, for example,
|
||||
the expression `a+b+(c+d)*e*f+g`. Operator precedence parsing considers this
|
||||
as a stream of primary expressions separated by binary operators. As such,
|
||||
it will first parse the leading primary expression `a`, then it will see the
|
||||
pairs `[+, b] [+, (c+d)] [*, e] [*, f] and [+, g]`. Note that because parentheses
|
||||
are primary expressions, the binary expression parser doesn't need to worry
|
||||
about nested subexpressions like (c+d) at all.
|
||||
|
||||
|
||||
|
||||
To start, an expression is a primary expression potentially followed by a
|
||||
sequence of `[binop,primaryexpr]` pairs:
|
||||
|
||||
{% highlight python %}
|
||||
# expression ::= primary binoprhs
|
||||
def ParseExpression(self):
|
||||
left = self.ParsePrimary()
|
||||
return self.ParseBinOpRHS(left, 0)
|
||||
{% endhighlight %}
|
||||
|
||||
`ParseBinOpRHS` is the function that parses the sequence of pairs for
|
||||
us. It takes a precedence and a pointer to an expression for the part that has
|
||||
been parsed so far. Note that `x` is a perfectly valid expression: As such,
|
||||
`binoprhs` is allowed to be empty, in which case it returns the expression that
|
||||
is passed into it. In our example above, the code passes the expression for `a`
|
||||
into `ParseBinOpRHS` and the current token is `+`.
|
||||
|
||||
The precedence value passed into `ParseBinOpRHS` indicates the *
|
||||
minimal operator precedence* that the function is allowed to eat. For
|
||||
example, if the current pair stream is `[+, x]` and `ParseBinOpRHS` is
|
||||
passed in a precedence of 40, it will not consume any tokens (because the
|
||||
precedence of '+' is only 20). With this in mind, `ParseBinOpRHS` starts
|
||||
with:
|
||||
|
||||
{% highlight python %}
|
||||
# binoprhs ::= (operator primary)*
|
||||
def ParseBinOpRHS(self, left, left_precedence):
|
||||
# If this is a binary operator, find its precedence.
|
||||
while True:
|
||||
precedence = self.GetCurrentTokenPrecedence()
|
||||
|
||||
# If this is a binary operator that binds at least as tightly as the
|
||||
# current one, consume it; otherwise we are done.
|
||||
if precedence < left_precedence:
|
||||
return left
|
||||
{% endhighlight %}
|
||||
|
||||
This code gets the precedence of the current token and checks to see if if is
|
||||
too low. Because we defined invalid tokens to have a precedence of -1, this
|
||||
check implicitly knows that the pair-stream ends when the token stream runs out
|
||||
of binary operators. If this check succeeds, we know that the token is a binary
|
||||
operator and that it will be included in this expression:
|
||||
|
||||
{% highlight python %}
|
||||
binary_operator = self.current.char
|
||||
self.Next() # eat the operator.
|
||||
|
||||
# Parse the primary expression after the binary operator.
|
||||
right = self.ParsePrimary()
|
||||
{% endhighlight %}
|
||||
|
||||
As such, this code eats (and remembers) the binary operator and then parses
|
||||
the primary expression that follows. This builds up the whole pair, the first of
|
||||
which is `[+, b]` for the running example.
|
||||
|
||||
Now that we parsed the left-hand side of an expression and one pair of the
|
||||
RHS sequence, we have to decide which way the expression associates. In
|
||||
particular, we could have `(a+b) binop unparsed` or `a + (b binop unparsed)`.
|
||||
To determine this, we look ahead at `binop` to determine its precedence and
|
||||
compare it to BinOp's precedence (which is '+' in this case):
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
# If binary_operator binds less tightly with right than the operator after
|
||||
# right, let the pending operator take right as its left.
|
||||
next_precedence = self.GetCurrentTokenPrecedence()
|
||||
if precedence < next_precedence:
|
||||
{% endhighlight %}
|
||||
|
||||
If the precedence of the binop to the right of `RHS` is lower or equal to the
|
||||
precedence of our current operator, then we know that the parentheses associate
|
||||
as `(a+b) binop ...`. In our example, the current operator is `+` and the next
|
||||
operator is `+`, we know that they have the same precedence. In this case we'll
|
||||
create the AST node for `a+b`, and then continue parsing:
|
||||
|
||||
{% highlight python %}
|
||||
if precedence < next_precedence:
|
||||
... if body omitted ...
|
||||
|
||||
# Merge left/right.
|
||||
left = BinaryOperatorExpressionNode(binary_operator, left, right);
|
||||
{% endhighlight %}
|
||||
|
||||
In our example above, this will turn `a+b+` into `(a+b)` and execute the next
|
||||
iteration of the loop, with `+` as the current token. The code above will eat,
|
||||
remember, and parse `(c+d)` as the primary expression, which makes the
|
||||
current pair equal to `[+, (c+d)]`. It will then evaluate the 'if' conditional
|
||||
above with `*` as the binop to the right of the primary. In this case, the
|
||||
precedence of `*` is higher than the precedence of `+` so the if condition will
|
||||
be entered.
|
||||
|
||||
The critical question left here is `how can the if condition parse the right
|
||||
hand side in full`? In particular, to build the AST correctly for our example,
|
||||
it needs to get all of ` ( c + d ) * e * f` as the RHS expression variable. The code to
|
||||
do this is surprisingly simple (code from the above two blocks duplicated for
|
||||
context):
|
||||
|
||||
{% highlight python %}
|
||||
# If binary_operator binds less tightly with right than the operator after
|
||||
# right, let the pending operator take right as its left.
|
||||
next_precedence = self.GetCurrentTokenPrecedence()
|
||||
if precedence < next_precedence:
|
||||
right = self.ParseBinOpRHS(right, precedence + 1)
|
||||
|
||||
# Merge left/right.
|
||||
left = BinaryOperatorExpressionNode(binary_operator, left, right)
|
||||
{% endhighlight %}
|
||||
|
||||
At this point, we know that the binary operator to the RHS of our primary
|
||||
has higher precedence than the binop we are currently parsing. As such, we know
|
||||
that any sequence of pairs whose operators are all higher precedence than `+`
|
||||
should be parsed together and returned as `RHS`. To do this, we recursively
|
||||
invoke the `ParseBinOpRHS` function specifying `precedence + 1` as the
|
||||
minimum precedence required for it to continue. In our example above, this
|
||||
will cause it to return the AST node for `(c+d)*e*f` as RHS, which is then set
|
||||
as the RHS of the '+' expression.
|
||||
|
||||
Finally, on the next iteration of the while loop, the `+g` piece is parsed
|
||||
and added to the AST. With this little bit of code (11 non-trivial lines), we
|
||||
correctly handle fully general binary expression parsing in a very elegant way.
|
||||
This was a whirlwind tour of this code, and it is somewhat subtle. I recommend
|
||||
running through it with a few tough examples to see how it works.
|
||||
|
||||
|
||||
This wraps up handling of expressions. At this point, we can point the
|
||||
parser at an arbitrary token stream and build an expression from it, stopping
|
||||
at the first token that is not part of the expression. Next up we need to
|
||||
handle function definitions, etc.
|
||||
|
||||
|
||||
* * *
|
||||
|
||||
# Parsing the Rest # {#parsertop}
|
||||
|
||||
|
||||
The next thing missing is handling of function prototypes. In Kaleidoscope,
|
||||
these are used both for 'extern' function declarations as well as function body
|
||||
definitions. The code to do this is straight-forward and not very interesting
|
||||
(once you've survived expressions):
|
||||
|
||||
{% highlight python %}
|
||||
# prototype ::= id '(' id* ')'
|
||||
def ParsePrototype(self):
|
||||
if not isinstance(self.current, IdentifierToken):
|
||||
raise RuntimeError('Expected function name in prototype.')
|
||||
|
||||
function_name = self.current.name
|
||||
self.Next() # eat function name.
|
||||
|
||||
if self.current != CharacterToken('('):
|
||||
raise RuntimeError('Expected "(" in prototype.')
|
||||
self.Next() # eat '('.
|
||||
|
||||
arg_names = []
|
||||
while isinstance(self.current, IdentifierToken):
|
||||
arg_names.append(self.current.name)
|
||||
self.Next()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")" in prototype.')
|
||||
|
||||
# Success.
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return PrototypeNode(function_name, arg_names)
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
Given this, a function definition is very simple, just a prototype plus
|
||||
an expression to implement the body:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
# definition ::= 'def' prototype expression
|
||||
def ParseDefinition(self):
|
||||
self.Next() # eat def.
|
||||
proto = self.ParsePrototype()
|
||||
body = self.ParseExpression()
|
||||
return FunctionNode(proto, body)
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
In addition, we support 'extern' to declare functions like 'sin' and 'cos' as
|
||||
well as to support forward declaration of user functions. These 'extern's are
|
||||
just prototypes with no body:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
# external ::= 'extern' prototype
|
||||
def ParseExtern(self):
|
||||
self.Next() # eat extern.
|
||||
return self.ParsePrototype()
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
Finally, we'll also let the user type in arbitrary top-level expressions and
|
||||
evaluate them on the fly. We will handle this by defining anonymous nullary
|
||||
(zero argument) functions for them:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
# toplevelexpr ::= expression
|
||||
def ParseTopLevelExpr(self):
|
||||
proto = PrototypeNode('', [])
|
||||
return FunctionNode(proto, self.ParseExpression())
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
Now that we have all the pieces, let's build a little driver that will let us
|
||||
actually *execute* this code we've built!
|
||||
|
||||
|
||||
|
||||
* * *
|
||||
|
||||
# The Driver # {#driver}
|
||||
|
||||
|
||||
The driver for this simply invokes all of the parsing pieces with a top-level
|
||||
dispatch loop. There isn't much interesting here, so I'll just include the
|
||||
top-level loop. See [below](#code) for full code.
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
# Run the main "interpreter loop".
|
||||
while True:
|
||||
print 'ready>',
|
||||
try:
|
||||
raw = raw_input()
|
||||
except KeyboardInterrupt:
|
||||
return
|
||||
|
||||
parser = Parser(Tokenize(raw), operator_precedence)
|
||||
while True:
|
||||
# top ::= definition | external | expression | EOF
|
||||
if isinstance(parser.current, EOFToken):
|
||||
break
|
||||
if isinstance(parser.current, DefToken):
|
||||
parser.HandleDefinition()
|
||||
elif isinstance(parser.current, ExternToken):
|
||||
parser.HandleExtern()
|
||||
else:
|
||||
parser.HandleTopLevelExpression()
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
Here we create a new `Parser` for each line read, and try to parse out
|
||||
all the expressions, declarations and definitions in the line. We also allow the
|
||||
user to quit using Ctrl+C.
|
||||
|
||||
|
||||
|
||||
* * *
|
||||
|
||||
# Conclusions # {#conclusions}
|
||||
|
||||
|
||||
With just under 330 lines of commented code (200 lines of non-comment,
|
||||
non-blank code), we fully defined our minimal language, including a lexer,
|
||||
parser, and AST builder. With this done, the executable will validate
|
||||
Kaleidoscope code and tell us if it is grammatically invalid. For
|
||||
example, here is a sample interaction:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
$ python kaleidoscope.py
|
||||
ready> def foo(x y) x+foo(y, 4.0)
|
||||
Parsed a function definition.
|
||||
ready> def foo(x y) x+y y
|
||||
Parsed a function definition.
|
||||
Parsed a top-level expression.
|
||||
ready> def foo(x y) x+y )
|
||||
Parsed a function definition.
|
||||
Error: Unknown token when expecting an expression.
|
||||
ready> extern sin(a);
|
||||
Parsed an extern.
|
||||
ready> ^C
|
||||
$
|
||||
{% endhighlight %}
|
||||
|
||||
There is a lot of room for extension here. You can define new AST nodes,
|
||||
extend the language in many ways, etc. In the
|
||||
[next installment](PythonLangImpl3.html), we will describe how to
|
||||
generate LLVM Intermediate Representation (IR) from the AST.
|
||||
|
||||
|
||||
* * *
|
||||
|
||||
# Full Code Listing # {#code}
|
||||
|
||||
|
||||
Here is the complete code listing for this and the previous chapter.
|
||||
Note that it is fully self-contained: you don't need LLVM or any external
|
||||
libraries at all for this.
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
#!/usr/bin/env python
|
||||
|
||||
import re
|
||||
|
||||
################################################################################
|
||||
## Lexer
|
||||
################################################################################
|
||||
|
||||
# The lexer yields one of these types for each token.
|
||||
class EOFToken(object):
|
||||
pass
|
||||
|
||||
class DefToken(object):
|
||||
pass
|
||||
|
||||
class ExternToken(object):
|
||||
pass
|
||||
|
||||
class IdentifierToken(object):
|
||||
def __init__(self, name): self.name = name
|
||||
|
||||
class NumberToken(object):
|
||||
def __init__(self, value): self.value = value
|
||||
|
||||
class CharacterToken(object):
|
||||
def __init__(self, char): self.char = char
|
||||
def __eq__(self, other):
|
||||
return isinstance(other, CharacterToken) and self.char == other.char
|
||||
def __ne__(self, other): return not self == other
|
||||
|
||||
# Regular expressions that tokens and comments of our language.
|
||||
REGEX_NUMBER = re.compile('[0-9]+(?:\.[0-9]+)?')
|
||||
REGEX_IDENTIFIER = re.compile('[a-zA-Z][a-zA-Z0-9]*')
|
||||
REGEX_COMMENT = re.compile('#.*')
|
||||
|
||||
def Tokenize(string):
|
||||
while string:
|
||||
# Skip whitespace.
|
||||
if string[0].isspace():
|
||||
string = string[1:]
|
||||
continue
|
||||
|
||||
# Run regexes.
|
||||
comment_match = REGEX_COMMENT.match(string)
|
||||
number_match = REGEX_NUMBER.match(string)
|
||||
identifier_match = REGEX_IDENTIFIER.match(string)
|
||||
|
||||
# Check if any of the regexes matched and yield the appropriate result.
|
||||
if comment_match:
|
||||
comment = comment_match.group(0)
|
||||
string = string[len(comment):]
|
||||
elif number_match:
|
||||
number = number_match.group(0)
|
||||
yield NumberToken(float(number))
|
||||
string = string[len(number):]
|
||||
elif identifier_match:
|
||||
identifier = identifier_match.group(0)
|
||||
# Check if we matched a keyword.
|
||||
if identifier == 'def':
|
||||
yield DefToken()
|
||||
elif identifier == 'extern':
|
||||
yield ExternToken()
|
||||
else:
|
||||
yield IdentifierToken(identifier)
|
||||
string = string[len(identifier):]
|
||||
else:
|
||||
# Yield the ASCII value of the unknown character.
|
||||
yield CharacterToken(string[0])
|
||||
string = string[1:]
|
||||
|
||||
yield EOFToken()
|
||||
|
||||
################################################################################
|
||||
## Abstract Syntax Tree (aka Parse Tree)
|
||||
################################################################################
|
||||
|
||||
# Base class for all expression nodes.
|
||||
class ExpressionNode(object):
|
||||
pass
|
||||
|
||||
# Expression class for numeric literals like "1.0".
|
||||
class NumberExpressionNode(ExpressionNode):
|
||||
def __init__(self, value):
|
||||
self.value = value
|
||||
|
||||
# Expression class for referencing a variable, like "a".
|
||||
class VariableExpressionNode(ExpressionNode):
|
||||
def __init__(self, name):
|
||||
self.name = name
|
||||
|
||||
# Expression class for a binary operator.
|
||||
class BinaryOperatorExpressionNode(ExpressionNode):
|
||||
def __init__(self, operator, left, right):
|
||||
self.operator = operator
|
||||
self.left = left
|
||||
self.right = right
|
||||
|
||||
# Expression class for function calls.
|
||||
class CallExpressionNode(ExpressionNode):
|
||||
def __init__(self, callee, args):
|
||||
self.callee = callee
|
||||
self.args = args
|
||||
|
||||
# This class represents the "prototype" for a function, which captures its name,
|
||||
# and its argument names (thus implicitly the number of arguments the function
|
||||
# takes).
|
||||
class PrototypeNode(object):
|
||||
def __init__(self, name, args):
|
||||
self.name = name
|
||||
self.args = args
|
||||
|
||||
# This class represents a function definition itself.
|
||||
class FunctionNode(object):
|
||||
def __init__(self, prototype, body):
|
||||
self.prototype = prototype
|
||||
self.body = body
|
||||
|
||||
|
||||
################################################################################
|
||||
## Parser
|
||||
################################################################################
|
||||
|
||||
class Parser(object):
|
||||
|
||||
def __init__(self, tokens, binop_precedence):
|
||||
self.tokens = tokens
|
||||
self.binop_precedence = binop_precedence
|
||||
self.Next()
|
||||
|
||||
# Provide a simple token buffer. Parser.current is the current token the
|
||||
# parser is looking at. Parser.Next() reads another token from the lexer and
|
||||
# updates Parser.current with its results.
|
||||
def Next(self):
|
||||
self.current = self.tokens.next()
|
||||
|
||||
# Gets the precedence of the current token, or -1 if the token is not a binary
|
||||
# operator.
|
||||
def GetCurrentTokenPrecedence(self):
|
||||
if isinstance(self.current, CharacterToken):
|
||||
return self.binop_precedence.get(self.current.char, -1)
|
||||
else:
|
||||
return -1
|
||||
|
||||
# identifierexpr ::= identifier | identifier '(' expression* ')'
|
||||
def ParseIdentifierExpr(self):
|
||||
identifier_name = self.current.name
|
||||
self.Next() # eat identifier.
|
||||
|
||||
if self.current != CharacterToken('('): # Simple variable reference.
|
||||
return VariableExpressionNode(identifier_name)
|
||||
|
||||
# Call.
|
||||
self.Next() # eat '('.
|
||||
args = []
|
||||
if self.current != CharacterToken(')'):
|
||||
while True:
|
||||
args.append(self.ParseExpression())
|
||||
if self.current == CharacterToken(')'):
|
||||
break
|
||||
elif self.current != CharacterToken(','):
|
||||
raise RuntimeError('Expected ")" or "," in argument list.')
|
||||
self.Next()
|
||||
|
||||
self.Next() # eat ')'.
|
||||
return CallExpressionNode(identifier_name, args)
|
||||
|
||||
# numberexpr ::= number
|
||||
def ParseNumberExpr(self):
|
||||
result = NumberExpressionNode(self.current.value)
|
||||
self.Next() # consume the number.
|
||||
return result
|
||||
|
||||
# parenexpr ::= '(' expression ')'
|
||||
def ParseParenExpr(self):
|
||||
self.Next() # eat '('.
|
||||
|
||||
contents = self.ParseExpression()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")".')
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return contents
|
||||
|
||||
# primary ::= identifierexpr | numberexpr | parenexpr
|
||||
def ParsePrimary(self):
|
||||
if isinstance(self.current, IdentifierToken):
|
||||
return self.ParseIdentifierExpr()
|
||||
elif isinstance(self.current, NumberToken):
|
||||
return self.ParseNumberExpr()
|
||||
elif self.current == CharacterToken('('):
|
||||
return self.ParseParenExpr()
|
||||
else:
|
||||
raise RuntimeError('Unknown token when expecting an expression.')
|
||||
|
||||
# binoprhs ::= (operator primary)*
|
||||
def ParseBinOpRHS(self, left, left_precedence):
|
||||
# If this is a binary operator, find its precedence.
|
||||
while True:
|
||||
precedence = self.GetCurrentTokenPrecedence()
|
||||
|
||||
# If this is a binary operator that binds at least as tightly as the
|
||||
# current one, consume it; otherwise we are done.
|
||||
if precedence < left_precedence:
|
||||
return left
|
||||
|
||||
binary_operator = self.current.char
|
||||
self.Next() # eat the operator.
|
||||
|
||||
# Parse the primary expression after the binary operator.
|
||||
right = self.ParsePrimary()
|
||||
|
||||
# If binary_operator binds less tightly with right than the operator after
|
||||
# right, let the pending operator take right as its left.
|
||||
next_precedence = self.GetCurrentTokenPrecedence()
|
||||
if precedence < next_precedence:
|
||||
right = self.ParseBinOpRHS(right, precedence + 1)
|
||||
|
||||
# Merge left/right.
|
||||
left = BinaryOperatorExpressionNode(binary_operator, left, right)
|
||||
|
||||
# expression ::= primary binoprhs
|
||||
def ParseExpression(self):
|
||||
left = self.ParsePrimary()
|
||||
return self.ParseBinOpRHS(left, 0)
|
||||
|
||||
# prototype ::= id '(' id* ')'
|
||||
def ParsePrototype(self):
|
||||
if not isinstance(self.current, IdentifierToken):
|
||||
raise RuntimeError('Expected function name in prototype.')
|
||||
|
||||
function_name = self.current.name
|
||||
self.Next() # eat function name.
|
||||
|
||||
if self.current != CharacterToken('('):
|
||||
raise RuntimeError('Expected "(" in prototype.')
|
||||
self.Next() # eat '('.
|
||||
|
||||
arg_names = []
|
||||
while isinstance(self.current, IdentifierToken):
|
||||
arg_names.append(self.current.name)
|
||||
self.Next()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")" in prototype.')
|
||||
|
||||
# Success.
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return PrototypeNode(function_name, arg_names)
|
||||
|
||||
# definition ::= 'def' prototype expression
|
||||
def ParseDefinition(self):
|
||||
self.Next() # eat def.
|
||||
proto = self.ParsePrototype()
|
||||
body = self.ParseExpression()
|
||||
return FunctionNode(proto, body)
|
||||
|
||||
# toplevelexpr ::= expression
|
||||
def ParseTopLevelExpr(self):
|
||||
proto = PrototypeNode('', [])
|
||||
return FunctionNode(proto, self.ParseExpression())
|
||||
|
||||
# external ::= 'extern' prototype
|
||||
def ParseExtern(self):
|
||||
self.Next() # eat extern.
|
||||
return self.ParsePrototype()
|
||||
|
||||
# Top-Level parsing
|
||||
def HandleDefinition(self):
|
||||
self.Handle(self.ParseDefinition, 'Parsed a function definition.')
|
||||
|
||||
def HandleExtern(self):
|
||||
self.Handle(self.ParseExtern, 'Parsed an extern.')
|
||||
|
||||
def HandleTopLevelExpression(self):
|
||||
self.Handle(self.ParseTopLevelExpr, 'Parsed a top-level expression.')
|
||||
|
||||
def Handle(self, function, message):
|
||||
try:
|
||||
function()
|
||||
print message
|
||||
except Exception, e:
|
||||
print 'Error:', e
|
||||
try:
|
||||
self.Next() # Skip for error recovery.
|
||||
except:
|
||||
pass
|
||||
|
||||
################################################################################
|
||||
## Main driver code.
|
||||
################################################################################
|
||||
|
||||
def main():
|
||||
# Install standard binary operators.
|
||||
# 1 is lowest possible precedence. 40 is the highest.
|
||||
operator_precedence = {
|
||||
'<': 10,
|
||||
'+': 20,
|
||||
'-': 20,
|
||||
'*': 40
|
||||
}
|
||||
|
||||
# Run the main "interpreter loop".
|
||||
while True:
|
||||
print 'ready>',
|
||||
try:
|
||||
raw = raw_input()
|
||||
except KeyboardInterrupt:
|
||||
return
|
||||
|
||||
parser = Parser(Tokenize(raw), operator_precedence)
|
||||
while True:
|
||||
# top ::= definition | external | expression | EOF
|
||||
if isinstance(parser.current, EOFToken):
|
||||
break
|
||||
if isinstance(parser.current, DefToken):
|
||||
parser.HandleDefinition()
|
||||
elif isinstance(parser.current, ExternToken):
|
||||
parser.HandleExtern()
|
||||
else:
|
||||
parser.HandleTopLevelExpression()
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
* * *
|
||||
|
||||
**[Next: Implementing Code Generation to LLVM IR](PythonLangImpl3.html)**
|
||||
892
docs/source/doc/kaleidoscope/PythonLangImpl2.rst
Normal file
892
docs/source/doc/kaleidoscope/PythonLangImpl2.rst
Normal file
|
|
@ -0,0 +1,892 @@
|
|||
+------------------------------------+
|
||||
| layout: page |
|
||||
+------------------------------------+
|
||||
| title: "Kaleidoscope: Chapter 2" |
|
||||
+------------------------------------+
|
||||
|
||||
Implementing a Parser and AST
|
||||
=============================
|
||||
|
||||
Written by `Chris Lattner <mailto:sabre@nondot.org>`_ and `Max
|
||||
Shawabkeh <http://max99x.com>`_
|
||||
|
||||
**Chapter 2**
|
||||
|
||||
- This will become a table of contents (this text will be scraped).
|
||||
{:toc}
|
||||
|
||||
**`Chapter 3: Code generation to LLVM IR <PythonLangImpl3.html>`_**
|
||||
|
||||
Introduction # {#intro}
|
||||
=======================
|
||||
|
||||
Welcome to Chapter 2 of the `Implementing a language with
|
||||
LLVM <http://www.llvm.org/docs/tutorial/index.html>`_ tutorial. This
|
||||
chapter shows you how to use the lexer, built in `Chapter
|
||||
1 <PythonLangImpl1.html>`_, to build a full
|
||||
`parser <http://en.wikipedia.org/wiki/Parsing>`_ for our Kaleidoscope
|
||||
language. Once we have a parser, we'll define and build an `Abstract
|
||||
Syntax Tree <http://en.wikipedia.org/wiki/Abstract_syntax_tree>`_ (AST).
|
||||
|
||||
The parser we will build uses a combination of `Recursive Descent
|
||||
Parsing <http://en.wikipedia.org/wiki/Recursive_descent_parser>`_ and
|
||||
`Operator-Precedence
|
||||
Parsing <http://en.wikipedia.org/wiki/Operator-precedence_parser>`_ to
|
||||
parse the Kaleidoscope language (the latter for binary expressions and
|
||||
the former for everything else). Before we get to parsing though, lets
|
||||
talk about the output of the parser: the Abstract Syntax Tree.
|
||||
|
||||
--------------
|
||||
|
||||
The Abstract Syntax Tree (AST) # {#ast}
|
||||
=======================================
|
||||
|
||||
The AST for a program captures its behavior in such a way that it is
|
||||
easy for later stages of the compiler (e.g. code generation) to
|
||||
interpret. We basically want one object for each construct in the
|
||||
language, and the AST should closely model the language. In
|
||||
Kaleidoscope, we have expressions, a prototype, and a function object.
|
||||
We'll start with expressions first:
|
||||
|
||||
{% highlight python %} # Base class for all expression nodes. class
|
||||
ExpressionNode(object): pass
|
||||
|
||||
Expression class for numeric literals like "1.0".
|
||||
=================================================
|
||||
|
||||
class NumberExpressionNode(ExpressionNode): def **init**\ (self, value):
|
||||
self.value = value
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
The code above shows the definition of the base ExpressionNode class and
|
||||
one subclass which we use for numeric literals. The important thing to
|
||||
note about this code is that the NumberExpressionNode class captures the
|
||||
numeric value of the literal as an instance variable. This allows later
|
||||
phases of the compiler to know what the stored numeric value is.
|
||||
|
||||
Right now we only create the AST, so there are no useful methods on
|
||||
them. It would be very easy to add a virtual method to pretty print the
|
||||
code, for example. Here are the other expression AST node definitions
|
||||
that we'll use in the basic form of the Kaleidoscope language:
|
||||
|
||||
{% highlight python %} # Expression class for referencing a variable,
|
||||
like "a". class VariableExpressionNode(ExpressionNode): def
|
||||
**init**\ (self, name): self.name = name
|
||||
|
||||
Expression class for a binary operator.
|
||||
=======================================
|
||||
|
||||
class BinaryOperatorExpressionNode(ExpressionNode): def **init**\ (self,
|
||||
operator, left, right): self.operator = operator self.left = left
|
||||
self.right = right
|
||||
|
||||
Expression class for function calls.
|
||||
====================================
|
||||
|
||||
class CallExpressionNode(ExpressionNode): def **init**\ (self, callee,
|
||||
args): self.callee = callee self.args = args {% endhighlight %}
|
||||
|
||||
This is all (intentionally) rather straight-forward: variables capture
|
||||
the variable name, binary operators capture their opcode (e.g. '+'), and
|
||||
calls capture a function name as well as a list of any argument
|
||||
expressions. One thing that is nice about our AST is that it captures
|
||||
the language features without talking about the syntax of the language.
|
||||
Note that there is no discussion about precedence of binary operators,
|
||||
lexical structure, etc.
|
||||
|
||||
For our basic language, these are all of the expression nodes we'll
|
||||
define. Because it doesn't have conditional control flow, it isn't
|
||||
Turing-complete; we'll fix that in a later installment. The two things
|
||||
we need next are a way to talk about the interface to a function, and a
|
||||
way to talk about functions themselves:
|
||||
|
||||
{% highlight python %} # This class represents the "prototype" for a
|
||||
function, which captures its name, # and its argument names (thus
|
||||
implicitly the number of arguments the function # takes). class
|
||||
PrototypeNode(object): def **init**\ (self, name, args): self.name =
|
||||
name self.args = args
|
||||
|
||||
This class represents a function definition itself.
|
||||
===================================================
|
||||
|
||||
class FunctionNode(object): def **init**\ (self, prototype, body):
|
||||
self.prototype = prototype self.body = body {% endhighlight %}
|
||||
|
||||
In Kaleidoscope, functions are typed with just a count of their
|
||||
arguments. Since all values are double precision floating point, the
|
||||
type of each argument doesn't need to be stored anywhere. In a more
|
||||
aggressive and realistic language, the ``ExpressionNode`` class would
|
||||
probably have a type field.
|
||||
|
||||
With this scaffolding, we can now talk about parsing expressions and
|
||||
function bodies in Kaleidoscope.
|
||||
|
||||
--------------
|
||||
|
||||
Parser Basics # {#parserbasics}
|
||||
===============================
|
||||
|
||||
Now that we have an AST to build, we need to define the parser code to
|
||||
build it. The idea here is that we want to parse something like
|
||||
``x + y`` (which is returned as three tokens by the lexer) into an AST
|
||||
that could be generated with calls like this:
|
||||
|
||||
{% highlight python %} x = VariableExpressionNode('x') y =
|
||||
VariableExpressionNode('y') result = BinaryOperatorExpressionNode('+',
|
||||
x, y) {% endhighlight %}
|
||||
|
||||
In order to do this, we'll start by defining a lightweight ``Parser``
|
||||
class with some basic helper routines:
|
||||
|
||||
{% highlight python %} class Parser(object):
|
||||
|
||||
def **init**\ (self, tokens, binop\_precedence): self.tokens = tokens
|
||||
self.binop\_precedence = binop\_precedence self.Next()
|
||||
|
||||
# Provide a simple token buffer. Parser.current is the current token the
|
||||
# parser is looking at. Parser.Next() reads another token from the lexer
|
||||
and # updates Parser.current with its results. def Next(self):
|
||||
self.current = self.tokens.next() {% endhighlight %}
|
||||
|
||||
This implements a simple token buffer around the lexer. This allows us
|
||||
to look one token ahead at what the lexer is returning. Every function
|
||||
in our parser will assume that ``self.current`` is the current token
|
||||
that needs to be parsed. Note that the first token is read as soon as
|
||||
the parser is instantiated. Let us ignore the ``binop_precedence``
|
||||
parameter for now. It will be explained when we start `parsing binary
|
||||
operators <#parserbinops>`_.
|
||||
|
||||
With these basic helper functions, we can implement the first piece of
|
||||
our grammar: numeric literals.
|
||||
|
||||
--------------
|
||||
|
||||
Basic Expression Parsing # {#parserprimexprs}
|
||||
=============================================
|
||||
|
||||
We start with numeric literals, because they are the simplest to
|
||||
process. For each production in our grammar, we'll define a function
|
||||
which parses that production. For numeric literals, we have:
|
||||
|
||||
{% highlight python %} # numberexpr ::= number def
|
||||
ParseNumberExpr(self): result = NumberExpressionNode(self.current.value)
|
||||
self.Next() # consume the number. return result {% endhighlight %}
|
||||
|
||||
This method is very simple: it expects to be called when the current
|
||||
token is a ``NumberToken``. It takes the current number value, creates a
|
||||
``NumberExpressionNode``, advances to the next token, and finally
|
||||
returns.
|
||||
|
||||
There are some interesting aspects to this. The most important one is
|
||||
that this routine eats all of the tokens that correspond to the
|
||||
production and returns the lexer buffer with the next token (which is
|
||||
not part of the grammar production) ready to go. This is a fairly
|
||||
standard way to go for recursive descent parsers. For a better example,
|
||||
the parenthesis operator is defined like this:
|
||||
|
||||
{% highlight python %} # parenexpr ::= '(' expression ')' def
|
||||
ParseParenExpr(self): self.Next() # eat '('.
|
||||
|
||||
::
|
||||
|
||||
contents = self.ParseExpression()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")".')
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return contents
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
This function illustrates an interesting aspect of the parser. The
|
||||
function uses recursion by calling ``ParseExpression`` (we will soon see
|
||||
that ``ParseExpression`` can call ``ParseParenExpr``). This is powerful
|
||||
because it allows us to handle recursive grammars, and keeps each
|
||||
production very simple. Note that parentheses do not cause construction
|
||||
of AST nodes themselves. While we could do it this way, the most
|
||||
important role of parentheses are to guide the parser and provide
|
||||
grouping. Once the parser constructs the AST, parentheses are not
|
||||
needed.
|
||||
|
||||
The next simple production is for handling variable references and
|
||||
function calls:
|
||||
|
||||
{% highlight python %} # identifierexpr ::= identifier \| identifier '('
|
||||
expression\* ')' def ParseIdentifierExpr(self): identifier\_name =
|
||||
self.current.name self.Next() # eat identifier.
|
||||
|
||||
::
|
||||
|
||||
if self.current != CharacterToken('('): # Simple variable reference.
|
||||
return VariableExpressionNode(identifier_name);
|
||||
|
||||
# Call.
|
||||
self.Next() # eat '('.
|
||||
args = []
|
||||
if self.current != CharacterToken(')'):
|
||||
while True:
|
||||
args.append(self.ParseExpression())
|
||||
if self.current == CharacterToken(')'):
|
||||
break
|
||||
elif self.current != CharacterToken(','):
|
||||
raise RuntimeError('Expected ")" or "," in argument list.')
|
||||
self.Next()
|
||||
|
||||
self.Next() # eat ')'.
|
||||
return CallExpressionNode(identifier_name, args)
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
This routine follows the same style as the other routines. It expects to
|
||||
be called if the current token is an ``IdentifierToken``. It also has
|
||||
recursion and error handling. One interesting aspect of this is that it
|
||||
uses *look-ahead* to determine if the current identifier is a stand
|
||||
alone variable reference or if it is a function call expression. It
|
||||
handles this by checking to see if the token after the identifier is a
|
||||
'(' token, constructing either a ``VariableExpressionNode`` or
|
||||
``CallExpressionNode`` as appropriate.
|
||||
|
||||
Now that we have all of our simple expression-parsing logic in place, we
|
||||
can define a helper function to wrap it together into one entry point.
|
||||
We call this class of expressions "primary" expressions, for reasons
|
||||
that will become more clear `later in the
|
||||
tutorial <PythonLangImpl6.html#unary>`_. In order to parse an arbitrary
|
||||
primary expression, we need to determine what sort of expression it is:
|
||||
|
||||
{% highlight python %} # primary ::= identifierexpr \| numberexpr \|
|
||||
parenexpr def ParsePrimary(self): if isinstance(self.current,
|
||||
IdentifierToken): return self.ParseIdentifierExpr() elif
|
||||
isinstance(self.current, NumberToken): return self.ParseNumberExpr();
|
||||
elif self.current == CharacterToken('('): return self.ParseParenExpr()
|
||||
else: raise RuntimeError('Unknown token when expecting an expression.')
|
||||
{% endhighlight %}
|
||||
|
||||
Now that you see the definition of this function, it is more obvious why
|
||||
we can assume the state of ``Parser.current`` in the various functions.
|
||||
This uses look-ahead to determine which sort of expression is being
|
||||
inspected, and then parses it with a function call.
|
||||
|
||||
Now that basic expressions are handled, we need to handle binary
|
||||
expressions. They are a bit more complex.
|
||||
|
||||
--------------
|
||||
|
||||
Binary Expression Parsing # {#parserbinops}
|
||||
===========================================
|
||||
|
||||
Binary expressions are significantly harder to parse because they are
|
||||
often ambiguous. For example, when given the string ``x+y*z``, the
|
||||
parser can choose to parse it as either ``(x+y)*z`` or ``x+(y*z)``. With
|
||||
common definitions from mathematics, we expect the later parse, because
|
||||
``*`` (multiplication) has higher *precedence* than ``+`` (addition).
|
||||
|
||||
There are many ways to handle this, but an elegant and efficient way is
|
||||
to use `Operator-Precedence
|
||||
Parsing <http://en.wikipedia.org/wiki/Operator-precedence_parser>`_.
|
||||
This parsing technique uses the precedence of binary operators to guide
|
||||
recursion. To start with, we need a table of precedences. Remember the
|
||||
``binop_precedence`` parameter we passed to the ``Parser`` constructor?
|
||||
Now is the time to use it:
|
||||
|
||||
{% highlight python %} def main(): # Install standard binary operators.
|
||||
# 1 is lowest possible precedence. 40 is the highest.
|
||||
operator\_precedence = { '<': 10, '+': 20, '-': 20, '\*': 40 }
|
||||
|
||||
# Run the main ``interpreter loop``. while True:
|
||||
|
||||
::
|
||||
|
||||
...
|
||||
|
||||
parser = Parser(Tokenize(raw), operator_precedence)
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
For the basic form of Kaleidoscope, we will only support 4 binary
|
||||
operators (this can obviously be extended by you, our brave and intrepid
|
||||
reader). Having a dictionary makes it easy to add new operators and
|
||||
makes it clear that the algorithm doesn't depend on the specific
|
||||
operators involved, but it would be easy enough to eliminate the map and
|
||||
hardcode the comparisons.
|
||||
|
||||
We also define a helper function to get the precedence of the current
|
||||
token, or -1 if the token is not a binary operator:
|
||||
|
||||
{% highlight python %} # Gets the precedence of the current token, or -1
|
||||
if the token is not a binary # operator. def
|
||||
GetCurrentTokenPrecedence(self): if isinstance(self.current,
|
||||
CharacterToken): return self.binop\_precedence.get(self.current.char,
|
||||
-1) else: return -1 {% endhighlight %}
|
||||
|
||||
With the helper above defined, we can now start parsing binary
|
||||
expressions. The basic idea of operator precedence parsing is to break
|
||||
down an expression with potentially ambiguous binary operators into
|
||||
pieces. Consider, for example, the expression ``a+b+(c+d)*e*f+g``.
|
||||
Operator precedence parsing considers this as a stream of primary
|
||||
expressions separated by binary operators. As such, it will first parse
|
||||
the leading primary expression ``a``, then it will see the pairs
|
||||
``[+, b] [+, (c+d)] [*, e] [*, f] and [+, g]``. Note that because
|
||||
parentheses are primary expressions, the binary expression parser
|
||||
doesn't need to worry about nested subexpressions like (c+d) at all.
|
||||
|
||||
To start, an expression is a primary expression potentially followed by
|
||||
a sequence of ``[binop,primaryexpr]`` pairs:
|
||||
|
||||
{% highlight python %} # expression ::= primary binoprhs def
|
||||
ParseExpression(self): left = self.ParsePrimary() return
|
||||
self.ParseBinOpRHS(left, 0) {% endhighlight %}
|
||||
|
||||
``ParseBinOpRHS`` is the function that parses the sequence of pairs for
|
||||
us. It takes a precedence and a pointer to an expression for the part
|
||||
that has been parsed so far. Note that ``x`` is a perfectly valid
|
||||
expression: As such, ``binoprhs`` is allowed to be empty, in which case
|
||||
it returns the expression that is passed into it. In our example above,
|
||||
the code passes the expression for ``a`` into ``ParseBinOpRHS`` and the
|
||||
current token is ``+``.
|
||||
|
||||
The precedence value passed into ``ParseBinOpRHS`` indicates the \*
|
||||
minimal operator precedence\* that the function is allowed to eat. For
|
||||
example, if the current pair stream is ``[+, x]`` and ``ParseBinOpRHS``
|
||||
is passed in a precedence of 40, it will not consume any tokens (because
|
||||
the precedence of '+' is only 20). With this in mind, ``ParseBinOpRHS``
|
||||
starts with:
|
||||
|
||||
{% highlight python %} # binoprhs ::= (operator primary)\* def
|
||||
ParseBinOpRHS(self, left, left\_precedence): # If this is a binary
|
||||
operator, find its precedence. while True: precedence =
|
||||
self.GetCurrentTokenPrecedence()
|
||||
|
||||
::
|
||||
|
||||
# If this is a binary operator that binds at least as tightly as the
|
||||
# current one, consume it; otherwise we are done.
|
||||
if precedence < left_precedence:
|
||||
return left
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
This code gets the precedence of the current token and checks to see if
|
||||
if is too low. Because we defined invalid tokens to have a precedence of
|
||||
-1, this check implicitly knows that the pair-stream ends when the token
|
||||
stream runs out of binary operators. If this check succeeds, we know
|
||||
that the token is a binary operator and that it will be included in this
|
||||
expression:
|
||||
|
||||
{% highlight python %} binary\_operator = self.current.char self.Next()
|
||||
# eat the operator.
|
||||
|
||||
::
|
||||
|
||||
# Parse the primary expression after the binary operator.
|
||||
right = self.ParsePrimary()
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
As such, this code eats (and remembers) the binary operator and then
|
||||
parses the primary expression that follows. This builds up the whole
|
||||
pair, the first of which is ``[+, b]`` for the running example.
|
||||
|
||||
Now that we parsed the left-hand side of an expression and one pair of
|
||||
the RHS sequence, we have to decide which way the expression associates.
|
||||
In particular, we could have ``(a+b) binop unparsed`` or
|
||||
``a + (b binop unparsed)``. To determine this, we look ahead at
|
||||
``binop`` to determine its precedence and compare it to BinOp's
|
||||
precedence (which is '+' in this case):
|
||||
|
||||
{% highlight python %} # If binary\_operator binds less tightly with
|
||||
right than the operator after # right, let the pending operator take
|
||||
right as its left. next\_precedence = self.GetCurrentTokenPrecedence()
|
||||
if precedence < next\_precedence: {% endhighlight %}
|
||||
|
||||
If the precedence of the binop to the right of ``RHS`` is lower or equal
|
||||
to the precedence of our current operator, then we know that the
|
||||
parentheses associate as ``(a+b) binop ...``. In our example, the
|
||||
current operator is ``+`` and the next operator is ``+``, we know that
|
||||
they have the same precedence. In this case we'll create the AST node
|
||||
for ``a+b``, and then continue parsing:
|
||||
|
||||
{% highlight python %} if precedence < next\_precedence: ... if body
|
||||
omitted ...
|
||||
|
||||
::
|
||||
|
||||
# Merge left/right.
|
||||
left = BinaryOperatorExpressionNode(binary_operator, left, right);
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
In our example above, this will turn ``a+b+`` into ``(a+b)`` and execute
|
||||
the next iteration of the loop, with ``+`` as the current token. The
|
||||
code above will eat, remember, and parse ``(c+d)`` as the primary
|
||||
expression, which makes the current pair equal to ``[+, (c+d)]``. It
|
||||
will then evaluate the 'if' conditional above with ``*`` as the binop to
|
||||
the right of the primary. In this case, the precedence of ``*`` is
|
||||
higher than the precedence of ``+`` so the if condition will be entered.
|
||||
|
||||
The critical question left here is
|
||||
``how can the if condition parse the right hand side in full``? In
|
||||
particular, to build the AST correctly for our example, it needs to get
|
||||
all of ``( c + d ) * e * f`` as the RHS expression variable. The code to
|
||||
do this is surprisingly simple (code from the above two blocks
|
||||
duplicated for context):
|
||||
|
||||
{% highlight python %} # If binary\_operator binds less tightly with
|
||||
right than the operator after # right, let the pending operator take
|
||||
right as its left. next\_precedence = self.GetCurrentTokenPrecedence()
|
||||
if precedence < next\_precedence: right = self.ParseBinOpRHS(right,
|
||||
precedence + 1)
|
||||
|
||||
::
|
||||
|
||||
# Merge left/right.
|
||||
left = BinaryOperatorExpressionNode(binary_operator, left, right)
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
At this point, we know that the binary operator to the RHS of our
|
||||
primary has higher precedence than the binop we are currently parsing.
|
||||
As such, we know that any sequence of pairs whose operators are all
|
||||
higher precedence than ``+`` should be parsed together and returned as
|
||||
``RHS``. To do this, we recursively invoke the ``ParseBinOpRHS``
|
||||
function specifying ``precedence + 1`` as the minimum precedence
|
||||
required for it to continue. In our example above, this will cause it to
|
||||
return the AST node for ``(c+d)*e*f`` as RHS, which is then set as the
|
||||
RHS of the '+' expression.
|
||||
|
||||
Finally, on the next iteration of the while loop, the ``+g`` piece is
|
||||
parsed and added to the AST. With this little bit of code (11
|
||||
non-trivial lines), we correctly handle fully general binary expression
|
||||
parsing in a very elegant way. This was a whirlwind tour of this code,
|
||||
and it is somewhat subtle. I recommend running through it with a few
|
||||
tough examples to see how it works.
|
||||
|
||||
This wraps up handling of expressions. At this point, we can point the
|
||||
parser at an arbitrary token stream and build an expression from it,
|
||||
stopping at the first token that is not part of the expression. Next up
|
||||
we need to handle function definitions, etc.
|
||||
|
||||
--------------
|
||||
|
||||
Parsing the Rest # {#parsertop}
|
||||
===============================
|
||||
|
||||
The next thing missing is handling of function prototypes. In
|
||||
Kaleidoscope, these are used both for 'extern' function declarations as
|
||||
well as function body definitions. The code to do this is
|
||||
straight-forward and not very interesting (once you've survived
|
||||
expressions):
|
||||
|
||||
{% highlight python %} # prototype ::= id '(' id\* ')' def
|
||||
ParsePrototype(self): if not isinstance(self.current, IdentifierToken):
|
||||
raise RuntimeError('Expected function name in prototype.')
|
||||
|
||||
::
|
||||
|
||||
function_name = self.current.name
|
||||
self.Next() # eat function name.
|
||||
|
||||
if self.current != CharacterToken('('):
|
||||
raise RuntimeError('Expected "(" in prototype.')
|
||||
self.Next() # eat '('.
|
||||
|
||||
arg_names = []
|
||||
while isinstance(self.current, IdentifierToken):
|
||||
arg_names.append(self.current.name)
|
||||
self.Next()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")" in prototype.')
|
||||
|
||||
# Success.
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return PrototypeNode(function_name, arg_names)
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
Given this, a function definition is very simple, just a prototype plus
|
||||
an expression to implement the body:
|
||||
|
||||
{% highlight python %} # definition ::= 'def' prototype expression def
|
||||
ParseDefinition(self): self.Next() # eat def. proto =
|
||||
self.ParsePrototype() body = self.ParseExpression() return
|
||||
FunctionNode(proto, body) {% endhighlight %}
|
||||
|
||||
In addition, we support 'extern' to declare functions like 'sin' and
|
||||
'cos' as well as to support forward declaration of user functions. These
|
||||
'extern's are just prototypes with no body:
|
||||
|
||||
{% highlight python %} # external ::= 'extern' prototype def
|
||||
ParseExtern(self): self.Next() # eat extern. return
|
||||
self.ParsePrototype() {% endhighlight %}
|
||||
|
||||
Finally, we'll also let the user type in arbitrary top-level expressions
|
||||
and evaluate them on the fly. We will handle this by defining anonymous
|
||||
nullary (zero argument) functions for them:
|
||||
|
||||
{% highlight python %} # toplevelexpr ::= expression def
|
||||
ParseTopLevelExpr(self): proto = PrototypeNode('', []) return
|
||||
FunctionNode(proto, self.ParseExpression()) {% endhighlight %}
|
||||
|
||||
Now that we have all the pieces, let's build a little driver that will
|
||||
let us actually *execute* this code we've built!
|
||||
|
||||
--------------
|
||||
|
||||
The Driver # {#driver}
|
||||
======================
|
||||
|
||||
The driver for this simply invokes all of the parsing pieces with a
|
||||
top-level dispatch loop. There isn't much interesting here, so I'll just
|
||||
include the top-level loop. See `below <#code>`_ for full code.
|
||||
|
||||
{% highlight python %} # Run the main "interpreter loop". while True:
|
||||
print 'ready>', try: raw = raw\_input() except KeyboardInterrupt: return
|
||||
|
||||
::
|
||||
|
||||
parser = Parser(Tokenize(raw), operator_precedence)
|
||||
while True:
|
||||
# top ::= definition | external | expression | EOF
|
||||
if isinstance(parser.current, EOFToken):
|
||||
break
|
||||
if isinstance(parser.current, DefToken):
|
||||
parser.HandleDefinition()
|
||||
elif isinstance(parser.current, ExternToken):
|
||||
parser.HandleExtern()
|
||||
else:
|
||||
parser.HandleTopLevelExpression()
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
Here we create a new ``Parser`` for each line read, and try to parse out
|
||||
all the expressions, declarations and definitions in the line. We also
|
||||
allow the user to quit using Ctrl+C.
|
||||
|
||||
--------------
|
||||
|
||||
Conclusions # {#conclusions}
|
||||
============================
|
||||
|
||||
With just under 330 lines of commented code (200 lines of non-comment,
|
||||
non-blank code), we fully defined our minimal language, including a
|
||||
lexer, parser, and AST builder. With this done, the executable will
|
||||
validate Kaleidoscope code and tell us if it is grammatically invalid.
|
||||
For example, here is a sample interaction:
|
||||
|
||||
{% highlight python %} $ python kaleidoscope.py ready> def foo(x y)
|
||||
x+foo(y, 4.0) Parsed a function definition. ready> def foo(x y) x+y y
|
||||
Parsed a function definition. Parsed a top-level expression. ready> def
|
||||
foo(x y) x+y ) Parsed a function definition. Error: Unknown token when
|
||||
expecting an expression. ready> extern sin(a); Parsed an extern. ready>
|
||||
^C $ {% endhighlight %}
|
||||
|
||||
There is a lot of room for extension here. You can define new AST nodes,
|
||||
extend the language in many ways, etc. In the `next
|
||||
installment <PythonLangImpl3.html>`_, we will describe how to generate
|
||||
LLVM Intermediate Representation (IR) from the AST.
|
||||
|
||||
--------------
|
||||
|
||||
Full Code Listing # {#code}
|
||||
===========================
|
||||
|
||||
Here is the complete code listing for this and the previous chapter.
|
||||
Note that it is fully self-contained: you don't need LLVM or any
|
||||
external libraries at all for this.
|
||||
|
||||
{% highlight python %} #!/usr/bin/env python
|
||||
|
||||
import re
|
||||
|
||||
Lexer
|
||||
-----
|
||||
|
||||
The lexer yields one of these types for each token.
|
||||
===================================================
|
||||
|
||||
class EOFToken(object): pass
|
||||
|
||||
class DefToken(object): pass
|
||||
|
||||
class ExternToken(object): pass
|
||||
|
||||
class IdentifierToken(object): def **init**\ (self, name): self.name =
|
||||
name
|
||||
|
||||
class NumberToken(object): def **init**\ (self, value): self.value =
|
||||
value
|
||||
|
||||
class CharacterToken(object): def **init**\ (self, char): self.char =
|
||||
char def **eq**\ (self, other): return isinstance(other, CharacterToken)
|
||||
and self.char == other.char def **ne**\ (self, other): return not self
|
||||
== other
|
||||
|
||||
Regular expressions that tokens and comments of our language.
|
||||
=============================================================
|
||||
|
||||
REGEX\_NUMBER = re.compile('[0-9]+(?:.[0-9]+)?') REGEX\_IDENTIFIER =
|
||||
re.compile('[a-zA-Z][a-zA-Z0-9]\ *') REGEX\_COMMENT = re.compile('#.*')
|
||||
|
||||
def Tokenize(string): while string: # Skip whitespace. if
|
||||
string[0].isspace(): string = string[1:] continue
|
||||
|
||||
::
|
||||
|
||||
# Run regexes.
|
||||
comment_match = REGEX_COMMENT.match(string)
|
||||
number_match = REGEX_NUMBER.match(string)
|
||||
identifier_match = REGEX_IDENTIFIER.match(string)
|
||||
|
||||
# Check if any of the regexes matched and yield the appropriate result.
|
||||
if comment_match:
|
||||
comment = comment_match.group(0)
|
||||
string = string[len(comment):]
|
||||
elif number_match:
|
||||
number = number_match.group(0)
|
||||
yield NumberToken(float(number))
|
||||
string = string[len(number):]
|
||||
elif identifier_match:
|
||||
identifier = identifier_match.group(0)
|
||||
# Check if we matched a keyword.
|
||||
if identifier == 'def':
|
||||
yield DefToken()
|
||||
elif identifier == 'extern':
|
||||
yield ExternToken()
|
||||
else:
|
||||
yield IdentifierToken(identifier)
|
||||
string = string[len(identifier):]
|
||||
else:
|
||||
# Yield the ASCII value of the unknown character.
|
||||
yield CharacterToken(string[0])
|
||||
string = string[1:]
|
||||
|
||||
yield EOFToken()
|
||||
|
||||
Abstract Syntax Tree (aka Parse Tree)
|
||||
-------------------------------------
|
||||
|
||||
Base class for all expression nodes.
|
||||
====================================
|
||||
|
||||
class ExpressionNode(object): pass
|
||||
|
||||
Expression class for numeric literals like "1.0".
|
||||
=================================================
|
||||
|
||||
class NumberExpressionNode(ExpressionNode): def **init**\ (self, value):
|
||||
self.value = value
|
||||
|
||||
Expression class for referencing a variable, like "a".
|
||||
======================================================
|
||||
|
||||
class VariableExpressionNode(ExpressionNode): def **init**\ (self,
|
||||
name): self.name = name
|
||||
|
||||
Expression class for a binary operator.
|
||||
=======================================
|
||||
|
||||
class BinaryOperatorExpressionNode(ExpressionNode): def **init**\ (self,
|
||||
operator, left, right): self.operator = operator self.left = left
|
||||
self.right = right
|
||||
|
||||
Expression class for function calls.
|
||||
====================================
|
||||
|
||||
class CallExpressionNode(ExpressionNode): def **init**\ (self, callee,
|
||||
args): self.callee = callee self.args = args
|
||||
|
||||
This class represents the "prototype" for a function, which captures its name,
|
||||
==============================================================================
|
||||
|
||||
and its argument names (thus implicitly the number of arguments the function
|
||||
============================================================================
|
||||
|
||||
takes).
|
||||
=======
|
||||
|
||||
class PrototypeNode(object): def **init**\ (self, name, args): self.name
|
||||
= name self.args = args
|
||||
|
||||
This class represents a function definition itself.
|
||||
===================================================
|
||||
|
||||
class FunctionNode(object): def **init**\ (self, prototype, body):
|
||||
self.prototype = prototype self.body = body
|
||||
|
||||
Parser
|
||||
------
|
||||
|
||||
class Parser(object):
|
||||
|
||||
def **init**\ (self, tokens, binop\_precedence): self.tokens = tokens
|
||||
self.binop\_precedence = binop\_precedence self.Next()
|
||||
|
||||
# Provide a simple token buffer. Parser.current is the current token the
|
||||
# parser is looking at. Parser.Next() reads another token from the lexer
|
||||
and # updates Parser.current with its results. def Next(self):
|
||||
self.current = self.tokens.next()
|
||||
|
||||
# Gets the precedence of the current token, or -1 if the token is not a
|
||||
binary # operator. def GetCurrentTokenPrecedence(self): if
|
||||
isinstance(self.current, CharacterToken): return
|
||||
self.binop\_precedence.get(self.current.char, -1) else: return -1
|
||||
|
||||
# identifierexpr ::= identifier \| identifier '(' expression\* ')' def
|
||||
ParseIdentifierExpr(self): identifier\_name = self.current.name
|
||||
self.Next() # eat identifier.
|
||||
|
||||
::
|
||||
|
||||
if self.current != CharacterToken('('): # Simple variable reference.
|
||||
return VariableExpressionNode(identifier_name)
|
||||
|
||||
# Call.
|
||||
self.Next() # eat '('.
|
||||
args = []
|
||||
if self.current != CharacterToken(')'):
|
||||
while True:
|
||||
args.append(self.ParseExpression())
|
||||
if self.current == CharacterToken(')'):
|
||||
break
|
||||
elif self.current != CharacterToken(','):
|
||||
raise RuntimeError('Expected ")" or "," in argument list.')
|
||||
self.Next()
|
||||
|
||||
self.Next() # eat ')'.
|
||||
return CallExpressionNode(identifier_name, args)
|
||||
|
||||
# numberexpr ::= number def ParseNumberExpr(self): result =
|
||||
NumberExpressionNode(self.current.value) self.Next() # consume the
|
||||
number. return result
|
||||
|
||||
# parenexpr ::= '(' expression ')' def ParseParenExpr(self): self.Next()
|
||||
# eat '('.
|
||||
|
||||
::
|
||||
|
||||
contents = self.ParseExpression()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")".')
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return contents
|
||||
|
||||
# primary ::= identifierexpr \| numberexpr \| parenexpr def
|
||||
ParsePrimary(self): if isinstance(self.current, IdentifierToken): return
|
||||
self.ParseIdentifierExpr() elif isinstance(self.current, NumberToken):
|
||||
return self.ParseNumberExpr() elif self.current == CharacterToken('('):
|
||||
return self.ParseParenExpr() else: raise RuntimeError('Unknown token
|
||||
when expecting an expression.')
|
||||
|
||||
# binoprhs ::= (operator primary)\* def ParseBinOpRHS(self, left,
|
||||
left\_precedence): # If this is a binary operator, find its precedence.
|
||||
while True: precedence = self.GetCurrentTokenPrecedence()
|
||||
|
||||
::
|
||||
|
||||
# If this is a binary operator that binds at least as tightly as the
|
||||
# current one, consume it; otherwise we are done.
|
||||
if precedence < left_precedence:
|
||||
return left
|
||||
|
||||
binary_operator = self.current.char
|
||||
self.Next() # eat the operator.
|
||||
|
||||
# Parse the primary expression after the binary operator.
|
||||
right = self.ParsePrimary()
|
||||
|
||||
# If binary_operator binds less tightly with right than the operator after
|
||||
# right, let the pending operator take right as its left.
|
||||
next_precedence = self.GetCurrentTokenPrecedence()
|
||||
if precedence < next_precedence:
|
||||
right = self.ParseBinOpRHS(right, precedence + 1)
|
||||
|
||||
# Merge left/right.
|
||||
left = BinaryOperatorExpressionNode(binary_operator, left, right)
|
||||
|
||||
# expression ::= primary binoprhs def ParseExpression(self): left =
|
||||
self.ParsePrimary() return self.ParseBinOpRHS(left, 0)
|
||||
|
||||
# prototype ::= id '(' id\* ')' def ParsePrototype(self): if not
|
||||
isinstance(self.current, IdentifierToken): raise RuntimeError('Expected
|
||||
function name in prototype.')
|
||||
|
||||
::
|
||||
|
||||
function_name = self.current.name
|
||||
self.Next() # eat function name.
|
||||
|
||||
if self.current != CharacterToken('('):
|
||||
raise RuntimeError('Expected "(" in prototype.')
|
||||
self.Next() # eat '('.
|
||||
|
||||
arg_names = []
|
||||
while isinstance(self.current, IdentifierToken):
|
||||
arg_names.append(self.current.name)
|
||||
self.Next()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")" in prototype.')
|
||||
|
||||
# Success.
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return PrototypeNode(function_name, arg_names)
|
||||
|
||||
# definition ::= 'def' prototype expression def ParseDefinition(self):
|
||||
self.Next() # eat def. proto = self.ParsePrototype() body =
|
||||
self.ParseExpression() return FunctionNode(proto, body)
|
||||
|
||||
# toplevelexpr ::= expression def ParseTopLevelExpr(self): proto =
|
||||
PrototypeNode('', []) return FunctionNode(proto, self.ParseExpression())
|
||||
|
||||
# external ::= 'extern' prototype def ParseExtern(self): self.Next() #
|
||||
eat extern. return self.ParsePrototype()
|
||||
|
||||
# Top-Level parsing def HandleDefinition(self):
|
||||
self.Handle(self.ParseDefinition, 'Parsed a function definition.')
|
||||
|
||||
def HandleExtern(self): self.Handle(self.ParseExtern, 'Parsed an
|
||||
extern.')
|
||||
|
||||
def HandleTopLevelExpression(self): self.Handle(self.ParseTopLevelExpr,
|
||||
'Parsed a top-level expression.')
|
||||
|
||||
def Handle(self, function, message): try: function() print message
|
||||
except Exception, e: print 'Error:', e try: self.Next() # Skip for error
|
||||
recovery. except: pass
|
||||
|
||||
Main driver code.
|
||||
-----------------
|
||||
|
||||
def main(): # Install standard binary operators. # 1 is lowest possible
|
||||
precedence. 40 is the highest. operator\_precedence = { '<': 10, '+':
|
||||
20, '-': 20, '\*': 40 }
|
||||
|
||||
# Run the main "interpreter loop". while True: print 'ready>', try: raw
|
||||
= raw\_input() except KeyboardInterrupt: return
|
||||
|
||||
::
|
||||
|
||||
parser = Parser(Tokenize(raw), operator_precedence)
|
||||
while True:
|
||||
# top ::= definition | external | expression | EOF
|
||||
if isinstance(parser.current, EOFToken):
|
||||
break
|
||||
if isinstance(parser.current, DefToken):
|
||||
parser.HandleDefinition()
|
||||
elif isinstance(parser.current, ExternToken):
|
||||
parser.HandleExtern()
|
||||
else:
|
||||
parser.HandleTopLevelExpression()
|
||||
|
||||
if **name** == '**main**\ ': main() {% endhighlight %}
|
||||
|
||||
--------------
|
||||
|
||||
**`Next: Implementing Code Generation to LLVM
|
||||
IR <PythonLangImpl3.html>`_**
|
||||
File diff suppressed because it is too large
Load diff
936
docs/source/doc/kaleidoscope/PythonLangImpl3.rst
Normal file
936
docs/source/doc/kaleidoscope/PythonLangImpl3.rst
Normal file
|
|
@ -0,0 +1,936 @@
|
|||
+------------------------------------+
|
||||
| layout: page |
|
||||
+------------------------------------+
|
||||
| title: "Kaleidoscope: Chapter 3" |
|
||||
+------------------------------------+
|
||||
|
||||
Code generation to LLVM IR
|
||||
==========================
|
||||
|
||||
Written by `Chris Lattner <mailto:sabre@nondot.org>`_ and `Max
|
||||
Shawabkeh <http://max99x.com>`_
|
||||
|
||||
**Chapter 3**
|
||||
|
||||
- This will become a table of contents (this text will be scraped).
|
||||
{:toc}
|
||||
|
||||
**`Chapter 4: Adding JIT and Optimizer
|
||||
Support <PythonLangImpl4.html>`_**
|
||||
|
||||
Introduction # {#intro}
|
||||
=======================
|
||||
|
||||
Welcome to Chapter 3 of the `Implementing a language with
|
||||
LLVM <http://www.llvm.org/docs/tutorial/index.html>`_ tutorial. This
|
||||
chapter shows you how to transform the `Abstract Syntax
|
||||
Tree <PythonLangImpl2.html>`_, built in Chapter 2, into LLVM IR. This
|
||||
will teach you a little bit about how LLVM does things, as well as
|
||||
demonstrate how easy it is to use. It's much more work to build a lexer
|
||||
and parser than it is to generate LLVM IR code. :)
|
||||
|
||||
**Please note**: the code in this chapter and later requires llvm-py 0.6
|
||||
and LLVM 2.7. Earlier versions will most likely not work with it. Also
|
||||
note that you need to use a version of this tutorial that matches your
|
||||
llvm-py release: If you are using an official llvm-py release, use the
|
||||
version of the documentation on the `llvm-py examples
|
||||
page <http://www.mdevan.org/llvm-py/examples.html>`_
|
||||
|
||||
--------------
|
||||
|
||||
Code Generation Setup # {#basics}
|
||||
=================================
|
||||
|
||||
In order to generate LLVM IR, we want some simple setup to get started.
|
||||
First we define code generation methods in each AST node class:
|
||||
|
||||
{% highlight python %} # Expression class for numeric literals like
|
||||
"1.0". class NumberExpressionNode(ExpressionNode):
|
||||
|
||||
def **init**\ (self, value): self.value = value
|
||||
|
||||
def CodeGen(self): ...
|
||||
|
||||
Expression class for referencing a variable, like "a".
|
||||
======================================================
|
||||
|
||||
class VariableExpressionNode(ExpressionNode):
|
||||
|
||||
def **init**\ (self, name): self.name = name
|
||||
|
||||
def CodeGen(self): ...
|
||||
|
||||
... {% endhighlight %}
|
||||
|
||||
The ``CodeGen`` method says to emit IR for that AST node along with all
|
||||
the things it depends on, and they all return an LLVM Value object.
|
||||
"Value" is the class used to represent a "`Static Single Assignment
|
||||
(SSA) <http://en.wikipedia.org/wiki/Static_single_assignment_form>`_
|
||||
register" or "SSA value" in LLVM. The most distinct aspect of SSA values
|
||||
is that their value is computed as the related instruction executes, and
|
||||
it does not get a new value until (and if) the instruction re-executes.
|
||||
In other words, there is no way to "change" an SSA value. For more
|
||||
information, please read up on `Static Single
|
||||
Assignment <http://en.wikipedia.org/wiki/Static_single_assignment_form>`_
|
||||
- the concepts are really quite natural once you grok them.
|
||||
|
||||
We will also need to define some global variables which we will be used
|
||||
during code generation:
|
||||
|
||||
{% highlight python %} # The LLVM module, which holds all the IR code.
|
||||
g\_llvm\_module = Module.new('my cool jit')
|
||||
|
||||
The LLVM instruction builder. Created whenever a new function is entered.
|
||||
=========================================================================
|
||||
|
||||
g\_llvm\_builder = None
|
||||
|
||||
A dictionary that keeps track of which values are defined in the current scope
|
||||
==============================================================================
|
||||
|
||||
and what their LLVM representation is.
|
||||
======================================
|
||||
|
||||
g\_named\_values = {} {% endhighlight %}
|
||||
|
||||
``g_llvm_module`` is the LLVM construct that contains all of the
|
||||
functions and global variables in a chunk of code. In many ways, it is
|
||||
the top-level structure that the LLVM IR uses to contain code.
|
||||
|
||||
``g_llvm_builder`` is a helper object that makes it easy to generate
|
||||
LLVM instructions. Instances of the
|
||||
`llvm.core.Builder <llvm.core.Builder.html>`_ class keep track of the
|
||||
current place to insert instructions and have methods to create new
|
||||
instructions. Note that we do not initialize this variable; instead, it
|
||||
will be initialized whenever we start generating code for a function.
|
||||
|
||||
Finally, ``g_named_values`` is a dictionary that keeps track of which
|
||||
values are defined in the current scope and what their LLVM
|
||||
representation is. In other words, it is a symbol table for the code. In
|
||||
this form of Kaleidoscope, the only things that can be referenced are
|
||||
function parameters. As such, function parameters will be in this map
|
||||
when generating code for their function body.
|
||||
|
||||
With these basics in place, we can start talking about how to generate
|
||||
code for each expression. Note that this assumes that ``g_llvm_builder``
|
||||
has been set up to generate code *into* something. For now, we'll assume
|
||||
that this has already been done, and we'll just use it to emit code.
|
||||
|
||||
--------------
|
||||
|
||||
Expression Code Generation # {#exprs}
|
||||
=====================================
|
||||
|
||||
Generating LLVM code for expression nodes is very straightforward: less
|
||||
than 35 lines of commented code for all four of our expression nodes.
|
||||
First we'll do numeric literals:
|
||||
|
||||
{% highlight python %} def CodeGen(self): return
|
||||
Constant.real(Type.double(), self.value) {% endhighlight %}
|
||||
|
||||
In llvm-py, floating point numeric constants are represented with the
|
||||
``llvm.core.ConstantFP`` class. To create one, we can use the static
|
||||
``real()`` method in the ``llvm.core.Constant`` class. This code
|
||||
basically just creates and returns a ``ConstantFP``. Note that in the
|
||||
LLVM IR constants are all uniqued together and shared. For this reason,
|
||||
we create the constant through a factory method instead of instantiating
|
||||
one directly.
|
||||
|
||||
{% highlight python %} def CodeGen(self): if self.name in
|
||||
g\_named\_values: return g\_named\_values[self.name] else: raise
|
||||
RuntimeError('Unknown variable name: ' + self.name) {% endhighlight %}
|
||||
|
||||
References to variables are also quite simple using LLVM. In the simple
|
||||
version of Kaleidoscope, we assume that the variable has already been
|
||||
emitted somewhere and its value is available. In practice, the only
|
||||
values that can be in the ``g_named_values`` dictionary are function
|
||||
arguments. This code simply checks to see that the specified name is in
|
||||
the map (if not, an unknown variable is being referenced) and returns
|
||||
the value for it. In future chapters, we'll add support for `loop
|
||||
induction variables <PythonLangImpl5.html#for>`_ in the symbol table,
|
||||
and for `local variables <PythonLangImpl7.html#localvars>`_.
|
||||
|
||||
{% highlight python %} def CodeGen(self): left = self.left.CodeGen()
|
||||
right = self.right.CodeGen()
|
||||
|
||||
::
|
||||
|
||||
if self.operator == '+':
|
||||
return g_llvm_builder.fadd(left, right, 'addtmp')
|
||||
elif self.operator == '-':
|
||||
return g_llvm_builder.fsub(left, right, 'subtmp')
|
||||
elif self.operator == '*':
|
||||
return g_llvm_builder.fmul(left, right, 'multmp')
|
||||
elif self.operator == '<':
|
||||
result = g_llvm_builder.fcmp(FCMP_ULT, left, right, 'cmptmp')
|
||||
# Convert bool 0 or 1 to double 0.0 or 1.0.
|
||||
return g_llvm_builder.uitofp(result, Type.double(), 'booltmp')
|
||||
else:
|
||||
raise RuntimeError('Unknown binary operator.')
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
Binary operators start to get more interesting. The basic idea here is
|
||||
that we recursively emit code for the left-hand side of the expression,
|
||||
then the right-hand side, then we compute the result of the binary
|
||||
expression depending on which operator is being used.
|
||||
|
||||
In the example above, the LLVM builder class is starting to show its
|
||||
value. ``g_llvm_builder`` knows where to insert the newly created
|
||||
instruction, all you have to do is specify what instruction to create
|
||||
(e.g. with ``add``), which operands to use (``left`` and ``right`` here)
|
||||
and optionally provide a name for the generated instruction.
|
||||
|
||||
One nice thing about LLVM is that the name is just a hint. For instance,
|
||||
if the code above emits multiple "addtmp" variables, LLVM will
|
||||
automatically provide each one with an increasing, unique numeric
|
||||
suffix. Local value names for instructions are purely optional, but it
|
||||
makes it much easier to read the IR dumps.
|
||||
|
||||
`LLVM instructions <http://www.llvm.org/docs/LangRef.html#instref>`_ are
|
||||
constrained by strict rules: for example, the Left and Right operators
|
||||
of an `add instruction <http://www.llvm.org/docs/LangRef.html#i_add>`_
|
||||
must have the same type, and the result type of the add must match the
|
||||
operand types. Because all values in Kaleidoscope are doubles, this
|
||||
makes for very simple code for add, sub and mul.
|
||||
|
||||
On the other hand, LLVM specifies that the `fcmp
|
||||
instruction <http://www.llvm.org/docs/LangRef.html#i_fcmp>`_ always
|
||||
returns an 'i1' value (a one bit integer). The problem with this is that
|
||||
Kaleidoscope wants the value to be a 0.0 or 1.0 value. In order to get
|
||||
these semantics, we combine the fcmp instruction with a `uitofp
|
||||
instruction <http://www.llvm.org/docs/LangRef.html#i_uitofp>`_. This
|
||||
instruction converts its input integer into a floating point value by
|
||||
treating the input as an unsigned value. In contrast, if we used the
|
||||
`sitofp instruction <http://www.llvm.org/docs/LangRef.html#i_sitofp>`_,
|
||||
the Kaleidoscope ``<`` operator would return 0.0 and -1.0, depending on
|
||||
the input value.
|
||||
|
||||
{% highlight python %} def CodeGen(self): # Look up the name in the
|
||||
global module table. callee =
|
||||
g\_llvm\_module.get\_function\_named(self.callee)
|
||||
|
||||
::
|
||||
|
||||
# Check for argument mismatch error.
|
||||
if len(callee.args) != len(self.args):
|
||||
raise RuntimeError('Incorrect number of arguments passed.')
|
||||
|
||||
arg_values = [i.CodeGen() for i in self.args]
|
||||
|
||||
return g_llvm_builder.call(callee, arg_values, 'calltmp')
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
Code generation for function calls is quite straightforward with LLVM.
|
||||
The code above initially does a function name lookup in the LLVM
|
||||
Module's symbol table. Recall that the LLVM Module is the container that
|
||||
holds all of the functions we are JIT'ing. By giving each function the
|
||||
same name as what the user specifies, we can use the LLVM symbol table
|
||||
to resolve function names for us.
|
||||
|
||||
Once we have the function to call, we codegen each argument that is to
|
||||
be passed in, and create an LLVM `call
|
||||
instruction <http://www.llvm.org/docs/LangRef.html#i_call>`_. Note that
|
||||
LLVM uses the native C calling conventions by default, allowing these
|
||||
calls to also call into standard library functions like "sin" and "cos",
|
||||
with no additional effort.
|
||||
|
||||
This wraps up our handling of the four basic expressions that we have so
|
||||
far in Kaleidoscope. Feel free to go in and add some more. For example,
|
||||
by browsing the `LLVM language
|
||||
reference <http://www.llvm.org/docs/LangRef.html>`_ you'll find several
|
||||
other interesting instructions that are really easy to plug into our
|
||||
basic framework.
|
||||
|
||||
--------------
|
||||
|
||||
Function Code Generation # {#funcs}
|
||||
===================================
|
||||
|
||||
Code generation for prototypes and functions must handle a number of
|
||||
details, which make their code less beautiful than expression code
|
||||
generation, but allows us to illustrate some important points. First,
|
||||
let's talk about code generation for prototypes: they are used both for
|
||||
function bodies and external function declarations. The code starts
|
||||
with:
|
||||
|
||||
{% highlight python %} def CodeGen(self): # Make the function type, eg.
|
||||
double(double,double). funct\_type = Type.function( Type.double(),
|
||||
[Type.double()] \* len(self.args), False)
|
||||
|
||||
::
|
||||
|
||||
function = Function.new(g_llvm_module, funct_type, self.name)
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
The call to ``Type.function`` creates the ``FunctionType`` that should
|
||||
be used for a given Prototype. Since all function arguments in
|
||||
Kaleidoscope are of type double, the first line creates a list of "N"
|
||||
LLVM double types. It then uses the ``Type.function`` method to create a
|
||||
function type that takes "N" doubles as arguments, returns one double as
|
||||
a result, and that is not vararg (the False parameter indicates this).
|
||||
Note that Types in LLVM are uniqued just like Constants are, so you
|
||||
don't instantiate them directly.
|
||||
|
||||
The final line above actually creates the function that the prototype
|
||||
will correspond to. This indicates the type and name to use, as well as
|
||||
which module to insert into. Note that by default, the function will
|
||||
have `external
|
||||
linkage <<http://www.llvm.org/docs/LangRef.html#linkage>`_, which means
|
||||
that the function may be defined outside the current module and/or that
|
||||
it is callable by functions outside the module. The name passed in is
|
||||
the name the user specified: since ``g_llvm_module`` is specified, this
|
||||
name is registered in ``g_llvm_module``'s symbol table, which is used by
|
||||
the function call code above.
|
||||
|
||||
{% highlight python %} # If the name conflicted, there was already
|
||||
something with the same name. # If it has a body, don't allow
|
||||
redefinition or reextern. if function.name != self.name:
|
||||
function.delete() function =
|
||||
g\_llvm\_module.get\_function\_named(self.name) {% endhighlight %}
|
||||
|
||||
The Module symbol table works just like the Function symbol table when
|
||||
it comes to name conflicts: if a new function is created with a name was
|
||||
previously added to the symbol table, it will get implicitly renamed
|
||||
when added to the Module. The code above exploits this fact to determine
|
||||
if there was a previous definition of this function.
|
||||
|
||||
In Kaleidoscope, we choose to allow redefinitions of functions in two
|
||||
cases: first, we want to allow 'extern'ing a function more than once, as
|
||||
long as the prototypes for the externs match (since all arguments have
|
||||
the same type, we just have to check that the number of arguments
|
||||
match). Second, we want to allow 'extern'ing a function and then
|
||||
defining a body for it. This is useful when defining mutually recursive
|
||||
functions.
|
||||
|
||||
In order to implement this, the code above first checks to see if there
|
||||
is a collision on the name of the function. If so, it deletes the
|
||||
function we just created (by calling ``delete``) and then calling
|
||||
``get_function_named`` to get the existing function with the specified
|
||||
name.
|
||||
|
||||
{% highlight python %} # If the function already has a body, reject
|
||||
this. if not function.is\_declaration: raise RuntimeError('Redefinition
|
||||
of function.')
|
||||
|
||||
::
|
||||
|
||||
# If F took a different number of args, reject.
|
||||
if len(callee.args) != len(self.args):
|
||||
raise RuntimeError('Redeclaration of a function with different number '
|
||||
'of args.')
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
In order to verify the logic above, we first check to see if the
|
||||
pre-existing function is a forward declaration. Since we don't allow
|
||||
anything after a full definition of the function, the code rejects this
|
||||
case. If the previous reference to a function was an 'extern', we simply
|
||||
verify that the number of arguments for that definition and this one
|
||||
match up. If not, we emit an error.
|
||||
|
||||
{% highlight python %} # Set names for all arguments and add them to the
|
||||
variables symbol table. for arg, arg\_name in zip(function.args,
|
||||
self.args): arg.name = arg\_name # Add arguments to variable symbol
|
||||
table. g\_named\_values[arg\_name] = arg
|
||||
|
||||
::
|
||||
|
||||
return function
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
The last bit of code for prototypes loops over all of the arguments in
|
||||
the function, setting the name of the LLVM Argument objects to match,
|
||||
and registering the arguments in the ``g_named_values`` map for future
|
||||
use by the ``VariableExpressionNode``. Note that we don't check for
|
||||
conflicting argument names here (e.g. "extern foo(a b a)"). Doing so
|
||||
would be very straight-forward with the mechanics we have already used
|
||||
above. Once this is all set up, it returns the Function object to the
|
||||
caller.
|
||||
|
||||
{% highlight python %} def CodeGen(self): # Clear scope.
|
||||
g\_named\_values.clear()
|
||||
|
||||
::
|
||||
|
||||
# Create a function object.
|
||||
function = self.prototype.CodeGen()
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
Code generation for function definitions starts out simply enough: we
|
||||
just clear out the ``g_named_values`` dictionary to make sure that there
|
||||
isn't anything in it from the last function we compiled and codegen the
|
||||
prototype. Code generation of the prototype ensures that there is an
|
||||
LLVM Function object that is ready to go for us.
|
||||
|
||||
{% highlight python %} # Create a new basic block to start insertion
|
||||
into. block = function.append\_basic\_block('entry') global
|
||||
g\_llvm\_builder g\_llvm\_builder = Builder.new(block) {% endhighlight
|
||||
%}
|
||||
|
||||
Now we get to the point where ``g_llvm_builder`` is set up. The first
|
||||
line creates a new `basic
|
||||
block <http://en.wikipedia.org/wiki/Basic_block>`_ (named "entry"),
|
||||
which is inserted into the function. The second line declares that the
|
||||
global ``g_llvm_builder`` object is to be changed. The last line creates
|
||||
a new builder that is set up to insert new instructions into the basic
|
||||
block we just created. Basic blocks in LLVM are an important part of
|
||||
functions that define the `Control Flow
|
||||
Graph <http://en.wikipedia.org/wiki/Control_flow_graph>`_. Since we
|
||||
don't have any control flow, our functions will only contain one block
|
||||
at this point. We'll fix this in `Chapter 5 <PythonLangImpl5.html>`_ :).
|
||||
|
||||
{% highlight python %} # Finish off the function. try: return\_value =
|
||||
self.body.CodeGen() g\_llvm\_builder.ret(return\_value)
|
||||
|
||||
::
|
||||
|
||||
# Validate the generated code, checking for consistency.
|
||||
function.verify()
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
Once the insertion point is set up, we call the ``CodeGen`` method for
|
||||
the root expression of the function. If no error happens, this emits
|
||||
code to compute the expression into the entry block and returns the
|
||||
value that was computed. Assuming no error, we then create an LLVM `ret
|
||||
instruction <http://www.llvm.org/docs/LangRef.html#i_ret>`_, which
|
||||
completes the function. Once the function is built, we call ``verify``,
|
||||
which is provided by LLVM. This function does a variety of consistency
|
||||
checks on the generated code, to determine if our compiler is doing
|
||||
everything right. Using this is important: it can catch a lot of bugs.
|
||||
Once the function is finished and validated, we return it.
|
||||
|
||||
{% highlight python %} except: function.delete() raise
|
||||
|
||||
::
|
||||
|
||||
return function
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
The only piece left here is handling of the error case. For simplicity,
|
||||
we handle this by merely deleting the function we produced with the
|
||||
``delete`` method. This allows the user to redefine a function that they
|
||||
incorrectly typed in before: if we didn't delete it, it would live in
|
||||
the symbol table, with a body, preventing future redefinition.
|
||||
|
||||
This code does have a bug, though. Since the ``PrototypeNode::CodeGen``
|
||||
can return a previously defined forward declaration, our code can
|
||||
actually delete a forward declaration. There are a number of ways to fix
|
||||
this bug; see what you can come up with! Here is a testcase:
|
||||
|
||||
{% highlight python %} extern foo(a b) # ok, defines foo. def foo(a b) c
|
||||
# error, 'c' is invalid. def bar() foo(1, 2) # error, unknown function
|
||||
"foo" {% endhighlight %}
|
||||
|
||||
--------------
|
||||
|
||||
Driver Changes and Closing Thoughts # {#driver}
|
||||
===============================================
|
||||
|
||||
For now, code generation to LLVM doesn't really get us much, except that
|
||||
we can look at the pretty IR calls. The sample code inserts calls to
|
||||
CodeGen into the ``Handle*`` functions, and then dumps out the LLVM IR.
|
||||
This gives a nice way to look at the LLVM IR for simple functions. For
|
||||
example:
|
||||
|
||||
{% highlight bash %} ready> 4+5 Read a top-level expression: define
|
||||
double @0() { entry: ret double 9.000000e+00 } {% endhighlight %}
|
||||
|
||||
Note how the parser turns the top-level expression into anonymous
|
||||
functions for us. This will be handy when we add JIT support in the next
|
||||
chapter. Also note that the code is very literally transcribed, no
|
||||
optimizations are being performed except simple constant folding done by
|
||||
the Builder. We will add optimizations explicitly in the next chapter.
|
||||
|
||||
{% highlight bash %} ready> def foo(a b) a\ *a + 2*\ a\ *b + b*\ b Read
|
||||
a function definition: define double @foo(double %a, double %b) { entry:
|
||||
%multmp = fmul double %a, %a ; [#uses=1] %multmp1 = fmul double
|
||||
2.000000e+00, %a ; [#uses=1] %multmp2 = fmul double %multmp1, %b ;
|
||||
[#uses=1] %addtmp = fadd double %multmp, %multmp2 ; [#uses=1] %multmp3 =
|
||||
fmul double %b, %b ; [#uses=1] %addtmp4 = fadd double %addtmp, %multmp3
|
||||
; [#uses=1] ret double %addtmp4 } {% endhighlight %}
|
||||
|
||||
This shows some simple arithmetic. Notice the striking similarity to the
|
||||
LLVM builder calls that we use to create the instructions.
|
||||
|
||||
{% highlight bash %} ready> def bar(a) foo(a, 4.0) + bar(31337) Read a
|
||||
function definition: define double @bar(double %a) { entry: %calltmp =
|
||||
call double @foo(double %a, double 4.000000e+00) ; [#uses=1] %calltmp1 =
|
||||
call double @bar(double 3.133700e+04) ; [#uses=1] %addtmp = fadd double
|
||||
%calltmp, %calltmp1 ; [#uses=1] ret double %addtmp } {% endhighlight %}
|
||||
|
||||
This shows some function calls. Note that this function will take a long
|
||||
time to execute if you call it. In the future we'll add conditional
|
||||
control flow to actually make recursion useful :).
|
||||
|
||||
{% highlight bash %} ready> extern cos(x) Read extern: declare double
|
||||
@cos(double)
|
||||
|
||||
ready> cos(1.234) Read a top-level expression: define double @1() {
|
||||
entry: %calltmp = call double @cos(double 1.234000e+00) ; [#uses=1] ret
|
||||
double %calltmp } {% endhighlight %}
|
||||
|
||||
This shows an extern for the libm "cos" function, and a call to it.
|
||||
|
||||
{% highlight bash %} ready> ^C ; ModuleID = 'my cool jit'
|
||||
|
||||
define double @0() { entry: ret double 9.000000e+00 }
|
||||
|
||||
define double @foo(double %a, double %b) { entry: %multmp = fmul double
|
||||
%a, %a ; [#uses=1] %multmp1 = fmul double 2.000000e+00, %a ; [#uses=1]
|
||||
%multmp2 = fmul double %multmp1, %b ; [#uses=1] %addtmp = fadd double
|
||||
%multmp, %multmp2 ; [#uses=1] %multmp3 = fmul double %b, %b ; [#uses=1]
|
||||
%addtmp4 = fadd double %addtmp, %multmp3 ; [#uses=1] ret double %addtmp4
|
||||
}
|
||||
|
||||
define double @bar(double %a) { entry: %calltmp = call double
|
||||
@foo(double %a, double 4.000000e+00) ; [#uses=1] %calltmp1 = call double
|
||||
@bar(double 3.133700e+04) ; [#uses=1] %addtmp = fadd double %calltmp,
|
||||
%calltmp1 ; [#uses=1] ret double %addtmp }
|
||||
|
||||
declare double @cos(double)
|
||||
|
||||
define double @1() { entry: %calltmp = call double @cos(double
|
||||
1.234000e+00) ; [#uses=1] ret double %calltmp } {% endhighlight %}
|
||||
|
||||
When you quit the current demo, it dumps out the IR for the entire
|
||||
module generated. Here you can see the big picture with all the
|
||||
functions referencing each other.
|
||||
|
||||
This wraps up the third chapter of the Kaleidoscope tutorial. Up next,
|
||||
we'll describe how to `add JIT codegen and optimizer
|
||||
support <PythonLangImpl4.html>`_ to this so we can actually start
|
||||
running code!
|
||||
|
||||
--------------
|
||||
|
||||
Full Code Listing # {#code}
|
||||
===========================
|
||||
|
||||
Here is the complete code listing for our running example, enhanced with
|
||||
the LLVM code generator. Because this uses the llvm-py libraries, you
|
||||
need to `download <../download.html>`_ and
|
||||
`install <../userguide.html#install>`_ them.
|
||||
|
||||
{% highlight python %} #!/usr/bin/env python
|
||||
|
||||
import re from llvm.core import Module, Constant, Type, Function,
|
||||
Builder, FCMP\_ULT
|
||||
|
||||
Globals
|
||||
-------
|
||||
|
||||
The LLVM module, which holds all the IR code.
|
||||
=============================================
|
||||
|
||||
g\_llvm\_module = Module.new('my cool jit')
|
||||
|
||||
The LLVM instruction builder. Created whenever a new function is entered.
|
||||
=========================================================================
|
||||
|
||||
g\_llvm\_builder = None
|
||||
|
||||
A dictionary that keeps track of which values are defined in the current scope
|
||||
==============================================================================
|
||||
|
||||
and what their LLVM representation is.
|
||||
======================================
|
||||
|
||||
g\_named\_values = {}
|
||||
|
||||
Lexer
|
||||
-----
|
||||
|
||||
The lexer yields one of these types for each token.
|
||||
===================================================
|
||||
|
||||
class EOFToken(object): pass
|
||||
|
||||
class DefToken(object): pass
|
||||
|
||||
class ExternToken(object): pass
|
||||
|
||||
class IdentifierToken(object): def **init**\ (self, name): self.name =
|
||||
name
|
||||
|
||||
class NumberToken(object): def **init**\ (self, value): self.value =
|
||||
value
|
||||
|
||||
class CharacterToken(object): def **init**\ (self, char): self.char =
|
||||
char def **eq**\ (self, other): return isinstance(other, CharacterToken)
|
||||
and self.char == other.char def **ne**\ (self, other): return not self
|
||||
== other
|
||||
|
||||
Regular expressions that tokens and comments of our language.
|
||||
=============================================================
|
||||
|
||||
REGEX\_NUMBER = re.compile('[0-9]+(?:.[0-9]+)?') REGEX\_IDENTIFIER =
|
||||
re.compile('[a-zA-Z][a-zA-Z0-9]\ *') REGEX\_COMMENT = re.compile('#.*')
|
||||
|
||||
def Tokenize(string): while string: # Skip whitespace. if
|
||||
string[0].isspace(): string = string[1:] continue
|
||||
|
||||
::
|
||||
|
||||
# Run regexes.
|
||||
comment_match = REGEX_COMMENT.match(string)
|
||||
number_match = REGEX_NUMBER.match(string)
|
||||
identifier_match = REGEX_IDENTIFIER.match(string)
|
||||
|
||||
# Check if any of the regexes matched and yield the appropriate result.
|
||||
if comment_match:
|
||||
comment = comment_match.group(0)
|
||||
string = string[len(comment):]
|
||||
elif number_match:
|
||||
number = number_match.group(0)
|
||||
yield NumberToken(float(number))
|
||||
string = string[len(number):]
|
||||
elif identifier_match:
|
||||
identifier = identifier_match.group(0)
|
||||
# Check if we matched a keyword.
|
||||
if identifier == 'def':
|
||||
yield DefToken()
|
||||
elif identifier == 'extern':
|
||||
yield ExternToken()
|
||||
else:
|
||||
yield IdentifierToken(identifier)
|
||||
string = string[len(identifier):]
|
||||
else:
|
||||
# Yield the ASCII value of the unknown character.
|
||||
yield CharacterToken(string[0])
|
||||
string = string[1:]
|
||||
|
||||
yield EOFToken()
|
||||
|
||||
Abstract Syntax Tree (aka Parse Tree)
|
||||
-------------------------------------
|
||||
|
||||
Base class for all expression nodes.
|
||||
====================================
|
||||
|
||||
class ExpressionNode(object): pass
|
||||
|
||||
Expression class for numeric literals like "1.0".
|
||||
=================================================
|
||||
|
||||
class NumberExpressionNode(ExpressionNode):
|
||||
|
||||
def **init**\ (self, value): self.value = value
|
||||
|
||||
def CodeGen(self): return Constant.real(Type.double(), self.value)
|
||||
|
||||
Expression class for referencing a variable, like "a".
|
||||
======================================================
|
||||
|
||||
class VariableExpressionNode(ExpressionNode):
|
||||
|
||||
def **init**\ (self, name): self.name = name
|
||||
|
||||
def CodeGen(self): if self.name in g\_named\_values: return
|
||||
g\_named\_values[self.name] else: raise RuntimeError('Unknown variable
|
||||
name: ' + self.name)
|
||||
|
||||
Expression class for a binary operator.
|
||||
=======================================
|
||||
|
||||
class BinaryOperatorExpressionNode(ExpressionNode):
|
||||
|
||||
def **init**\ (self, operator, left, right): self.operator = operator
|
||||
self.left = left self.right = right
|
||||
|
||||
def CodeGen(self): left = self.left.CodeGen() right =
|
||||
self.right.CodeGen()
|
||||
|
||||
::
|
||||
|
||||
if self.operator == '+':
|
||||
return g_llvm_builder.fadd(left, right, 'addtmp')
|
||||
elif self.operator == '-':
|
||||
return g_llvm_builder.fsub(left, right, 'subtmp')
|
||||
elif self.operator == '*':
|
||||
return g_llvm_builder.fmul(left, right, 'multmp')
|
||||
elif self.operator == '<':
|
||||
result = g_llvm_builder.fcmp(FCMP_ULT, left, right, 'cmptmp')
|
||||
# Convert bool 0 or 1 to double 0.0 or 1.0.
|
||||
return g_llvm_builder.uitofp(result, Type.double(), 'booltmp')
|
||||
else:
|
||||
raise RuntimeError('Unknown binary operator.')
|
||||
|
||||
Expression class for function calls.
|
||||
====================================
|
||||
|
||||
class CallExpressionNode(ExpressionNode):
|
||||
|
||||
def **init**\ (self, callee, args): self.callee = callee self.args =
|
||||
args
|
||||
|
||||
def CodeGen(self): # Look up the name in the global module table. callee
|
||||
= g\_llvm\_module.get\_function\_named(self.callee)
|
||||
|
||||
::
|
||||
|
||||
# Check for argument mismatch error.
|
||||
if len(callee.args) != len(self.args):
|
||||
raise RuntimeError('Incorrect number of arguments passed.')
|
||||
|
||||
arg_values = [i.CodeGen() for i in self.args]
|
||||
|
||||
return g_llvm_builder.call(callee, arg_values, 'calltmp')
|
||||
|
||||
This class represents the "prototype" for a function, which captures its name,
|
||||
==============================================================================
|
||||
|
||||
and its argument names (thus implicitly the number of arguments the function
|
||||
============================================================================
|
||||
|
||||
takes).
|
||||
=======
|
||||
|
||||
class PrototypeNode(object):
|
||||
|
||||
def **init**\ (self, name, args): self.name = name self.args = args
|
||||
|
||||
def CodeGen(self): # Make the function type, eg. double(double,double).
|
||||
funct\_type = Type.function( Type.double(), [Type.double()] \*
|
||||
len(self.args), False)
|
||||
|
||||
::
|
||||
|
||||
function = Function.new(g_llvm_module, funct_type, self.name)
|
||||
|
||||
# If the name conflicted, there was already something with the same name.
|
||||
# If it has a body, don't allow redefinition or reextern.
|
||||
if function.name != self.name:
|
||||
function.delete()
|
||||
function = g_llvm_module.get_function_named(self.name)
|
||||
|
||||
# If the function already has a body, reject this.
|
||||
if not function.is_declaration:
|
||||
raise RuntimeError('Redefinition of function.')
|
||||
|
||||
# If F took a different number of args, reject.
|
||||
if len(callee.args) != len(self.args):
|
||||
raise RuntimeError('Redeclaration of a function with different number '
|
||||
'of args.')
|
||||
|
||||
# Set names for all arguments and add them to the variables symbol table.
|
||||
for arg, arg_name in zip(function.args, self.args):
|
||||
arg.name = arg_name
|
||||
# Add arguments to variable symbol table.
|
||||
g_named_values[arg_name] = arg
|
||||
|
||||
return function
|
||||
|
||||
This class represents a function definition itself.
|
||||
===================================================
|
||||
|
||||
class FunctionNode(object):
|
||||
|
||||
def **init**\ (self, prototype, body): self.prototype = prototype
|
||||
self.body = body
|
||||
|
||||
def CodeGen(self): # Clear scope. g\_named\_values.clear()
|
||||
|
||||
::
|
||||
|
||||
# Create a function object.
|
||||
function = self.prototype.CodeGen()
|
||||
|
||||
# Create a new basic block to start insertion into.
|
||||
block = function.append_basic_block('entry')
|
||||
global g_llvm_builder
|
||||
g_llvm_builder = Builder.new(block)
|
||||
|
||||
# Finish off the function.
|
||||
try:
|
||||
return_value = self.body.CodeGen()
|
||||
g_llvm_builder.ret(return_value)
|
||||
|
||||
# Validate the generated code, checking for consistency.
|
||||
function.verify()
|
||||
except:
|
||||
function.delete()
|
||||
raise
|
||||
|
||||
return function
|
||||
|
||||
Parser
|
||||
------
|
||||
|
||||
class Parser(object):
|
||||
|
||||
def **init**\ (self, tokens, binop\_precedence): self.tokens = tokens
|
||||
self.binop\_precedence = binop\_precedence self.Next()
|
||||
|
||||
# Provide a simple token buffer. Parser.current is the current token the
|
||||
# parser is looking at. Parser.Next() reads another token from the lexer
|
||||
and # updates Parser.current with its results. def Next(self):
|
||||
self.current = self.tokens.next()
|
||||
|
||||
# Gets the precedence of the current token, or -1 if the token is not a
|
||||
binary # operator. def GetCurrentTokenPrecedence(self): if
|
||||
isinstance(self.current, CharacterToken): return
|
||||
self.binop\_precedence.get(self.current.char, -1) else: return -1
|
||||
|
||||
# identifierexpr ::= identifier \| identifier '(' expression\* ')' def
|
||||
ParseIdentifierExpr(self): identifier\_name = self.current.name
|
||||
self.Next() # eat identifier.
|
||||
|
||||
::
|
||||
|
||||
if self.current != CharacterToken('('): # Simple variable reference.
|
||||
return VariableExpressionNode(identifier_name)
|
||||
|
||||
# Call.
|
||||
self.Next() # eat '('.
|
||||
args = []
|
||||
if self.current != CharacterToken(')'):
|
||||
while True:
|
||||
args.append(self.ParseExpression())
|
||||
if self.current == CharacterToken(')'):
|
||||
break
|
||||
elif self.current != CharacterToken(','):
|
||||
raise RuntimeError('Expected ")" or "," in argument list.')
|
||||
self.Next()
|
||||
|
||||
self.Next() # eat ')'.
|
||||
return CallExpressionNode(identifier_name, args)
|
||||
|
||||
# numberexpr ::= number def ParseNumberExpr(self): result =
|
||||
NumberExpressionNode(self.current.value) self.Next() # consume the
|
||||
number. return result
|
||||
|
||||
# parenexpr ::= '(' expression ')' def ParseParenExpr(self): self.Next()
|
||||
# eat '('.
|
||||
|
||||
::
|
||||
|
||||
contents = self.ParseExpression()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")".')
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return contents
|
||||
|
||||
# primary ::= identifierexpr \| numberexpr \| parenexpr def
|
||||
ParsePrimary(self): if isinstance(self.current, IdentifierToken): return
|
||||
self.ParseIdentifierExpr() elif isinstance(self.current, NumberToken):
|
||||
return self.ParseNumberExpr() elif self.current == CharacterToken('('):
|
||||
return self.ParseParenExpr() else: raise RuntimeError('Unknown token
|
||||
when expecting an expression.')
|
||||
|
||||
# binoprhs ::= (operator primary)\* def ParseBinOpRHS(self, left,
|
||||
left\_precedence): # If this is a binary operator, find its precedence.
|
||||
while True: precedence = self.GetCurrentTokenPrecedence()
|
||||
|
||||
::
|
||||
|
||||
# If this is a binary operator that binds at least as tightly as the
|
||||
# current one, consume it; otherwise we are done.
|
||||
if precedence < left_precedence:
|
||||
return left
|
||||
|
||||
binary_operator = self.current.char
|
||||
self.Next() # eat the operator.
|
||||
|
||||
# Parse the primary expression after the binary operator.
|
||||
right = self.ParsePrimary()
|
||||
|
||||
# If binary_operator binds less tightly with right than the operator after
|
||||
# right, let the pending operator take right as its left.
|
||||
next_precedence = self.GetCurrentTokenPrecedence()
|
||||
if precedence < next_precedence:
|
||||
right = self.ParseBinOpRHS(right, precedence + 1)
|
||||
|
||||
# Merge left/right.
|
||||
left = BinaryOperatorExpressionNode(binary_operator, left, right)
|
||||
|
||||
# expression ::= primary binoprhs def ParseExpression(self): left =
|
||||
self.ParsePrimary() return self.ParseBinOpRHS(left, 0)
|
||||
|
||||
# prototype ::= id '(' id\* ')' def ParsePrototype(self): if not
|
||||
isinstance(self.current, IdentifierToken): raise RuntimeError('Expected
|
||||
function name in prototype.')
|
||||
|
||||
::
|
||||
|
||||
function_name = self.current.name
|
||||
self.Next() # eat function name.
|
||||
|
||||
if self.current != CharacterToken('('):
|
||||
raise RuntimeError('Expected "(" in prototype.')
|
||||
self.Next() # eat '('.
|
||||
|
||||
arg_names = []
|
||||
while isinstance(self.current, IdentifierToken):
|
||||
arg_names.append(self.current.name)
|
||||
self.Next()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")" in prototype.')
|
||||
|
||||
# Success.
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return PrototypeNode(function_name, arg_names)
|
||||
|
||||
# definition ::= 'def' prototype expression def ParseDefinition(self):
|
||||
self.Next() # eat def. proto = self.ParsePrototype() body =
|
||||
self.ParseExpression() return FunctionNode(proto, body)
|
||||
|
||||
# toplevelexpr ::= expression def ParseTopLevelExpr(self): proto =
|
||||
PrototypeNode('', []) return FunctionNode(proto, self.ParseExpression())
|
||||
|
||||
# external ::= 'extern' prototype def ParseExtern(self): self.Next() #
|
||||
eat extern. return self.ParsePrototype()
|
||||
|
||||
# Top-Level parsing def HandleDefinition(self):
|
||||
self.Handle(self.ParseDefinition, 'Read a function definition:')
|
||||
|
||||
def HandleExtern(self): self.Handle(self.ParseExtern, 'Read an extern:')
|
||||
|
||||
def HandleTopLevelExpression(self): self.Handle(self.ParseTopLevelExpr,
|
||||
'Read a top-level expression:')
|
||||
|
||||
def Handle(self, function, message): try: print message,
|
||||
function().CodeGen() except Exception, e: print 'Error:', e try:
|
||||
self.Next() # Skip for error recovery. except: pass
|
||||
|
||||
Main driver code.
|
||||
-----------------
|
||||
|
||||
def main(): # Install standard binary operators. # 1 is lowest possible
|
||||
precedence. 40 is the highest. operator\_precedence = { '<': 10, '+':
|
||||
20, '-': 20, '\*': 40 }
|
||||
|
||||
# Run the main "interpreter loop". while True: print 'ready>', try: raw
|
||||
= raw\_input() except KeyboardInterrupt: break
|
||||
|
||||
::
|
||||
|
||||
parser = Parser(Tokenize(raw), operator_precedence)
|
||||
while True:
|
||||
# top ::= definition | external | expression | EOF
|
||||
if isinstance(parser.current, EOFToken):
|
||||
break
|
||||
if isinstance(parser.current, DefToken):
|
||||
parser.HandleDefinition()
|
||||
elif isinstance(parser.current, ExternToken):
|
||||
parser.HandleExtern()
|
||||
else:
|
||||
parser.HandleTopLevelExpression()
|
||||
|
||||
# Print out all of the generated code. print '', g\_llvm\_module
|
||||
|
||||
if **name** == '**main**\ ': main() {% endhighlight %}
|
||||
|
||||
--------------
|
||||
|
||||
**`Next: Adding JIT and Optimizer Support <PythonLangImpl4.html>`_**
|
||||
|
|
@ -1,941 +0,0 @@
|
|||
---
|
||||
layout: page
|
||||
title: "Kaleidoscope: Chapter 4"
|
||||
---
|
||||
|
||||
# Adding JIT and Optimizer Support
|
||||
|
||||
Written by [Chris Lattner](mailto:sabre@nondot.org)
|
||||
and [Max Shawabkeh](http://max99x.com)
|
||||
|
||||
|
||||
**Chapter 4**
|
||||
|
||||
|
||||
* This will become a table of contents (this text will be scraped).
|
||||
{:toc}
|
||||
|
||||
|
||||
**[Chapter 5: Extending the Language: Control Flow](PythonLangImpl5.html)**
|
||||
|
||||
|
||||
# Introduction # {#intro}
|
||||
|
||||
Welcome to Chapter 4 of the
|
||||
[Implementing a language with LLVM](http://www.llvm.org/docs/tutorial/index.html)
|
||||
tutorial. Chapters 1-3 described the implementation of a simple
|
||||
language and added support for generating LLVM IR. This chapter describes
|
||||
two new techniques: adding optimizer support to your language, and adding JIT
|
||||
compiler support. These additions will demonstrate how to get nice, efficient
|
||||
code for the Kaleidoscope language.
|
||||
|
||||
|
||||
* * *
|
||||
|
||||
# Trivial Constant Folding # {#trivialconstfold}
|
||||
|
||||
Our demonstration for Chapter 3 is elegant and easy to extend. Unfortunately,
|
||||
it does not produce wonderful code. The LLVM Builder, however, does give us
|
||||
obvious optimizations when compiling simple code:
|
||||
|
||||
|
||||
{% highlight bash %}
|
||||
ready> def test(x) 1+2+x
|
||||
Read function definition:
|
||||
define double @test(double %x) {
|
||||
entry:
|
||||
%addtmp = fadd double 3.000000e+00, %x
|
||||
ret double %addtmp
|
||||
}
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
This code is not a literal transcription of the AST built by parsing the
|
||||
input. That would be:
|
||||
|
||||
|
||||
{% highlight bash %}
|
||||
ready> def test(x) 1+2+x
|
||||
Read function definition:
|
||||
define double @test(double %x) {
|
||||
entry:
|
||||
%addtmp = fadd double 2.000000e+00, 1.000000e+00
|
||||
%addtmp1 = fadd double %addtmp, %x
|
||||
ret double %addtmp1
|
||||
}
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
Constant folding, as seen above, in particular, is a very common and very
|
||||
important optimization: so much so that many language implementors implement
|
||||
constant folding support in their AST representation.
|
||||
|
||||
With LLVM, you don't need this support in the AST. Since all calls to build
|
||||
LLVM IR go through the LLVM IR builder, the builder itself checked to see if
|
||||
there was a constant folding opportunity when you call it. If so, it just does
|
||||
the constant fold and return the constant instead of creating an instruction.
|
||||
|
||||
Well, that was easy :). In practice, we recommend always using
|
||||
`llvm.core.Builder` when generating code like this. It has no
|
||||
"syntactic overhead" for its use (you don't have to uglify your compiler with
|
||||
constant checks everywhere) and it can dramatically reduce the amount of
|
||||
LLVM IR that is generated in some cases (particular for languages with a macro
|
||||
preprocessor or that use a lot of constants).
|
||||
|
||||
On the other hand, the `Builder` is limited by the fact that it does
|
||||
all of its analysis inline with the code as it is built. If you take a slightly
|
||||
more complex example:
|
||||
|
||||
|
||||
{% highlight bash %}
|
||||
ready> def test(x) (1+2+x)*(x+(1+2))
|
||||
Read a function definition:
|
||||
define double @test(double %x) {
|
||||
entry:
|
||||
%addtmp = fadd double 3.000000e+00, %x ; <double> [#uses=1]
|
||||
%addtmp1 = fadd double %x, 3.000000e+00 ; <double> [#uses=1]
|
||||
%multmp = fmul double %addtmp, %addtmp1 ; <double> [#uses=1]
|
||||
ret double %multmp
|
||||
}
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
In this case, the LHS and RHS of the multiplication are the same value. We'd
|
||||
really like to see this generate"`tmp = x+3; result = tmp*tmp;` instead
|
||||
of computing `x+3` twice.
|
||||
|
||||
Unfortunately, no amount of local analysis will be able to detect and correct
|
||||
this. This requires two transformations: reassociation of expressions (to
|
||||
make the add's lexically identical) and Common Subexpression Elimination (CSE)
|
||||
to delete the redundant add instruction. Fortunately, LLVM provides a broad
|
||||
range of optimizations that you can use, in the form of "passes".
|
||||
|
||||
|
||||
* * *
|
||||
|
||||
# LLVM Optimization Passes # {#optimizerpasses}
|
||||
|
||||
LLVM provides many optimization passes, which do many different sorts of
|
||||
things and have different tradeoffs. Unlike other systems, LLVM doesn't hold
|
||||
to the mistaken notion that one set of optimizations is right for all languages
|
||||
and for all situations. LLVM allows a compiler implementor to make complete
|
||||
decisions about what optimizations to use, in which order, and in what
|
||||
situation.
|
||||
|
||||
As a concrete example, LLVM supports both "whole module" passes, which look
|
||||
across as large of body of code as they can (often a whole file, but if run
|
||||
at link time, this can be a substantial portion of the whole program). It also
|
||||
supports and includes "per-function" passes which just operate on a single
|
||||
function at a time, without looking at other functions. For more information
|
||||
on passes and how they are run, see the
|
||||
[How to Write a Pass](http://www.llvm.org/docs/WritingAnLLVMPass.html)
|
||||
document and the
|
||||
[List of LLVM Passes](http://www.llvm.org/docs/Passes.html).
|
||||
|
||||
For Kaleidoscope, we are currently generating functions on the fly, one at
|
||||
a time, as the user types them in. We aren't shooting for the ultimate
|
||||
optimization experience in this setting, but we also want to catch the easy and
|
||||
quick stuff where possible. As such, we will choose to run a few per-function
|
||||
optimizations as the user types the function in. If we wanted to make a "static
|
||||
Kaleidoscope compiler", we would use exactly the code we have now, except that
|
||||
we would defer running the optimizer until the entire file has been parsed.
|
||||
|
||||
In order to get per-function optimizations going, we need to set up a
|
||||
[FunctionPassManager](http://www.llvm.org/docs/WritingAnLLVMPass.html#passmanager)
|
||||
to hold and organize the LLVM optimizations that we want
|
||||
to run. Once we have that, we can add a set of optimizations to run. The code
|
||||
looks like this:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
# The function optimization passes manager.
|
||||
g_llvm_pass_manager = FunctionPassManager.new(g_llvm_module)
|
||||
|
||||
# The LLVM execution engine.
|
||||
g_llvm_executor = ExecutionEngine.new(g_llvm_module)
|
||||
|
||||
...
|
||||
|
||||
def main():
|
||||
# Set up the optimizer pipeline. Start with registering info about how the
|
||||
# target lays out data structures.
|
||||
g_llvm_pass_manager.add(g_llvm_executor.target_data)
|
||||
# Do simple "peephole" optimizations and bit-twiddling optzns.
|
||||
g_llvm_pass_manager.add(PASS_INSTRUCTION_COMBINING)
|
||||
# Reassociate expressions.
|
||||
g_llvm_pass_manager.add(PASS_REASSOCIATE)
|
||||
# Eliminate Common SubExpressions.
|
||||
g_llvm_pass_manager.add(PASS_GVN)
|
||||
# Simplify the control flow graph (deleting unreachable blocks, etc).
|
||||
g_llvm_pass_manager.add(PASS_CFG_SIMPLIFICATION)
|
||||
|
||||
g_llvm_pass_manager.initialize()
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
This code defines a `FunctionPassManager`,
|
||||
`g_llvm_pass_manager`. Once it is set up, we use a series of "add" calls
|
||||
to add a bunch of LLVM passes. The first pass is basically boilerplate, it adds
|
||||
a pass so that later optimizations know how the data structures in the program
|
||||
are laid out. (The "`g_llvm_executor`" variable is related to the JIT,
|
||||
which we will get to in the next section.) In this case, we choose to add 4
|
||||
optimization passes. The passes we chose here are a pretty standard set of
|
||||
"cleanup" optimizations that are useful for a wide variety of code. I won't
|
||||
delve into what they do but, believe me, they are a good starting place :).
|
||||
|
||||
Once the pass manager is set up, we need to make use of it. We do this by
|
||||
running it after our newly created function is constructed (in
|
||||
`FunctionNode.CodeGen`), but before it is returned to the client:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
return_value = self.body.CodeGen()
|
||||
g_llvm_builder.ret(return_value)
|
||||
|
||||
# Validate the generated code, checking for consistency.
|
||||
function.verify()
|
||||
|
||||
# Optimize the function.
|
||||
g_llvm_pass_manager.run(function)
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
As you can see, this is pretty straightforward. The
|
||||
`FunctionPassManager` optimizes and updates the LLVM Function in place,
|
||||
improving (hopefully) its body. With this in place, we can try our test above
|
||||
again:
|
||||
|
||||
|
||||
{% highlight bash %}
|
||||
ready> def test(x) (1+2+x)*(x+(1+2))
|
||||
Read a function definition:
|
||||
define double @test(double %x) {
|
||||
entry:
|
||||
%addtmp = fadd double %x, 3.000000e+00 ; <double> [#uses=2]
|
||||
%multmp = fmul double %addtmp, %addtmp ; <double> [#uses=1]
|
||||
ret double %multmp
|
||||
}
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
As expected, we now get our nicely optimized code, saving a floating point
|
||||
add instruction from every execution of this function.
|
||||
|
||||
LLVM provides a wide variety of optimizations that can be used in certain
|
||||
circumstances. Some
|
||||
[documentation about the various passes](http://www.llvm.org/docs/Passes.html)
|
||||
is available, but it isn't very complete. Another good source of
|
||||
ideas can come from looking at the passes that `llvm-gcc` or
|
||||
`llvm-ld` run to get started. The `opt` tool allows you to
|
||||
experiment with passes from the command line, so you can see if they do
|
||||
anything.
|
||||
|
||||
Now that we have reasonable code coming out of our front-end, lets talk about
|
||||
executing it!
|
||||
|
||||
* * *
|
||||
|
||||
# Adding a JIT Compiler # {#jit}
|
||||
|
||||
Code that is available in LLVM IR can have a wide variety of tools
|
||||
applied to it. For example, you can run optimizations on it (as we did above),
|
||||
you can dump it out in textual or binary forms, you can compile the code to an
|
||||
assembly file (.s) for some target, or you can JIT compile it. The nice thing
|
||||
about the LLVM IR representation is that it is the "common currency" between
|
||||
many different parts of the compiler.
|
||||
|
||||
|
||||
In this section, we'll add JIT compiler support to our interpreter. The
|
||||
basic idea that we want for Kaleidoscope is to have the user enter function
|
||||
bodies as they do now, but immediately evaluate the top-level expressions they
|
||||
type in. For example, if they type in "1 + 2", we should evaluate and print
|
||||
out 3. If they define a function, they should be able to call it from the
|
||||
command line.
|
||||
|
||||
In order to do this, we first declare and initialize the JIT. This is done
|
||||
by adding and initializing a global variable:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
# The LLVM execution engine.
|
||||
g_llvm_executor = ExecutionEngine.new(g_llvm_module)
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
This creates an abstract "Execution Engine" which can be either a JIT
|
||||
compiler or the LLVM interpreter. LLVM will automatically pick a JIT compiler
|
||||
for you if one is available for your platform, otherwise it will fall back to
|
||||
the interpreter.
|
||||
|
||||
Once the `ExecutionEngine` is created, the JIT is ready to be used.
|
||||
We can use the `run_function` method of the execution engine to execute
|
||||
a compiled function and get its return value. In our case, this means that we
|
||||
can change the code that parses a top-level expression to look like this:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
def HandleTopLevelExpression(self):
|
||||
try:
|
||||
function = self.ParseTopLevelExpr().CodeGen()
|
||||
result = g_llvm_executor.run_function(function, [])
|
||||
print 'Evaluated to:', result.as_real(Type.double())
|
||||
except Exception, e:
|
||||
print 'Error:', e
|
||||
try:
|
||||
self.Next() # Skip for error recovery.
|
||||
except:
|
||||
pass
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
Recall that we compile top-level expressions into a self-contained LLVM
|
||||
function that takes no arguments and returns the computed double.
|
||||
|
||||
With just these two changes, lets see how Kaleidoscope works now!
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
ready> 4+5
|
||||
Read a top level expression:
|
||||
define double @0() {
|
||||
entry:
|
||||
ret double 9.000000e+00
|
||||
}
|
||||
|
||||
Evaluated to: 9.0
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
Well this looks like it is basically working. The dump of the function
|
||||
shows the "no argument function that always returns double" that we synthesize
|
||||
for each top-level expression that is typed in. This demonstrates very basic
|
||||
functionality, but can we do more?
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
ready> def testfunc(x y) x + y*2
|
||||
Read a function definition:
|
||||
define double @testfunc(double %x, double %y) {
|
||||
entry:
|
||||
%multmp = fmul double %y, 2.000000e+00 ; <double> [#uses=1]
|
||||
%addtmp = fadd double %multmp, %x ; <double> [#uses=1]
|
||||
ret double %addtmp
|
||||
}
|
||||
|
||||
ready> testfunc(4, 10)
|
||||
Read a top level expression:
|
||||
define double @0() {
|
||||
entry:
|
||||
%calltmp = call double @testfunc(double 4.000000e+00, double 1.000000e+01) ; <double> [#uses=1]
|
||||
ret double %calltmp
|
||||
}
|
||||
|
||||
*Evaluated to: 24.0*
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
This illustrates that we can now call user code, but there is something a bit
|
||||
subtle going on here. Note that we only invoke the JIT on the anonymous
|
||||
functions that *call testfunc*, but we never invoked it
|
||||
on *testfunc* itself. What actually happened here is that the JIT
|
||||
scanned for all non-JIT'd functions transitively called from the anonymous
|
||||
function and compiled all of them before returning from `run_function()`.
|
||||
|
||||
|
||||
The JIT provides a number of other more advanced interfaces for things like
|
||||
freeing allocated machine code, rejit'ing functions to update them, etc.
|
||||
However, even with this simple code, we get some surprisingly powerful
|
||||
capabilities - check this out (I removed the dump of the anonymous functions,
|
||||
you should get the idea by now :) :
|
||||
|
||||
|
||||
{% highlight bash %}
|
||||
ready> extern sin(x)
|
||||
Read an extern:
|
||||
declare double @sin(double)
|
||||
|
||||
ready> extern cos(x)
|
||||
Read an extern:
|
||||
declare double @cos(double)
|
||||
|
||||
ready> sin(1.0)
|
||||
*Evaluated to: 0.841470984808*
|
||||
|
||||
ready> def foo(x) sin(x)*sin(x) + cos(x)*cos(x)
|
||||
Read a function definition:
|
||||
define double @foo(double %x) {
|
||||
entry:
|
||||
%calltmp = call double @sin(double %x) ; <double> [#uses=1]
|
||||
%calltmp1 = call double @sin(double %x) ; <double> [#uses=1]
|
||||
%multmp = fmul double %calltmp, %calltmp1 ; <double> [#uses=1]
|
||||
%calltmp2 = call double @cos(double %x) ; <double> [#uses=1]
|
||||
%calltmp3 = call double @cos(double %x) ; <double> [#uses=1]
|
||||
%multmp4 = fmul double %calltmp2, %calltmp3 ; <double> [#uses=1]
|
||||
%addtmp = fadd double %multmp, %multmp4 ; <double> [#uses=1]
|
||||
ret double %addtmp
|
||||
}
|
||||
|
||||
ready> foo(4.0)
|
||||
*Evaluated to: 1.000000*
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
Whoa, how does the JIT know about sin and cos? The answer is surprisingly
|
||||
simple: in this example, the JIT started execution of a function and got to a
|
||||
function call. It realized that the function was not yet JIT compiled and
|
||||
invoked the standard set of routines to resolve the function. In this case,
|
||||
there is no body defined for the function, so the JIT ended up calling
|
||||
`dlsym("sin")` on the Python process that is hosting our Kaleidoscope
|
||||
prompt. Since `sin` is defined within the JIT's address space, it
|
||||
simply patches up calls in the module to call the libm version of `sin`
|
||||
directly.
|
||||
|
||||
One interesting application of this is that we can now extend the language
|
||||
by writing arbitrary C++ code to implement operations. For example, we can
|
||||
create a C file with the following simple function:
|
||||
|
||||
|
||||
|
||||
{% highlight c %}
|
||||
#include <stdio.h>
|
||||
|
||||
double putchard(double x) {
|
||||
putchar((char)x);
|
||||
return 0;
|
||||
}
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
We can then compile this into a shared library with GCC:
|
||||
|
||||
|
||||
{% highlight bash %}
|
||||
gcc -shared -fPIC -o putchard.so putchard.c
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
Now we can load this library into the Python process using
|
||||
`llvm.core.load_library_permanently` and access it from Kaleidoscope to
|
||||
produce simple output to the console:
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
>>> import llvm.core
|
||||
>>> llvm.core.load_library_permanently('/home/max/llvm-py-tutorial/putchard.so')
|
||||
>>> import kaleidoscope
|
||||
>>> kaleidoscope.main()
|
||||
ready> extern putchard(x)
|
||||
Read an extern:
|
||||
declare double @putchard(double)
|
||||
|
||||
ready> putchard(65) + putchard(66) + putchard(67) + putchard(10)
|
||||
*ABC*
|
||||
Evaluated to: 0.0
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
Similar code could be used to implement file I/O, console input, and many
|
||||
other capabilities in Kaleidoscope.
|
||||
|
||||
This completes the JIT and optimizer chapter of the Kaleidoscope tutorial. At
|
||||
this point, we can compile a non-Turing-complete programming language, optimize
|
||||
and JIT compile it in a user-driven way. Next up we'll look into
|
||||
[extending the language with control flow constructs](PythonLangImpl5.html),
|
||||
tackling some interesting LLVM IR issues along the way.
|
||||
|
||||
|
||||
* * *
|
||||
|
||||
# Full Code Listing # {#code}
|
||||
|
||||
Here is the complete code listing for our running example, enhanced with the
|
||||
LLVM JIT and optimizer:
|
||||
|
||||
|
||||
|
||||
{% highlight python %}
|
||||
#!/usr/bin/env python
|
||||
|
||||
import re
|
||||
from llvm.core import Module, Constant, Type, Function, Builder, FCMP_ULT
|
||||
from llvm.ee import ExecutionEngine, TargetData
|
||||
from llvm.passes import FunctionPassManager
|
||||
from llvm.passes import (PASS_INSTRUCTION_COMBINING,
|
||||
PASS_REASSOCIATE,
|
||||
PASS_GVN,
|
||||
PASS_CFG_SIMPLIFICATION)
|
||||
|
||||
################################################################################
|
||||
## Globals
|
||||
################################################################################
|
||||
|
||||
# The LLVM module, which holds all the IR code.
|
||||
g_llvm_module = Module.new('my cool jit')
|
||||
|
||||
# The LLVM instruction builder. Created whenever a new function is entered.
|
||||
g_llvm_builder = None
|
||||
|
||||
# A dictionary that keeps track of which values are defined in the current scope
|
||||
# and what their LLVM representation is.
|
||||
g_named_values = {}
|
||||
|
||||
# The function optimization passes manager.
|
||||
g_llvm_pass_manager = FunctionPassManager.new(g_llvm_module)
|
||||
|
||||
# The LLVM execution engine.
|
||||
g_llvm_executor = ExecutionEngine.new(g_llvm_module)
|
||||
|
||||
################################################################################
|
||||
## Lexer
|
||||
################################################################################
|
||||
|
||||
# The lexer yields one of these types for each token.
|
||||
class EOFToken(object):
|
||||
pass
|
||||
|
||||
class DefToken(object):
|
||||
pass
|
||||
|
||||
class ExternToken(object):
|
||||
pass
|
||||
|
||||
class IdentifierToken(object):
|
||||
def __init__(self, name): self.name = name
|
||||
|
||||
class NumberToken(object):
|
||||
def __init__(self, value): self.value = value
|
||||
|
||||
class CharacterToken(object):
|
||||
def __init__(self, char): self.char = char
|
||||
def __eq__(self, other):
|
||||
return isinstance(other, CharacterToken) and self.char == other.char
|
||||
def __ne__(self, other): return not self == other
|
||||
|
||||
# Regular expressions that tokens and comments of our language.
|
||||
REGEX_NUMBER = re.compile('[0-9]+(?:\.[0-9]+)?')
|
||||
REGEX_IDENTIFIER = re.compile('[a-zA-Z][a-zA-Z0-9]*')
|
||||
REGEX_COMMENT = re.compile('#.*')
|
||||
|
||||
def Tokenize(string):
|
||||
while string:
|
||||
# Skip whitespace.
|
||||
if string[0].isspace():
|
||||
string = string[1:]
|
||||
continue
|
||||
|
||||
# Run regexes.
|
||||
comment_match = REGEX_COMMENT.match(string)
|
||||
number_match = REGEX_NUMBER.match(string)
|
||||
identifier_match = REGEX_IDENTIFIER.match(string)
|
||||
|
||||
# Check if any of the regexes matched and yield the appropriate result.
|
||||
if comment_match:
|
||||
comment = comment_match.group(0)
|
||||
string = string[len(comment):]
|
||||
elif number_match:
|
||||
number = number_match.group(0)
|
||||
yield NumberToken(float(number))
|
||||
string = string[len(number):]
|
||||
elif identifier_match:
|
||||
identifier = identifier_match.group(0)
|
||||
# Check if we matched a keyword.
|
||||
if identifier == 'def':
|
||||
yield DefToken()
|
||||
elif identifier == 'extern':
|
||||
yield ExternToken()
|
||||
else:
|
||||
yield IdentifierToken(identifier)
|
||||
string = string[len(identifier):]
|
||||
else:
|
||||
# Yield the ASCII value of the unknown character.
|
||||
yield CharacterToken(string[0])
|
||||
string = string[1:]
|
||||
|
||||
yield EOFToken()
|
||||
|
||||
################################################################################
|
||||
## Abstract Syntax Tree (aka Parse Tree)
|
||||
################################################################################
|
||||
|
||||
# Base class for all expression nodes.
|
||||
class ExpressionNode(object):
|
||||
pass
|
||||
|
||||
# Expression class for numeric literals like "1.0".
|
||||
class NumberExpressionNode(ExpressionNode):
|
||||
|
||||
def __init__(self, value):
|
||||
self.value = value
|
||||
|
||||
def CodeGen(self):
|
||||
return Constant.real(Type.double(), self.value)
|
||||
|
||||
# Expression class for referencing a variable, like "a".
|
||||
class VariableExpressionNode(ExpressionNode):
|
||||
|
||||
def __init__(self, name):
|
||||
self.name = name
|
||||
|
||||
def CodeGen(self):
|
||||
if self.name in g_named_values:
|
||||
return g_named_values[self.name]
|
||||
else:
|
||||
raise RuntimeError('Unknown variable name: ' + self.name)
|
||||
|
||||
# Expression class for a binary operator.
|
||||
class BinaryOperatorExpressionNode(ExpressionNode):
|
||||
|
||||
def __init__(self, operator, left, right):
|
||||
self.operator = operator
|
||||
self.left = left
|
||||
self.right = right
|
||||
|
||||
def CodeGen(self):
|
||||
left = self.left.CodeGen()
|
||||
right = self.right.CodeGen()
|
||||
|
||||
if self.operator == '+':
|
||||
return g_llvm_builder.fadd(left, right, 'addtmp')
|
||||
elif self.operator == '-':
|
||||
return g_llvm_builder.fsub(left, right, 'subtmp')
|
||||
elif self.operator == '*':
|
||||
return g_llvm_builder.fmul(left, right, 'multmp')
|
||||
elif self.operator == '<':
|
||||
result = g_llvm_builder.fcmp(FCMP_ULT, left, right, 'cmptmp')
|
||||
# Convert bool 0 or 1 to double 0.0 or 1.0.
|
||||
return g_llvm_builder.uitofp(result, Type.double(), 'booltmp')
|
||||
else:
|
||||
raise RuntimeError('Unknown binary operator.')
|
||||
|
||||
# Expression class for function calls.
|
||||
class CallExpressionNode(ExpressionNode):
|
||||
|
||||
def __init__(self, callee, args):
|
||||
self.callee = callee
|
||||
self.args = args
|
||||
|
||||
def CodeGen(self):
|
||||
# Look up the name in the global module table.
|
||||
callee = g_llvm_module.get_function_named(self.callee)
|
||||
|
||||
# Check for argument mismatch error.
|
||||
if len(callee.args) != len(self.args):
|
||||
raise RuntimeError('Incorrect number of arguments passed.')
|
||||
|
||||
arg_values = [i.CodeGen() for i in self.args]
|
||||
|
||||
return g_llvm_builder.call(callee, arg_values, 'calltmp')
|
||||
|
||||
# This class represents the "prototype" for a function, which captures its name,
|
||||
# and its argument names (thus implicitly the number of arguments the function
|
||||
# takes).
|
||||
class PrototypeNode(object):
|
||||
|
||||
def __init__(self, name, args):
|
||||
self.name = name
|
||||
self.args = args
|
||||
|
||||
def CodeGen(self):
|
||||
# Make the function type, eg. double(double,double).
|
||||
funct_type = Type.function(
|
||||
Type.double(), [Type.double()] * len(self.args), False)
|
||||
|
||||
function = Function.new(g_llvm_module, funct_type, self.name)
|
||||
|
||||
# If the name conflicted, there was already something with the same name.
|
||||
# If it has a body, don't allow redefinition or reextern.
|
||||
if function.name != self.name:
|
||||
function.delete()
|
||||
function = g_llvm_module.get_function_named(self.name)
|
||||
|
||||
# If the function already has a body, reject this.
|
||||
if not function.is_declaration:
|
||||
raise RuntimeError('Redefinition of function.')
|
||||
|
||||
# If F took a different number of args, reject.
|
||||
if len(callee.args) != len(self.args):
|
||||
raise RuntimeError('Redeclaration of a function with different number '
|
||||
'of args.')
|
||||
|
||||
# Set names for all arguments and add them to the variables symbol table.
|
||||
for arg, arg_name in zip(function.args, self.args):
|
||||
arg.name = arg_name
|
||||
# Add arguments to variable symbol table.
|
||||
g_named_values[arg_name] = arg
|
||||
|
||||
return function
|
||||
|
||||
# This class represents a function definition itself.
|
||||
class FunctionNode(object):
|
||||
|
||||
def __init__(self, prototype, body):
|
||||
self.prototype = prototype
|
||||
self.body = body
|
||||
|
||||
def CodeGen(self):
|
||||
# Clear scope.
|
||||
g_named_values.clear()
|
||||
|
||||
# Create a function object.
|
||||
function = self.prototype.CodeGen()
|
||||
|
||||
# Create a new basic block to start insertion into.
|
||||
block = function.append_basic_block('entry')
|
||||
global g_llvm_builder
|
||||
g_llvm_builder = Builder.new(block)
|
||||
|
||||
# Finish off the function.
|
||||
try:
|
||||
return_value = self.body.CodeGen()
|
||||
g_llvm_builder.ret(return_value)
|
||||
|
||||
# Validate the generated code, checking for consistency.
|
||||
function.verify()
|
||||
|
||||
# Optimize the function.
|
||||
g_llvm_pass_manager.run(function)
|
||||
except:
|
||||
function.delete()
|
||||
raise
|
||||
|
||||
return function
|
||||
|
||||
|
||||
################################################################################
|
||||
## Parser
|
||||
################################################################################
|
||||
|
||||
class Parser(object):
|
||||
|
||||
def __init__(self, tokens, binop_precedence):
|
||||
self.tokens = tokens
|
||||
self.binop_precedence = binop_precedence
|
||||
self.Next()
|
||||
|
||||
# Provide a simple token buffer. Parser.current is the current token the
|
||||
# parser is looking at. Parser.Next() reads another token from the lexer and
|
||||
# updates Parser.current with its results.
|
||||
def Next(self):
|
||||
self.current = self.tokens.next()
|
||||
|
||||
# Gets the precedence of the current token, or -1 if the token is not a binary
|
||||
# operator.
|
||||
def GetCurrentTokenPrecedence(self):
|
||||
if isinstance(self.current, CharacterToken):
|
||||
return self.binop_precedence.get(self.current.char, -1)
|
||||
else:
|
||||
return -1
|
||||
|
||||
# identifierexpr ::= identifier | identifier '(' expression* ')'
|
||||
def ParseIdentifierExpr(self):
|
||||
identifier_name = self.current.name
|
||||
self.Next() # eat identifier.
|
||||
|
||||
if self.current != CharacterToken('('): # Simple variable reference.
|
||||
return VariableExpressionNode(identifier_name)
|
||||
|
||||
# Call.
|
||||
self.Next() # eat '('.
|
||||
args = []
|
||||
if self.current != CharacterToken(')'):
|
||||
while True:
|
||||
args.append(self.ParseExpression())
|
||||
if self.current == CharacterToken(')'):
|
||||
break
|
||||
elif self.current != CharacterToken(','):
|
||||
raise RuntimeError('Expected ")" or "," in argument list.')
|
||||
self.Next()
|
||||
|
||||
self.Next() # eat ')'.
|
||||
return CallExpressionNode(identifier_name, args)
|
||||
|
||||
# numberexpr ::= number
|
||||
def ParseNumberExpr(self):
|
||||
result = NumberExpressionNode(self.current.value)
|
||||
self.Next() # consume the number.
|
||||
return result
|
||||
|
||||
# parenexpr ::= '(' expression ')'
|
||||
def ParseParenExpr(self):
|
||||
self.Next() # eat '('.
|
||||
|
||||
contents = self.ParseExpression()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")".')
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return contents
|
||||
|
||||
# primary ::= identifierexpr | numberexpr | parenexpr
|
||||
def ParsePrimary(self):
|
||||
if isinstance(self.current, IdentifierToken):
|
||||
return self.ParseIdentifierExpr()
|
||||
elif isinstance(self.current, NumberToken):
|
||||
return self.ParseNumberExpr()
|
||||
elif self.current == CharacterToken('('):
|
||||
return self.ParseParenExpr()
|
||||
else:
|
||||
raise RuntimeError('Unknown token when expecting an expression.')
|
||||
|
||||
# binoprhs ::= (operator primary)*
|
||||
def ParseBinOpRHS(self, left, left_precedence):
|
||||
# If this is a binary operator, find its precedence.
|
||||
while True:
|
||||
precedence = self.GetCurrentTokenPrecedence()
|
||||
|
||||
# If this is a binary operator that binds at least as tightly as the
|
||||
# current one, consume it; otherwise we are done.
|
||||
if precedence < left_precedence:
|
||||
return left
|
||||
|
||||
binary_operator = self.current.char
|
||||
self.Next() # eat the operator.
|
||||
|
||||
# Parse the primary expression after the binary operator.
|
||||
right = self.ParsePrimary()
|
||||
|
||||
# If binary_operator binds less tightly with right than the operator after
|
||||
# right, let the pending operator take right as its left.
|
||||
next_precedence = self.GetCurrentTokenPrecedence()
|
||||
if precedence < next_precedence:
|
||||
right = self.ParseBinOpRHS(right, precedence + 1)
|
||||
|
||||
# Merge left/right.
|
||||
left = BinaryOperatorExpressionNode(binary_operator, left, right)
|
||||
|
||||
# expression ::= primary binoprhs
|
||||
def ParseExpression(self):
|
||||
left = self.ParsePrimary()
|
||||
return self.ParseBinOpRHS(left, 0)
|
||||
|
||||
# prototype ::= id '(' id* ')'
|
||||
def ParsePrototype(self):
|
||||
if not isinstance(self.current, IdentifierToken):
|
||||
raise RuntimeError('Expected function name in prototype.')
|
||||
|
||||
function_name = self.current.name
|
||||
self.Next() # eat function name.
|
||||
|
||||
if self.current != CharacterToken('('):
|
||||
raise RuntimeError('Expected "(" in prototype.')
|
||||
self.Next() # eat '('.
|
||||
|
||||
arg_names = []
|
||||
while isinstance(self.current, IdentifierToken):
|
||||
arg_names.append(self.current.name)
|
||||
self.Next()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")" in prototype.')
|
||||
|
||||
# Success.
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return PrototypeNode(function_name, arg_names)
|
||||
|
||||
# definition ::= 'def' prototype expression
|
||||
def ParseDefinition(self):
|
||||
self.Next() # eat def.
|
||||
proto = self.ParsePrototype()
|
||||
body = self.ParseExpression()
|
||||
return FunctionNode(proto, body)
|
||||
|
||||
# toplevelexpr ::= expression
|
||||
def ParseTopLevelExpr(self):
|
||||
proto = PrototypeNode('', [])
|
||||
return FunctionNode(proto, self.ParseExpression())
|
||||
|
||||
# external ::= 'extern' prototype
|
||||
def ParseExtern(self):
|
||||
self.Next() # eat extern.
|
||||
return self.ParsePrototype()
|
||||
|
||||
# Top-Level parsing
|
||||
def HandleDefinition(self):
|
||||
self.Handle(self.ParseDefinition, 'Read a function definition:')
|
||||
|
||||
def HandleExtern(self):
|
||||
self.Handle(self.ParseExtern, 'Read an extern:')
|
||||
|
||||
def HandleTopLevelExpression(self):
|
||||
try:
|
||||
function = self.ParseTopLevelExpr().CodeGen()
|
||||
result = g_llvm_executor.run_function(function, [])
|
||||
print 'Evaluated to:', result.as_real(Type.double())
|
||||
except Exception, e:
|
||||
print 'Error:', e
|
||||
try:
|
||||
self.Next() # Skip for error recovery.
|
||||
except:
|
||||
pass
|
||||
|
||||
def Handle(self, function, message):
|
||||
try:
|
||||
print message, function().CodeGen()
|
||||
except Exception, e:
|
||||
print 'Error:', e
|
||||
try:
|
||||
self.Next() # Skip for error recovery.
|
||||
except:
|
||||
pass
|
||||
|
||||
################################################################################
|
||||
## Main driver code.
|
||||
################################################################################
|
||||
|
||||
def main():
|
||||
# Set up the optimizer pipeline. Start with registering info about how the
|
||||
# target lays out data structures.
|
||||
g_llvm_pass_manager.add(g_llvm_executor.target_data)
|
||||
# Do simple "peephole" optimizations and bit-twiddling optzns.
|
||||
g_llvm_pass_manager.add(PASS_INSTRUCTION_COMBINING)
|
||||
# Reassociate expressions.
|
||||
g_llvm_pass_manager.add(PASS_REASSOCIATE)
|
||||
# Eliminate Common SubExpressions.
|
||||
g_llvm_pass_manager.add(PASS_GVN)
|
||||
# Simplify the control flow graph (deleting unreachable blocks, etc).
|
||||
g_llvm_pass_manager.add(PASS_CFG_SIMPLIFICATION)
|
||||
|
||||
g_llvm_pass_manager.initialize()
|
||||
|
||||
# Install standard binary operators.
|
||||
# 1 is lowest possible precedence. 40 is the highest.
|
||||
operator_precedence = {
|
||||
'<': 10,
|
||||
'+': 20,
|
||||
'-': 20,
|
||||
'*': 40
|
||||
}
|
||||
|
||||
# Run the main "interpreter loop".
|
||||
while True:
|
||||
print 'ready>',
|
||||
try:
|
||||
raw = raw_input()
|
||||
except KeyboardInterrupt:
|
||||
break
|
||||
|
||||
parser = Parser(Tokenize(raw), operator_precedence)
|
||||
while True:
|
||||
# top ::= definition | external | expression | EOF
|
||||
if isinstance(parser.current, EOFToken):
|
||||
break
|
||||
if isinstance(parser.current, DefToken):
|
||||
parser.HandleDefinition()
|
||||
elif isinstance(parser.current, ExternToken):
|
||||
parser.HandleExtern()
|
||||
else:
|
||||
parser.HandleTopLevelExpression()
|
||||
|
||||
# Print out all of the generated code.
|
||||
print '\n', g_llvm_module
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
{% endhighlight %}
|
||||
|
||||
* * *
|
||||
|
||||
**[Next: Extending the language: control flow](PythonLangImpl5.html)**
|
||||
|
||||
803
docs/source/doc/kaleidoscope/PythonLangImpl4.rst
Normal file
803
docs/source/doc/kaleidoscope/PythonLangImpl4.rst
Normal file
|
|
@ -0,0 +1,803 @@
|
|||
+------------------------------------+
|
||||
| layout: page |
|
||||
+------------------------------------+
|
||||
| title: "Kaleidoscope: Chapter 4" |
|
||||
+------------------------------------+
|
||||
|
||||
Adding JIT and Optimizer Support
|
||||
================================
|
||||
|
||||
Written by `Chris Lattner <mailto:sabre@nondot.org>`_ and `Max
|
||||
Shawabkeh <http://max99x.com>`_
|
||||
|
||||
**Chapter 4**
|
||||
|
||||
- This will become a table of contents (this text will be scraped).
|
||||
{:toc}
|
||||
|
||||
**`Chapter 5: Extending the Language: Control
|
||||
Flow <PythonLangImpl5.html>`_**
|
||||
|
||||
Introduction # {#intro}
|
||||
=======================
|
||||
|
||||
Welcome to Chapter 4 of the `Implementing a language with
|
||||
LLVM <http://www.llvm.org/docs/tutorial/index.html>`_ tutorial. Chapters
|
||||
1-3 described the implementation of a simple language and added support
|
||||
for generating LLVM IR. This chapter describes two new techniques:
|
||||
adding optimizer support to your language, and adding JIT compiler
|
||||
support. These additions will demonstrate how to get nice, efficient
|
||||
code for the Kaleidoscope language.
|
||||
|
||||
--------------
|
||||
|
||||
Trivial Constant Folding # {#trivialconstfold}
|
||||
==============================================
|
||||
|
||||
Our demonstration for Chapter 3 is elegant and easy to extend.
|
||||
Unfortunately, it does not produce wonderful code. The LLVM Builder,
|
||||
however, does give us obvious optimizations when compiling simple code:
|
||||
|
||||
{% highlight bash %} ready> def test(x) 1+2+x Read function definition:
|
||||
define double @test(double %x) { entry: %addtmp = fadd double
|
||||
3.000000e+00, %x ret double %addtmp } {% endhighlight %}
|
||||
|
||||
This code is not a literal transcription of the AST built by parsing the
|
||||
input. That would be:
|
||||
|
||||
{% highlight bash %} ready> def test(x) 1+2+x Read function definition:
|
||||
define double @test(double %x) { entry: %addtmp = fadd double
|
||||
2.000000e+00, 1.000000e+00 %addtmp1 = fadd double %addtmp, %x ret double
|
||||
%addtmp1 } {% endhighlight %}
|
||||
|
||||
Constant folding, as seen above, in particular, is a very common and
|
||||
very important optimization: so much so that many language implementors
|
||||
implement constant folding support in their AST representation.
|
||||
|
||||
With LLVM, you don't need this support in the AST. Since all calls to
|
||||
build LLVM IR go through the LLVM IR builder, the builder itself checked
|
||||
to see if there was a constant folding opportunity when you call it. If
|
||||
so, it just does the constant fold and return the constant instead of
|
||||
creating an instruction.
|
||||
|
||||
Well, that was easy :). In practice, we recommend always using
|
||||
``llvm.core.Builder`` when generating code like this. It has no
|
||||
"syntactic overhead" for its use (you don't have to uglify your compiler
|
||||
with constant checks everywhere) and it can dramatically reduce the
|
||||
amount of LLVM IR that is generated in some cases (particular for
|
||||
languages with a macro preprocessor or that use a lot of constants).
|
||||
|
||||
On the other hand, the ``Builder`` is limited by the fact that it does
|
||||
all of its analysis inline with the code as it is built. If you take a
|
||||
slightly more complex example:
|
||||
|
||||
{% highlight bash %} ready> def test(x) (1+2+x)\*(x+(1+2)) Read a
|
||||
function definition: define double @test(double %x) { entry: %addtmp =
|
||||
fadd double 3.000000e+00, %x ; [#uses=1] %addtmp1 = fadd double %x,
|
||||
3.000000e+00 ; [#uses=1] %multmp = fmul double %addtmp, %addtmp1 ;
|
||||
[#uses=1] ret double %multmp } {% endhighlight %}
|
||||
|
||||
In this case, the LHS and RHS of the multiplication are the same value.
|
||||
We'd really like to see this generate"``tmp = x+3; result = tmp*tmp;``
|
||||
instead of computing ``x+3`` twice.
|
||||
|
||||
Unfortunately, no amount of local analysis will be able to detect and
|
||||
correct this. This requires two transformations: reassociation of
|
||||
expressions (to make the add's lexically identical) and Common
|
||||
Subexpression Elimination (CSE) to delete the redundant add instruction.
|
||||
Fortunately, LLVM provides a broad range of optimizations that you can
|
||||
use, in the form of "passes".
|
||||
|
||||
--------------
|
||||
|
||||
LLVM Optimization Passes # {#optimizerpasses}
|
||||
=============================================
|
||||
|
||||
LLVM provides many optimization passes, which do many different sorts of
|
||||
things and have different tradeoffs. Unlike other systems, LLVM doesn't
|
||||
hold to the mistaken notion that one set of optimizations is right for
|
||||
all languages and for all situations. LLVM allows a compiler implementor
|
||||
to make complete decisions about what optimizations to use, in which
|
||||
order, and in what situation.
|
||||
|
||||
As a concrete example, LLVM supports both "whole module" passes, which
|
||||
look across as large of body of code as they can (often a whole file,
|
||||
but if run at link time, this can be a substantial portion of the whole
|
||||
program). It also supports and includes "per-function" passes which just
|
||||
operate on a single function at a time, without looking at other
|
||||
functions. For more information on passes and how they are run, see the
|
||||
`How to Write a Pass <http://www.llvm.org/docs/WritingAnLLVMPass.html>`_
|
||||
document and the `List of LLVM
|
||||
Passes <http://www.llvm.org/docs/Passes.html>`_.
|
||||
|
||||
For Kaleidoscope, we are currently generating functions on the fly, one
|
||||
at a time, as the user types them in. We aren't shooting for the
|
||||
ultimate optimization experience in this setting, but we also want to
|
||||
catch the easy and quick stuff where possible. As such, we will choose
|
||||
to run a few per-function optimizations as the user types the function
|
||||
in. If we wanted to make a "static Kaleidoscope compiler", we would use
|
||||
exactly the code we have now, except that we would defer running the
|
||||
optimizer until the entire file has been parsed.
|
||||
|
||||
In order to get per-function optimizations going, we need to set up a
|
||||
`FunctionPassManager <http://www.llvm.org/docs/WritingAnLLVMPass.html#passmanager>`_
|
||||
to hold and organize the LLVM optimizations that we want to run. Once we
|
||||
have that, we can add a set of optimizations to run. The code looks like
|
||||
this:
|
||||
|
||||
{% highlight python %} # The function optimization passes manager.
|
||||
g\_llvm\_pass\_manager = FunctionPassManager.new(g\_llvm\_module)
|
||||
|
||||
The LLVM execution engine.
|
||||
==========================
|
||||
|
||||
g\_llvm\_executor = ExecutionEngine.new(g\_llvm\_module)
|
||||
|
||||
...
|
||||
|
||||
def main(): # Set up the optimizer pipeline. Start with registering info
|
||||
about how the # target lays out data structures.
|
||||
g\_llvm\_pass\_manager.add(g\_llvm\_executor.target\_data) # Do simple
|
||||
"peephole" optimizations and bit-twiddling optzns.
|
||||
g\_llvm\_pass\_manager.add(PASS\_INSTRUCTION\_COMBINING) # Reassociate
|
||||
expressions. g\_llvm\_pass\_manager.add(PASS\_REASSOCIATE) # Eliminate
|
||||
Common SubExpressions. g\_llvm\_pass\_manager.add(PASS\_GVN) # Simplify
|
||||
the control flow graph (deleting unreachable blocks, etc).
|
||||
g\_llvm\_pass\_manager.add(PASS\_CFG\_SIMPLIFICATION)
|
||||
|
||||
g\_llvm\_pass\_manager.initialize() {% endhighlight %}
|
||||
|
||||
This code defines a ``FunctionPassManager``, ``g_llvm_pass_manager``.
|
||||
Once it is set up, we use a series of "add" calls to add a bunch of LLVM
|
||||
passes. The first pass is basically boilerplate, it adds a pass so that
|
||||
later optimizations know how the data structures in the program are laid
|
||||
out. (The "``g_llvm_executor``\ " variable is related to the JIT, which
|
||||
we will get to in the next section.) In this case, we choose to add 4
|
||||
optimization passes. The passes we chose here are a pretty standard set
|
||||
of "cleanup" optimizations that are useful for a wide variety of code. I
|
||||
won't delve into what they do but, believe me, they are a good starting
|
||||
place :).
|
||||
|
||||
Once the pass manager is set up, we need to make use of it. We do this
|
||||
by running it after our newly created function is constructed (in
|
||||
``FunctionNode.CodeGen``), but before it is returned to the client:
|
||||
|
||||
{% highlight python %} return\_value = self.body.CodeGen()
|
||||
g\_llvm\_builder.ret(return\_value)
|
||||
|
||||
::
|
||||
|
||||
# Validate the generated code, checking for consistency.
|
||||
function.verify()
|
||||
|
||||
# Optimize the function.
|
||||
g_llvm_pass_manager.run(function)
|
||||
|
||||
{% endhighlight %}
|
||||
|
||||
As you can see, this is pretty straightforward. The
|
||||
``FunctionPassManager`` optimizes and updates the LLVM Function in
|
||||
place, improving (hopefully) its body. With this in place, we can try
|
||||
our test above again:
|
||||
|
||||
{% highlight bash %} ready> def test(x) (1+2+x)\*(x+(1+2)) Read a
|
||||
function definition: define double @test(double %x) { entry: %addtmp =
|
||||
fadd double %x, 3.000000e+00 ; [#uses=2] %multmp = fmul double %addtmp,
|
||||
%addtmp ; [#uses=1] ret double %multmp } {% endhighlight %}
|
||||
|
||||
As expected, we now get our nicely optimized code, saving a floating
|
||||
point add instruction from every execution of this function.
|
||||
|
||||
LLVM provides a wide variety of optimizations that can be used in
|
||||
certain circumstances. Some `documentation about the various
|
||||
passes <http://www.llvm.org/docs/Passes.html>`_ is available, but it
|
||||
isn't very complete. Another good source of ideas can come from looking
|
||||
at the passes that ``llvm-gcc`` or ``llvm-ld`` run to get started. The
|
||||
``opt`` tool allows you to experiment with passes from the command line,
|
||||
so you can see if they do anything.
|
||||
|
||||
Now that we have reasonable code coming out of our front-end, lets talk
|
||||
about executing it!
|
||||
|
||||
--------------
|
||||
|
||||
Adding a JIT Compiler # {#jit}
|
||||
==============================
|
||||
|
||||
Code that is available in LLVM IR can have a wide variety of tools
|
||||
applied to it. For example, you can run optimizations on it (as we did
|
||||
above), you can dump it out in textual or binary forms, you can compile
|
||||
the code to an assembly file (.s) for some target, or you can JIT
|
||||
compile it. The nice thing about the LLVM IR representation is that it
|
||||
is the "common currency" between many different parts of the compiler.
|
||||
|
||||
In this section, we'll add JIT compiler support to our interpreter. The
|
||||
basic idea that we want for Kaleidoscope is to have the user enter
|
||||
function bodies as they do now, but immediately evaluate the top-level
|
||||
expressions they type in. For example, if they type in "1 + 2", we
|
||||
should evaluate and print out 3. If they define a function, they should
|
||||
be able to call it from the command line.
|
||||
|
||||
In order to do this, we first declare and initialize the JIT. This is
|
||||
done by adding and initializing a global variable:
|
||||
|
||||
{% highlight python %} # The LLVM execution engine. g\_llvm\_executor =
|
||||
ExecutionEngine.new(g\_llvm\_module) {% endhighlight %}
|
||||
|
||||
This creates an abstract "Execution Engine" which can be either a JIT
|
||||
compiler or the LLVM interpreter. LLVM will automatically pick a JIT
|
||||
compiler for you if one is available for your platform, otherwise it
|
||||
will fall back to the interpreter.
|
||||
|
||||
Once the ``ExecutionEngine`` is created, the JIT is ready to be used. We
|
||||
can use the ``run_function`` method of the execution engine to execute a
|
||||
compiled function and get its return value. In our case, this means that
|
||||
we can change the code that parses a top-level expression to look like
|
||||
this:
|
||||
|
||||
{% highlight python %} def HandleTopLevelExpression(self): try: function
|
||||
= self.ParseTopLevelExpr().CodeGen() result =
|
||||
g\_llvm\_executor.run\_function(function, []) print 'Evaluated to:',
|
||||
result.as\_real(Type.double()) except Exception, e: print 'Error:', e
|
||||
try: self.Next() # Skip for error recovery. except: pass {% endhighlight
|
||||
%}
|
||||
|
||||
Recall that we compile top-level expressions into a self-contained LLVM
|
||||
function that takes no arguments and returns the computed double.
|
||||
|
||||
With just these two changes, lets see how Kaleidoscope works now!
|
||||
|
||||
{% highlight python %} ready> 4+5 Read a top level expression: define
|
||||
double @0() { entry: ret double 9.000000e+00 }
|
||||
|
||||
Evaluated to: 9.0 {% endhighlight %}
|
||||
|
||||
Well this looks like it is basically working. The dump of the function
|
||||
shows the "no argument function that always returns double" that we
|
||||
synthesize for each top-level expression that is typed in. This
|
||||
demonstrates very basic functionality, but can we do more?
|
||||
|
||||
{% highlight python %} ready> def testfunc(x y) x + y\*2 Read a function
|
||||
definition: define double @testfunc(double %x, double %y) { entry:
|
||||
%multmp = fmul double %y, 2.000000e+00 ; [#uses=1] %addtmp = fadd double
|
||||
%multmp, %x ; [#uses=1] ret double %addtmp }
|
||||
|
||||
ready> testfunc(4, 10) Read a top level expression: define double @0() {
|
||||
entry: %calltmp = call double @testfunc(double 4.000000e+00, double
|
||||
1.000000e+01) ; [#uses=1] ret double %calltmp }
|
||||
|
||||
*Evaluated to: 24.0* {% endhighlight %}
|
||||
|
||||
This illustrates that we can now call user code, but there is something
|
||||
a bit subtle going on here. Note that we only invoke the JIT on the
|
||||
anonymous functions that *call testfunc*, but we never invoked it on
|
||||
*testfunc* itself. What actually happened here is that the JIT scanned
|
||||
for all non-JIT'd functions transitively called from the anonymous
|
||||
function and compiled all of them before returning from
|
||||
``run_function()``.
|
||||
|
||||
The JIT provides a number of other more advanced interfaces for things
|
||||
like freeing allocated machine code, rejit'ing functions to update them,
|
||||
etc. However, even with this simple code, we get some surprisingly
|
||||
powerful capabilities - check this out (I removed the dump of the
|
||||
anonymous functions, you should get the idea by now :) :
|
||||
|
||||
{% highlight bash %} ready> extern sin(x) Read an extern: declare double
|
||||
@sin(double)
|
||||
|
||||
ready> extern cos(x) Read an extern: declare double @cos(double)
|
||||
|
||||
ready> sin(1.0) *Evaluated to: 0.841470984808*
|
||||
|
||||
ready> def foo(x) sin(x)\ *sin(x) + cos(x)*\ cos(x) Read a function
|
||||
definition: define double @foo(double %x) { entry: %calltmp = call
|
||||
double @sin(double %x) ; [#uses=1] %calltmp1 = call double @sin(double
|
||||
%x) ; [#uses=1] %multmp = fmul double %calltmp, %calltmp1 ; [#uses=1]
|
||||
%calltmp2 = call double @cos(double %x) ; [#uses=1] %calltmp3 = call
|
||||
double @cos(double %x) ; [#uses=1] %multmp4 = fmul double %calltmp2,
|
||||
%calltmp3 ; [#uses=1] %addtmp = fadd double %multmp, %multmp4 ;
|
||||
[#uses=1] ret double %addtmp }
|
||||
|
||||
ready> foo(4.0) *Evaluated to: 1.000000* {% endhighlight %}
|
||||
|
||||
Whoa, how does the JIT know about sin and cos? The answer is
|
||||
surprisingly simple: in this example, the JIT started execution of a
|
||||
function and got to a function call. It realized that the function was
|
||||
not yet JIT compiled and invoked the standard set of routines to resolve
|
||||
the function. In this case, there is no body defined for the function,
|
||||
so the JIT ended up calling ``dlsym("sin")`` on the Python process that
|
||||
is hosting our Kaleidoscope prompt. Since ``sin`` is defined within the
|
||||
JIT's address space, it simply patches up calls in the module to call
|
||||
the libm version of ``sin`` directly.
|
||||
|
||||
One interesting application of this is that we can now extend the
|
||||
language by writing arbitrary C++ code to implement operations. For
|
||||
example, we can create a C file with the following simple function:
|
||||
|
||||
{% highlight c %} #include
|
||||
|
||||
double putchard(double x) { putchar((char)x); return 0; } {%
|
||||
endhighlight %}
|
||||
|
||||
We can then compile this into a shared library with GCC:
|
||||
|
||||
{% highlight bash %} gcc -shared -fPIC -o putchard.so putchard.c {%
|
||||
endhighlight %}
|
||||
|
||||
Now we can load this library into the Python process using
|
||||
``llvm.core.load_library_permanently`` and access it from Kaleidoscope
|
||||
to produce simple output to the console:
|
||||
|
||||
{% highlight python %} >>> import llvm.core >>>
|
||||
llvm.core.load\_library\_permanently('/home/max/llvm-py-tutorial/putchard.so')
|
||||
>>> import kaleidoscope >>> kaleidoscope.main() ready> extern
|
||||
putchard(x) Read an extern: declare double @putchard(double)
|
||||
|
||||
ready> putchard(65) + putchard(66) + putchard(67) + putchard(10) *ABC*
|
||||
Evaluated to: 0.0 {% endhighlight %}
|
||||
|
||||
Similar code could be used to implement file I/O, console input, and
|
||||
many other capabilities in Kaleidoscope.
|
||||
|
||||
This completes the JIT and optimizer chapter of the Kaleidoscope
|
||||
tutorial. At this point, we can compile a non-Turing-complete
|
||||
programming language, optimize and JIT compile it in a user-driven way.
|
||||
Next up we'll look into `extending the language with control flow
|
||||
constructs <PythonLangImpl5.html>`_, tackling some interesting LLVM IR
|
||||
issues along the way.
|
||||
|
||||
--------------
|
||||
|
||||
Full Code Listing # {#code}
|
||||
===========================
|
||||
|
||||
Here is the complete code listing for our running example, enhanced with
|
||||
the LLVM JIT and optimizer:
|
||||
|
||||
{% highlight python %} #!/usr/bin/env python
|
||||
|
||||
import re from llvm.core import Module, Constant, Type, Function,
|
||||
Builder, FCMP\_ULT from llvm.ee import ExecutionEngine, TargetData from
|
||||
llvm.passes import FunctionPassManager from llvm.passes import
|
||||
(PASS\_INSTRUCTION\_COMBINING, PASS\_REASSOCIATE, PASS\_GVN,
|
||||
PASS\_CFG\_SIMPLIFICATION)
|
||||
|
||||
Globals
|
||||
-------
|
||||
|
||||
The LLVM module, which holds all the IR code.
|
||||
=============================================
|
||||
|
||||
g\_llvm\_module = Module.new('my cool jit')
|
||||
|
||||
The LLVM instruction builder. Created whenever a new function is entered.
|
||||
=========================================================================
|
||||
|
||||
g\_llvm\_builder = None
|
||||
|
||||
A dictionary that keeps track of which values are defined in the current scope
|
||||
==============================================================================
|
||||
|
||||
and what their LLVM representation is.
|
||||
======================================
|
||||
|
||||
g\_named\_values = {}
|
||||
|
||||
The function optimization passes manager.
|
||||
=========================================
|
||||
|
||||
g\_llvm\_pass\_manager = FunctionPassManager.new(g\_llvm\_module)
|
||||
|
||||
The LLVM execution engine.
|
||||
==========================
|
||||
|
||||
g\_llvm\_executor = ExecutionEngine.new(g\_llvm\_module)
|
||||
|
||||
Lexer
|
||||
-----
|
||||
|
||||
The lexer yields one of these types for each token.
|
||||
===================================================
|
||||
|
||||
class EOFToken(object): pass
|
||||
|
||||
class DefToken(object): pass
|
||||
|
||||
class ExternToken(object): pass
|
||||
|
||||
class IdentifierToken(object): def **init**\ (self, name): self.name =
|
||||
name
|
||||
|
||||
class NumberToken(object): def **init**\ (self, value): self.value =
|
||||
value
|
||||
|
||||
class CharacterToken(object): def **init**\ (self, char): self.char =
|
||||
char def **eq**\ (self, other): return isinstance(other, CharacterToken)
|
||||
and self.char == other.char def **ne**\ (self, other): return not self
|
||||
== other
|
||||
|
||||
Regular expressions that tokens and comments of our language.
|
||||
=============================================================
|
||||
|
||||
REGEX\_NUMBER = re.compile('[0-9]+(?:.[0-9]+)?') REGEX\_IDENTIFIER =
|
||||
re.compile('[a-zA-Z][a-zA-Z0-9]\ *') REGEX\_COMMENT = re.compile('#.*')
|
||||
|
||||
def Tokenize(string): while string: # Skip whitespace. if
|
||||
string[0].isspace(): string = string[1:] continue
|
||||
|
||||
::
|
||||
|
||||
# Run regexes.
|
||||
comment_match = REGEX_COMMENT.match(string)
|
||||
number_match = REGEX_NUMBER.match(string)
|
||||
identifier_match = REGEX_IDENTIFIER.match(string)
|
||||
|
||||
# Check if any of the regexes matched and yield the appropriate result.
|
||||
if comment_match:
|
||||
comment = comment_match.group(0)
|
||||
string = string[len(comment):]
|
||||
elif number_match:
|
||||
number = number_match.group(0)
|
||||
yield NumberToken(float(number))
|
||||
string = string[len(number):]
|
||||
elif identifier_match:
|
||||
identifier = identifier_match.group(0)
|
||||
# Check if we matched a keyword.
|
||||
if identifier == 'def':
|
||||
yield DefToken()
|
||||
elif identifier == 'extern':
|
||||
yield ExternToken()
|
||||
else:
|
||||
yield IdentifierToken(identifier)
|
||||
string = string[len(identifier):]
|
||||
else:
|
||||
# Yield the ASCII value of the unknown character.
|
||||
yield CharacterToken(string[0])
|
||||
string = string[1:]
|
||||
|
||||
yield EOFToken()
|
||||
|
||||
Abstract Syntax Tree (aka Parse Tree)
|
||||
-------------------------------------
|
||||
|
||||
Base class for all expression nodes.
|
||||
====================================
|
||||
|
||||
class ExpressionNode(object): pass
|
||||
|
||||
Expression class for numeric literals like "1.0".
|
||||
=================================================
|
||||
|
||||
class NumberExpressionNode(ExpressionNode):
|
||||
|
||||
def **init**\ (self, value): self.value = value
|
||||
|
||||
def CodeGen(self): return Constant.real(Type.double(), self.value)
|
||||
|
||||
Expression class for referencing a variable, like "a".
|
||||
======================================================
|
||||
|
||||
class VariableExpressionNode(ExpressionNode):
|
||||
|
||||
def **init**\ (self, name): self.name = name
|
||||
|
||||
def CodeGen(self): if self.name in g\_named\_values: return
|
||||
g\_named\_values[self.name] else: raise RuntimeError('Unknown variable
|
||||
name: ' + self.name)
|
||||
|
||||
Expression class for a binary operator.
|
||||
=======================================
|
||||
|
||||
class BinaryOperatorExpressionNode(ExpressionNode):
|
||||
|
||||
def **init**\ (self, operator, left, right): self.operator = operator
|
||||
self.left = left self.right = right
|
||||
|
||||
def CodeGen(self): left = self.left.CodeGen() right =
|
||||
self.right.CodeGen()
|
||||
|
||||
::
|
||||
|
||||
if self.operator == '+':
|
||||
return g_llvm_builder.fadd(left, right, 'addtmp')
|
||||
elif self.operator == '-':
|
||||
return g_llvm_builder.fsub(left, right, 'subtmp')
|
||||
elif self.operator == '*':
|
||||
return g_llvm_builder.fmul(left, right, 'multmp')
|
||||
elif self.operator == '<':
|
||||
result = g_llvm_builder.fcmp(FCMP_ULT, left, right, 'cmptmp')
|
||||
# Convert bool 0 or 1 to double 0.0 or 1.0.
|
||||
return g_llvm_builder.uitofp(result, Type.double(), 'booltmp')
|
||||
else:
|
||||
raise RuntimeError('Unknown binary operator.')
|
||||
|
||||
Expression class for function calls.
|
||||
====================================
|
||||
|
||||
class CallExpressionNode(ExpressionNode):
|
||||
|
||||
def **init**\ (self, callee, args): self.callee = callee self.args =
|
||||
args
|
||||
|
||||
def CodeGen(self): # Look up the name in the global module table. callee
|
||||
= g\_llvm\_module.get\_function\_named(self.callee)
|
||||
|
||||
::
|
||||
|
||||
# Check for argument mismatch error.
|
||||
if len(callee.args) != len(self.args):
|
||||
raise RuntimeError('Incorrect number of arguments passed.')
|
||||
|
||||
arg_values = [i.CodeGen() for i in self.args]
|
||||
|
||||
return g_llvm_builder.call(callee, arg_values, 'calltmp')
|
||||
|
||||
This class represents the "prototype" for a function, which captures its name,
|
||||
==============================================================================
|
||||
|
||||
and its argument names (thus implicitly the number of arguments the function
|
||||
============================================================================
|
||||
|
||||
takes).
|
||||
=======
|
||||
|
||||
class PrototypeNode(object):
|
||||
|
||||
def **init**\ (self, name, args): self.name = name self.args = args
|
||||
|
||||
def CodeGen(self): # Make the function type, eg. double(double,double).
|
||||
funct\_type = Type.function( Type.double(), [Type.double()] \*
|
||||
len(self.args), False)
|
||||
|
||||
::
|
||||
|
||||
function = Function.new(g_llvm_module, funct_type, self.name)
|
||||
|
||||
# If the name conflicted, there was already something with the same name.
|
||||
# If it has a body, don't allow redefinition or reextern.
|
||||
if function.name != self.name:
|
||||
function.delete()
|
||||
function = g_llvm_module.get_function_named(self.name)
|
||||
|
||||
# If the function already has a body, reject this.
|
||||
if not function.is_declaration:
|
||||
raise RuntimeError('Redefinition of function.')
|
||||
|
||||
# If F took a different number of args, reject.
|
||||
if len(callee.args) != len(self.args):
|
||||
raise RuntimeError('Redeclaration of a function with different number '
|
||||
'of args.')
|
||||
|
||||
# Set names for all arguments and add them to the variables symbol table.
|
||||
for arg, arg_name in zip(function.args, self.args):
|
||||
arg.name = arg_name
|
||||
# Add arguments to variable symbol table.
|
||||
g_named_values[arg_name] = arg
|
||||
|
||||
return function
|
||||
|
||||
This class represents a function definition itself.
|
||||
===================================================
|
||||
|
||||
class FunctionNode(object):
|
||||
|
||||
def **init**\ (self, prototype, body): self.prototype = prototype
|
||||
self.body = body
|
||||
|
||||
def CodeGen(self): # Clear scope. g\_named\_values.clear()
|
||||
|
||||
::
|
||||
|
||||
# Create a function object.
|
||||
function = self.prototype.CodeGen()
|
||||
|
||||
# Create a new basic block to start insertion into.
|
||||
block = function.append_basic_block('entry')
|
||||
global g_llvm_builder
|
||||
g_llvm_builder = Builder.new(block)
|
||||
|
||||
# Finish off the function.
|
||||
try:
|
||||
return_value = self.body.CodeGen()
|
||||
g_llvm_builder.ret(return_value)
|
||||
|
||||
# Validate the generated code, checking for consistency.
|
||||
function.verify()
|
||||
|
||||
# Optimize the function.
|
||||
g_llvm_pass_manager.run(function)
|
||||
except:
|
||||
function.delete()
|
||||
raise
|
||||
|
||||
return function
|
||||
|
||||
Parser
|
||||
------
|
||||
|
||||
class Parser(object):
|
||||
|
||||
def **init**\ (self, tokens, binop\_precedence): self.tokens = tokens
|
||||
self.binop\_precedence = binop\_precedence self.Next()
|
||||
|
||||
# Provide a simple token buffer. Parser.current is the current token the
|
||||
# parser is looking at. Parser.Next() reads another token from the lexer
|
||||
and # updates Parser.current with its results. def Next(self):
|
||||
self.current = self.tokens.next()
|
||||
|
||||
# Gets the precedence of the current token, or -1 if the token is not a
|
||||
binary # operator. def GetCurrentTokenPrecedence(self): if
|
||||
isinstance(self.current, CharacterToken): return
|
||||
self.binop\_precedence.get(self.current.char, -1) else: return -1
|
||||
|
||||
# identifierexpr ::= identifier \| identifier '(' expression\* ')' def
|
||||
ParseIdentifierExpr(self): identifier\_name = self.current.name
|
||||
self.Next() # eat identifier.
|
||||
|
||||
::
|
||||
|
||||
if self.current != CharacterToken('('): # Simple variable reference.
|
||||
return VariableExpressionNode(identifier_name)
|
||||
|
||||
# Call.
|
||||
self.Next() # eat '('.
|
||||
args = []
|
||||
if self.current != CharacterToken(')'):
|
||||
while True:
|
||||
args.append(self.ParseExpression())
|
||||
if self.current == CharacterToken(')'):
|
||||
break
|
||||
elif self.current != CharacterToken(','):
|
||||
raise RuntimeError('Expected ")" or "," in argument list.')
|
||||
self.Next()
|
||||
|
||||
self.Next() # eat ')'.
|
||||
return CallExpressionNode(identifier_name, args)
|
||||
|
||||
# numberexpr ::= number def ParseNumberExpr(self): result =
|
||||
NumberExpressionNode(self.current.value) self.Next() # consume the
|
||||
number. return result
|
||||
|
||||
# parenexpr ::= '(' expression ')' def ParseParenExpr(self): self.Next()
|
||||
# eat '('.
|
||||
|
||||
::
|
||||
|
||||
contents = self.ParseExpression()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")".')
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return contents
|
||||
|
||||
# primary ::= identifierexpr \| numberexpr \| parenexpr def
|
||||
ParsePrimary(self): if isinstance(self.current, IdentifierToken): return
|
||||
self.ParseIdentifierExpr() elif isinstance(self.current, NumberToken):
|
||||
return self.ParseNumberExpr() elif self.current == CharacterToken('('):
|
||||
return self.ParseParenExpr() else: raise RuntimeError('Unknown token
|
||||
when expecting an expression.')
|
||||
|
||||
# binoprhs ::= (operator primary)\* def ParseBinOpRHS(self, left,
|
||||
left\_precedence): # If this is a binary operator, find its precedence.
|
||||
while True: precedence = self.GetCurrentTokenPrecedence()
|
||||
|
||||
::
|
||||
|
||||
# If this is a binary operator that binds at least as tightly as the
|
||||
# current one, consume it; otherwise we are done.
|
||||
if precedence < left_precedence:
|
||||
return left
|
||||
|
||||
binary_operator = self.current.char
|
||||
self.Next() # eat the operator.
|
||||
|
||||
# Parse the primary expression after the binary operator.
|
||||
right = self.ParsePrimary()
|
||||
|
||||
# If binary_operator binds less tightly with right than the operator after
|
||||
# right, let the pending operator take right as its left.
|
||||
next_precedence = self.GetCurrentTokenPrecedence()
|
||||
if precedence < next_precedence:
|
||||
right = self.ParseBinOpRHS(right, precedence + 1)
|
||||
|
||||
# Merge left/right.
|
||||
left = BinaryOperatorExpressionNode(binary_operator, left, right)
|
||||
|
||||
# expression ::= primary binoprhs def ParseExpression(self): left =
|
||||
self.ParsePrimary() return self.ParseBinOpRHS(left, 0)
|
||||
|
||||
# prototype ::= id '(' id\* ')' def ParsePrototype(self): if not
|
||||
isinstance(self.current, IdentifierToken): raise RuntimeError('Expected
|
||||
function name in prototype.')
|
||||
|
||||
::
|
||||
|
||||
function_name = self.current.name
|
||||
self.Next() # eat function name.
|
||||
|
||||
if self.current != CharacterToken('('):
|
||||
raise RuntimeError('Expected "(" in prototype.')
|
||||
self.Next() # eat '('.
|
||||
|
||||
arg_names = []
|
||||
while isinstance(self.current, IdentifierToken):
|
||||
arg_names.append(self.current.name)
|
||||
self.Next()
|
||||
|
||||
if self.current != CharacterToken(')'):
|
||||
raise RuntimeError('Expected ")" in prototype.')
|
||||
|
||||
# Success.
|
||||
self.Next() # eat ')'.
|
||||
|
||||
return PrototypeNode(function_name, arg_names)
|
||||
|
||||
# definition ::= 'def' prototype expression def ParseDefinition(self):
|
||||
self.Next() # eat def. proto = self.ParsePrototype() body =
|
||||
self.ParseExpression() return FunctionNode(proto, body)
|
||||
|
||||
# toplevelexpr ::= expression def ParseTopLevelExpr(self): proto =
|
||||
PrototypeNode('', []) return FunctionNode(proto, self.ParseExpression())
|
||||
|
||||
# external ::= 'extern' prototype def ParseExtern(self): self.Next() #
|
||||
eat extern. return self.ParsePrototype()
|
||||
|
||||
# Top-Level parsing def HandleDefinition(self):
|
||||
self.Handle(self.ParseDefinition, 'Read a function definition:')
|
||||
|
||||
def HandleExtern(self): self.Handle(self.ParseExtern, 'Read an extern:')
|
||||
|
||||
def HandleTopLevelExpression(self): try: function =
|
||||
self.ParseTopLevelExpr().CodeGen() result =
|
||||
g\_llvm\_executor.run\_function(function, []) print 'Evaluated to:',
|
||||
result.as\_real(Type.double()) except Exception, e: print 'Error:', e
|
||||
try: self.Next() # Skip for error recovery. except: pass
|
||||
|
||||
def Handle(self, function, message): try: print message,
|
||||
function().CodeGen() except Exception, e: print 'Error:', e try:
|
||||
self.Next() # Skip for error recovery. except: pass
|
||||
|
||||
Main driver code.
|
||||
-----------------
|
||||
|
||||
def main(): # Set up the optimizer pipeline. Start with registering info
|
||||
about how the # target lays out data structures.
|
||||
g\_llvm\_pass\_manager.add(g\_llvm\_executor.target\_data) # Do simple
|
||||
"peephole" optimizations and bit-twiddling optzns.
|
||||
g\_llvm\_pass\_manager.add(PASS\_INSTRUCTION\_COMBINING) # Reassociate
|
||||
expressions. g\_llvm\_pass\_manager.add(PASS\_REASSOCIATE) # Eliminate
|
||||
Common SubExpressions. g\_llvm\_pass\_manager.add(PASS\_GVN) # Simplify
|
||||
the control flow graph (deleting unreachable blocks, etc).
|
||||
g\_llvm\_pass\_manager.add(PASS\_CFG\_SIMPLIFICATION)
|
||||
|
||||
g\_llvm\_pass\_manager.initialize()
|
||||
|
||||
# Install standard binary operators. # 1 is lowest possible precedence.
|
||||
40 is the highest. operator\_precedence = { '<': 10, '+': 20, '-': 20,
|
||||
'\*': 40 }
|
||||
|
||||
# Run the main "interpreter loop". while True: print 'ready>', try: raw
|
||||
= raw\_input() except KeyboardInterrupt: break
|
||||
|
||||
::
|
||||
|
||||
parser = Parser(Tokenize(raw), operator_precedence)
|
||||
while True:
|
||||
# top ::= definition | external | expression | EOF
|
||||
if isinstance(parser.current, EOFToken):
|
||||
break
|
||||
if isinstance(parser.current, DefToken):
|
||||
parser.HandleDefinition()
|
||||
elif isinstance(parser.current, ExternToken):
|
||||
parser.HandleExtern()
|
||||
else:
|
||||
parser.HandleTopLevelExpression()
|
||||
|
||||
# Print out all of the generated code. print '', g\_llvm\_module
|
||||
|
||||
if **name** == '**main**\ ': main() {% endhighlight %}
|
||||
|
||||
--------------
|
||||
|
||||
**`Next: Extending the language: control flow <PythonLangImpl5.html>`_**
|
||||
File diff suppressed because it is too large
Load diff
1339
docs/source/doc/kaleidoscope/PythonLangImpl5.rst
Normal file
1339
docs/source/doc/kaleidoscope/PythonLangImpl5.rst
Normal file
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
1361
docs/source/doc/kaleidoscope/PythonLangImpl6.rst
Normal file
1361
docs/source/doc/kaleidoscope/PythonLangImpl6.rst
Normal file
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
1574
docs/source/doc/kaleidoscope/PythonLangImpl7.rst
Normal file
1574
docs/source/doc/kaleidoscope/PythonLangImpl7.rst
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -1,275 +0,0 @@
|
|||
---
|
||||
layout: page
|
||||
title: "Kaleidoscope: Chapter 8"
|
||||
---
|
||||
|
||||
# Conclusion and other useful LLVM tidbits
|
||||
|
||||
Written by [Chris Lattner](mailto:sabre@nondot.org)
|
||||
|
||||
**Chapter 8**
|
||||
|
||||
|
||||
* This will become a table of contents (this text will be scraped).
|
||||
{:toc}
|
||||
|
||||
# Tutorial Conclusion # {#conclusion}
|
||||
|
||||
Welcome to the the final chapter of the
|
||||
[Implementing a language with LLVM](http://www.llvm.org/docs/tutorial/index.html)
|
||||
tutorial. In the course of this tutorial, we have grown
|
||||
our little Kaleidoscope language from being a useless toy, to being a
|
||||
semi-interesting (but probably still useless) toy. :)
|
||||
|
||||
It is interesting to see how far we've come, and how little code it has
|
||||
taken. We built the entire lexer, parser, AST, code generator, and an
|
||||
interactive run-loop (with a JIT!) by-hand in under 540 lines of
|
||||
(non-comment/non-blank) code.
|
||||
|
||||
Our little language supports a couple of interesting features: it supports
|
||||
user defined binary and unary operators, it uses JIT compilation for immediate
|
||||
evaluation, and it supports a few control flow constructs with SSA construction.
|
||||
|
||||
|
||||
Part of the idea of this tutorial was to show you how easy and fun it can be
|
||||
to define, build, and play with languages. Building a compiler need not be a
|
||||
scary or mystical process! Now that you've seen some of the basics, I strongly
|
||||
encourage you to take the code and hack on it. For example, try adding:
|
||||
|
||||
|
||||
|
||||
* **global variables** -- While global variables have questional value in
|
||||
modern software engineering, they are often useful when putting together quick
|
||||
little hacks like the Kaleidoscope compiler itself. Fortunately, our current
|
||||
setup makes it very easy to add global variables: just have value lookup check
|
||||
to see if an unresolved variable is in the global variable symbol table before
|
||||
rejecting it. To create a new global variable, make an instance of the LLVM
|
||||
`GlobalVariable` class.
|
||||
|
||||
* **typed variables** -- Kaleidoscope currently only supports variables of
|
||||
type double. This gives the language a very nice elegance, because only
|
||||
supporting one type means that you never have to specify types. Different
|
||||
languages have different ways of handling this. The easiest way is to require
|
||||
the user to specify types for every variable definition, and record the type
|
||||
of the variable in the symbol table along with its Value\*.
|
||||
|
||||
* **arrays, structs, vectors, etc** -- Once you add types, you can start
|
||||
extending the type system in all sorts of interesting ways. Simple arrays are
|
||||
very easy and are quite useful for many different applications. Adding them is
|
||||
mostly an exercise in learning how the LLVM
|
||||
[getelementptr](http://www.llvm.org/docs/LangRef.html#i_getelementptr)
|
||||
instruction works: it is so nifty/unconventional, it
|
||||
[has its own FAQ](http://www.llvm.org/docs/GetElementPtr.html)! If you
|
||||
add support for recursive types (e.g. linked lists), make sure to read the
|
||||
[section in the LLVM Programmer's Manual](http://www.llvm.org/docs/ProgrammersManual.html#TypeResolve)
|
||||
that describes how to construct them.
|
||||
|
||||
* **standard runtime** -- Our current language allows the user to access
|
||||
arbitrary external functions, and we use it for things like "putchard". As you
|
||||
extend the language to add higher-level constructs, often these constructs make
|
||||
the most sense if they are lowered to calls into a language-supplied runtime.
|
||||
For example, if you add hash tables to the language, it would probably make
|
||||
sense to add the routines to a runtime, instead of inlining them all the way.
|
||||
|
||||
* **memory management** -- Currently we can only access the stack in
|
||||
Kaleidoscope. It would also be useful to be able to allocate heap memory,
|
||||
either with calls to the standard libc malloc/free interface or with a garbage
|
||||
collector. If you would like to use garbage collection, note that LLVM fully
|
||||
supports
|
||||
[Accurate Garbage Collection](http://www.llvm.org/docs/GarbageCollection.html)
|
||||
including algorithms that move objects and need to
|
||||
scan/update the stack.
|
||||
|
||||
* **debugger support** -- LLVM supports generation of
|
||||
[DWARF Debug info](http://www.llvm.org/docs/SourceLevelDebugging.html)
|
||||
which is understood by common debuggers like GDB. Adding support for debug
|
||||
info is fairly straightforward. The best way to understand it is to compile
|
||||
some C/C++ code with "`llvm-gcc -g -O0`" and taking a look at
|
||||
what it produces.
|
||||
|
||||
* **exception handling support** - LLVM supports generation of
|
||||
[zero cost exceptions](http://www.llvm.org/docs/ExceptionHandling.html)
|
||||
which interoperate with code compiled in other languages. You could also
|
||||
generate code by implicitly making every function return an error value and
|
||||
checking it. You could also make explicit use of setjmp/longjmp. There are
|
||||
many different ways to go here.
|
||||
|
||||
* **object orientation, generics, database access, complex numbers,
|
||||
geometric programming, ...** -- Really, there is
|
||||
no end of crazy features that you can add to the language.
|
||||
|
||||
* **unusual domains** -- We've been talking about applying LLVM to a domain
|
||||
that many people are interested in: building a compiler for a specific language.
|
||||
However, there are many other domains that can use compiler technology that are
|
||||
not typically considered. For example, LLVM has been used to implement OpenGL
|
||||
graphics acceleration, translate C++ code to ActionScript, and many other
|
||||
cute and clever things. Maybe you will be the first to JIT compile a regular
|
||||
expression interpreter into native code with LLVM?
|
||||
|
||||
|
||||
Have fun - try doing something crazy and unusual. Building a language like
|
||||
everyone else always has, is much less fun than trying something a little crazy
|
||||
or off the wall and seeing how it turns out. If you get stuck or want to talk
|
||||
about it, feel free to email the
|
||||
[llvmdev mailing list](http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev):
|
||||
it has lots of people who are interested in languages and are often
|
||||
willing to help out.
|
||||
|
||||
|
||||
Before we end this tutorial, I want to talk about some "tips and tricks" for
|
||||
generating LLVM IR. These are some of the more subtle things that may not be
|
||||
obvious, but are very useful if you want to take advantage of LLVM's
|
||||
capabilities.
|
||||
|
||||
* * *
|
||||
|
||||
# Properties of the LLVM IR # {#llvmirproperties}
|
||||
|
||||
We have a couple common questions about code in the LLVM IR form - let's just
|
||||
get these out of the way right now, shall we?
|
||||
|
||||
## Target Independence ## {#targetindep}
|
||||
|
||||
Kaleidoscope is an example of a "portable language": any program written in
|
||||
Kaleidoscope will work the same way on any target that it runs on. Many other
|
||||
languages have this property, e.g. LISP, Java, Haskell, Javascript, Python, etc.
|
||||
(note that while these languages are portable, not all their libraries are).
|
||||
|
||||
One nice aspect of LLVM is that it is often capable of preserving target
|
||||
independence in the IR: you can take the LLVM IR for a Kaleidoscope-compiled
|
||||
program and run it on any target that LLVM supports, even emitting C code and
|
||||
compiling that on targets that LLVM doesn't support natively. You can trivially
|
||||
tell that the Kaleidoscope compiler generates target-independent code because it
|
||||
never queries for any target-specific information when generating code.
|
||||
|
||||
The fact that LLVM provides a compact, target-independent, representation for
|
||||
code gets a lot of people excited. Unfortunately, these people are usually
|
||||
thinking about C or a language from the C family when they are asking questions
|
||||
about language portability. I say "unfortunately", because there is really no
|
||||
way to make (fully general) C code portable, other than shipping the source code
|
||||
around (and of course, C source code is not actually portable in general
|
||||
either - ever port a really old application from 32- to 64-bits?).
|
||||
|
||||
The problem with C (again, in its full generality) is that it is heavily
|
||||
laden with target specific assumptions. As one simple example, the preprocessor
|
||||
often destructively removes target-independence from the code when it processes
|
||||
the input text:
|
||||
|
||||
|
||||
{% highlight c %}
|
||||
#ifdef __i386__
|
||||
int X = 1;
|
||||
#else
|
||||
int X = 42;
|
||||
#endif
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
While it is possible to engineer more and more complex solutions to problems
|
||||
like this, it cannot be solved in full generality in a way that is better than
|
||||
shipping the actual source code.
|
||||
|
||||
That said, there are interesting subsets of C that can be made portable. If
|
||||
you are willing to fix primitive types to a fixed size (say int = 32-bits,
|
||||
and long = 64-bits), don't care about ABI compatibility with existing binaries,
|
||||
and are willing to give up some other minor features, you can have portable
|
||||
code. This can make sense for specialized domains such as an
|
||||
in-kernel language.
|
||||
|
||||
|
||||
## Safety Guarantees ## {#safety}
|
||||
|
||||
Many of the languages above are also "safe" languages: it is impossible for
|
||||
a program written in Java to corrupt its address space and crash the process
|
||||
(assuming the JVM has no bugs).
|
||||
Safety is an interesting property that requires a combination of language
|
||||
design, runtime support, and often operating system support.
|
||||
|
||||
It is certainly possible to implement a safe language in LLVM, but LLVM IR
|
||||
does not itself guarantee safety. The LLVM IR allows unsafe pointer casts,
|
||||
use after free bugs, buffer over-runs, and a variety of other problems. Safety
|
||||
needs to be implemented as a layer on top of LLVM and, conveniently, several
|
||||
groups have investigated this. Ask on the
|
||||
[llvmdev mailing list](http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev)
|
||||
if you are interested in more details.
|
||||
|
||||
|
||||
## Language-Specific Optimizations ## {#langspecific}
|
||||
|
||||
One thing about LLVM that turns off many people is that it does not solve all
|
||||
the world's problems in one system (sorry 'world hunger', someone else will have
|
||||
to solve you some other day). One specific complaint is that people perceive
|
||||
LLVM as being incapable of performing high-level language-specific optimization:
|
||||
LLVM "loses too much information".
|
||||
|
||||
Unfortunately, this is really not the place to give you a full and unified
|
||||
version of "Chris Lattner's theory of compiler design". Instead, I'll make a
|
||||
few observations:
|
||||
|
||||
First, you're right that LLVM does lose information. For example, as of this
|
||||
writing, there is no way to distinguish in the LLVM IR whether an SSA-value came
|
||||
from a C "int" or a C "long" on an ILP32 machine (other than debug info). Both
|
||||
get compiled down to an 'i32' value and the information about what it came from
|
||||
is lost. The more general issue here, is that the LLVM type system uses
|
||||
"structural equivalence" instead of "name equivalence". Another place this
|
||||
surprises people is if you have two types in a high-level language that have the
|
||||
same structure (e.g. two different structs that have a single int field): these
|
||||
types will compile down into a single LLVM type and it will be impossible to
|
||||
tell what it came from.
|
||||
|
||||
Second, while LLVM does lose information, LLVM is not a fixed target: we
|
||||
continue to enhance and improve it in many different ways. In addition to
|
||||
adding new features (LLVM did not always support exceptions or debug info), we
|
||||
also extend the IR to capture important information for optimization (e.g.
|
||||
whether an argument is sign or zero extended, information about pointers
|
||||
aliasing, etc). Many of the enhancements are user-driven: people want LLVM to
|
||||
include some specific feature, so they go ahead and extend it.
|
||||
|
||||
Third, it is *possible and easy* to add language-specific
|
||||
optimizations, and you have a number of choices in how to do it. As one trivial
|
||||
example, it is easy to add language-specific optimization passes that
|
||||
"know" things about code compiled for a language. In the case of the C family,
|
||||
there is an optimization pass that "knows" about the standard C library
|
||||
functions. If you call "exit(0)" in main(), it knows that it is safe to
|
||||
optimize that into "return 0;" because C specifies what the 'exit'
|
||||
function does.
|
||||
|
||||
In addition to simple library knowledge, it is possible to embed a variety of
|
||||
other language-specific information into the LLVM IR. If you have a specific
|
||||
need and run into a wall, please bring the topic up on the llvmdev list. At the
|
||||
very worst, you can always treat LLVM as if it were a "dumb code generator" and
|
||||
implement the high-level optimizations you desire in your front-end, on the
|
||||
language-specific AST.
|
||||
|
||||
* * *
|
||||
|
||||
# Tips and Tricks # {#tipsandtricks}
|
||||
|
||||
There is a variety of useful tips and tricks that you come to know after
|
||||
working on/with LLVM that aren't obvious at first glance. Instead of letting
|
||||
everyone rediscover them, this section talks about some of these issues.
|
||||
|
||||
## Implementing portable offsetof/sizeof ## {#offsetofsizeof}
|
||||
|
||||
One interesting thing that comes up, if you are trying to keep the code
|
||||
generated by your compiler "target independent", is that you often need to know
|
||||
the size of some LLVM type or the offset of some field in an llvm structure.
|
||||
For example, you might need to pass the size of a type into a function that
|
||||
allocates memory.
|
||||
|
||||
Unfortunately, this can vary widely across targets: for example the width of
|
||||
a pointer is trivially target-specific. However, there is a
|
||||
[clever way to use the getelementptr instruction](http://nondot.org/sabre/LLVMNotes/SizeOf-OffsetOf-VariableSizedStructs.txt)
|
||||
that allows you to compute this in a portable way.
|
||||
|
||||
## Garbage Collected Stack Frames ## {#gcstack}
|
||||
|
||||
Some languages want to explicitly manage their stack frames, often so that
|
||||
they are garbage collected or to allow easy implementation of closures. There
|
||||
are often better ways to implement these features than explicit stack frames,
|
||||
but [LLVM does support them](http://nondot.org/sabre/LLVMNotes/ExplicitlyManagedStackFrames.txt),
|
||||
if you want. It requires your front-end to convert the code into
|
||||
[Continuation Passing Style](http://en.wikipedia.org/wiki/Continuation-passing_style)
|
||||
and the use of tail calls (which LLVM also supports).
|
||||
|
||||
|
||||
286
docs/source/doc/kaleidoscope/PythonLangImpl8.rst
Normal file
286
docs/source/doc/kaleidoscope/PythonLangImpl8.rst
Normal file
|
|
@ -0,0 +1,286 @@
|
|||
+------------------------------------+
|
||||
| layout: page |
|
||||
+------------------------------------+
|
||||
| title: "Kaleidoscope: Chapter 8" |
|
||||
+------------------------------------+
|
||||
|
||||
Conclusion and other useful LLVM tidbits
|
||||
========================================
|
||||
|
||||
Written by `Chris Lattner <mailto:sabre@nondot.org>`_
|
||||
|
||||
**Chapter 8**
|
||||
|
||||
- This will become a table of contents (this text will be scraped).
|
||||
{:toc}
|
||||
|
||||
Tutorial Conclusion # {#conclusion}
|
||||
===================================
|
||||
|
||||
Welcome to the the final chapter of the `Implementing a language with
|
||||
LLVM <http://www.llvm.org/docs/tutorial/index.html>`_ tutorial. In the
|
||||
course of this tutorial, we have grown our little Kaleidoscope language
|
||||
from being a useless toy, to being a semi-interesting (but probably
|
||||
still useless) toy. :)
|
||||
|
||||
It is interesting to see how far we've come, and how little code it has
|
||||
taken. We built the entire lexer, parser, AST, code generator, and an
|
||||
interactive run-loop (with a JIT!) by-hand in under 540 lines of
|
||||
(non-comment/non-blank) code.
|
||||
|
||||
Our little language supports a couple of interesting features: it
|
||||
supports user defined binary and unary operators, it uses JIT
|
||||
compilation for immediate evaluation, and it supports a few control flow
|
||||
constructs with SSA construction.
|
||||
|
||||
Part of the idea of this tutorial was to show you how easy and fun it
|
||||
can be to define, build, and play with languages. Building a compiler
|
||||
need not be a scary or mystical process! Now that you've seen some of
|
||||
the basics, I strongly encourage you to take the code and hack on it.
|
||||
For example, try adding:
|
||||
|
||||
- **global variables** -- While global variables have questional value
|
||||
in modern software engineering, they are often useful when putting
|
||||
together quick little hacks like the Kaleidoscope compiler itself.
|
||||
Fortunately, our current setup makes it very easy to add global
|
||||
variables: just have value lookup check to see if an unresolved
|
||||
variable is in the global variable symbol table before rejecting it.
|
||||
To create a new global variable, make an instance of the LLVM
|
||||
``GlobalVariable`` class.
|
||||
|
||||
- **typed variables** -- Kaleidoscope currently only supports variables
|
||||
of type double. This gives the language a very nice elegance, because
|
||||
only supporting one type means that you never have to specify types.
|
||||
Different languages have different ways of handling this. The easiest
|
||||
way is to require the user to specify types for every variable
|
||||
definition, and record the type of the variable in the symbol table
|
||||
along with its Value\*.
|
||||
|
||||
- **arrays, structs, vectors, etc** -- Once you add types, you can
|
||||
start extending the type system in all sorts of interesting ways.
|
||||
Simple arrays are very easy and are quite useful for many different
|
||||
applications. Adding them is mostly an exercise in learning how the
|
||||
LLVM
|
||||
`getelementptr <http://www.llvm.org/docs/LangRef.html#i_getelementptr>`_
|
||||
instruction works: it is so nifty/unconventional, it `has its own
|
||||
FAQ <http://www.llvm.org/docs/GetElementPtr.html>`_! If you add
|
||||
support for recursive types (e.g. linked lists), make sure to read
|
||||
the `section in the LLVM Programmer's
|
||||
Manual <http://www.llvm.org/docs/ProgrammersManual.html#TypeResolve>`_
|
||||
that describes how to construct them.
|
||||
|
||||
- **standard runtime** -- Our current language allows the user to
|
||||
access arbitrary external functions, and we use it for things like
|
||||
"putchard". As you extend the language to add higher-level
|
||||
constructs, often these constructs make the most sense if they are
|
||||
lowered to calls into a language-supplied runtime. For example, if
|
||||
you add hash tables to the language, it would probably make sense to
|
||||
add the routines to a runtime, instead of inlining them all the way.
|
||||
|
||||
- **memory management** -- Currently we can only access the stack in
|
||||
Kaleidoscope. It would also be useful to be able to allocate heap
|
||||
memory, either with calls to the standard libc malloc/free interface
|
||||
or with a garbage collector. If you would like to use garbage
|
||||
collection, note that LLVM fully supports `Accurate Garbage
|
||||
Collection <http://www.llvm.org/docs/GarbageCollection.html>`_
|
||||
including algorithms that move objects and need to scan/update the
|
||||
stack.
|
||||
|
||||
- **debugger support** -- LLVM supports generation of `DWARF Debug
|
||||
info <http://www.llvm.org/docs/SourceLevelDebugging.html>`_ which is
|
||||
understood by common debuggers like GDB. Adding support for debug
|
||||
info is fairly straightforward. The best way to understand it is to
|
||||
compile some C/C++ code with "``llvm-gcc -g -O0``\ " and taking a
|
||||
look at what it produces.
|
||||
|
||||
- **exception handling support** - LLVM supports generation of `zero
|
||||
cost exceptions <http://www.llvm.org/docs/ExceptionHandling.html>`_
|
||||
which interoperate with code compiled in other languages. You could
|
||||
also generate code by implicitly making every function return an
|
||||
error value and checking it. You could also make explicit use of
|
||||
setjmp/longjmp. There are many different ways to go here.
|
||||
|
||||
- **object orientation, generics, database access, complex numbers,
|
||||
geometric programming, ...** -- Really, there is no end of crazy
|
||||
features that you can add to the language.
|
||||
|
||||
- **unusual domains** -- We've been talking about applying LLVM to a
|
||||
domain that many people are interested in: building a compiler for a
|
||||
specific language. However, there are many other domains that can use
|
||||
compiler technology that are not typically considered. For example,
|
||||
LLVM has been used to implement OpenGL graphics acceleration,
|
||||
translate C++ code to ActionScript, and many other cute and clever
|
||||
things. Maybe you will be the first to JIT compile a regular
|
||||
expression interpreter into native code with LLVM?
|
||||
|
||||
Have fun - try doing something crazy and unusual. Building a language
|
||||
like everyone else always has, is much less fun than trying something a
|
||||
little crazy or off the wall and seeing how it turns out. If you get
|
||||
stuck or want to talk about it, feel free to email the `llvmdev mailing
|
||||
list <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>`_: it has lots
|
||||
of people who are interested in languages and are often willing to help
|
||||
out.
|
||||
|
||||
Before we end this tutorial, I want to talk about some "tips and tricks"
|
||||
for generating LLVM IR. These are some of the more subtle things that
|
||||
may not be obvious, but are very useful if you want to take advantage of
|
||||
LLVM's capabilities.
|
||||
|
||||
--------------
|
||||
|
||||
Properties of the LLVM IR # {#llvmirproperties}
|
||||
===============================================
|
||||
|
||||
We have a couple common questions about code in the LLVM IR form - let's
|
||||
just get these out of the way right now, shall we?
|
||||
|
||||
Target Independence ## {#targetindep}
|
||||
-------------------------------------
|
||||
|
||||
Kaleidoscope is an example of a "portable language": any program written
|
||||
in Kaleidoscope will work the same way on any target that it runs on.
|
||||
Many other languages have this property, e.g. LISP, Java, Haskell,
|
||||
Javascript, Python, etc. (note that while these languages are portable,
|
||||
not all their libraries are).
|
||||
|
||||
One nice aspect of LLVM is that it is often capable of preserving target
|
||||
independence in the IR: you can take the LLVM IR for a
|
||||
Kaleidoscope-compiled program and run it on any target that LLVM
|
||||
supports, even emitting C code and compiling that on targets that LLVM
|
||||
doesn't support natively. You can trivially tell that the Kaleidoscope
|
||||
compiler generates target-independent code because it never queries for
|
||||
any target-specific information when generating code.
|
||||
|
||||
The fact that LLVM provides a compact, target-independent,
|
||||
representation for code gets a lot of people excited. Unfortunately,
|
||||
these people are usually thinking about C or a language from the C
|
||||
family when they are asking questions about language portability. I say
|
||||
"unfortunately", because there is really no way to make (fully general)
|
||||
C code portable, other than shipping the source code around (and of
|
||||
course, C source code is not actually portable in general either - ever
|
||||
port a really old application from 32- to 64-bits?).
|
||||
|
||||
The problem with C (again, in its full generality) is that it is heavily
|
||||
laden with target specific assumptions. As one simple example, the
|
||||
preprocessor often destructively removes target-independence from the
|
||||
code when it processes the input text:
|
||||
|
||||
{% highlight c %} #ifdef **i386** int X = 1; #else int X = 42; #endif {%
|
||||
endhighlight %}
|
||||
|
||||
While it is possible to engineer more and more complex solutions to
|
||||
problems like this, it cannot be solved in full generality in a way that
|
||||
is better than shipping the actual source code.
|
||||
|
||||
That said, there are interesting subsets of C that can be made portable.
|
||||
If you are willing to fix primitive types to a fixed size (say int =
|
||||
32-bits, and long = 64-bits), don't care about ABI compatibility with
|
||||
existing binaries, and are willing to give up some other minor features,
|
||||
you can have portable code. This can make sense for specialized domains
|
||||
such as an in-kernel language.
|
||||
|
||||
Safety Guarantees ## {#safety}
|
||||
------------------------------
|
||||
|
||||
Many of the languages above are also "safe" languages: it is impossible
|
||||
for a program written in Java to corrupt its address space and crash the
|
||||
process (assuming the JVM has no bugs). Safety is an interesting
|
||||
property that requires a combination of language design, runtime
|
||||
support, and often operating system support.
|
||||
|
||||
It is certainly possible to implement a safe language in LLVM, but LLVM
|
||||
IR does not itself guarantee safety. The LLVM IR allows unsafe pointer
|
||||
casts, use after free bugs, buffer over-runs, and a variety of other
|
||||
problems. Safety needs to be implemented as a layer on top of LLVM and,
|
||||
conveniently, several groups have investigated this. Ask on the `llvmdev
|
||||
mailing list <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>`_ if
|
||||
you are interested in more details.
|
||||
|
||||
Language-Specific Optimizations ## {#langspecific}
|
||||
--------------------------------------------------
|
||||
|
||||
One thing about LLVM that turns off many people is that it does not
|
||||
solve all the world's problems in one system (sorry 'world hunger',
|
||||
someone else will have to solve you some other day). One specific
|
||||
complaint is that people perceive LLVM as being incapable of performing
|
||||
high-level language-specific optimization: LLVM "loses too much
|
||||
information".
|
||||
|
||||
Unfortunately, this is really not the place to give you a full and
|
||||
unified version of "Chris Lattner's theory of compiler design". Instead,
|
||||
I'll make a few observations:
|
||||
|
||||
First, you're right that LLVM does lose information. For example, as of
|
||||
this writing, there is no way to distinguish in the LLVM IR whether an
|
||||
SSA-value came from a C "int" or a C "long" on an ILP32 machine (other
|
||||
than debug info). Both get compiled down to an 'i32' value and the
|
||||
information about what it came from is lost. The more general issue
|
||||
here, is that the LLVM type system uses "structural equivalence" instead
|
||||
of "name equivalence". Another place this surprises people is if you
|
||||
have two types in a high-level language that have the same structure
|
||||
(e.g. two different structs that have a single int field): these types
|
||||
will compile down into a single LLVM type and it will be impossible to
|
||||
tell what it came from.
|
||||
|
||||
Second, while LLVM does lose information, LLVM is not a fixed target: we
|
||||
continue to enhance and improve it in many different ways. In addition
|
||||
to adding new features (LLVM did not always support exceptions or debug
|
||||
info), we also extend the IR to capture important information for
|
||||
optimization (e.g. whether an argument is sign or zero extended,
|
||||
information about pointers aliasing, etc). Many of the enhancements are
|
||||
user-driven: people want LLVM to include some specific feature, so they
|
||||
go ahead and extend it.
|
||||
|
||||
Third, it is *possible and easy* to add language-specific optimizations,
|
||||
and you have a number of choices in how to do it. As one trivial
|
||||
example, it is easy to add language-specific optimization passes that
|
||||
"know" things about code compiled for a language. In the case of the C
|
||||
family, there is an optimization pass that "knows" about the standard C
|
||||
library functions. If you call "exit(0)" in main(), it knows that it is
|
||||
safe to optimize that into "return 0;" because C specifies what the
|
||||
'exit' function does.
|
||||
|
||||
In addition to simple library knowledge, it is possible to embed a
|
||||
variety of other language-specific information into the LLVM IR. If you
|
||||
have a specific need and run into a wall, please bring the topic up on
|
||||
the llvmdev list. At the very worst, you can always treat LLVM as if it
|
||||
were a "dumb code generator" and implement the high-level optimizations
|
||||
you desire in your front-end, on the language-specific AST.
|
||||
|
||||
--------------
|
||||
|
||||
Tips and Tricks # {#tipsandtricks}
|
||||
==================================
|
||||
|
||||
There is a variety of useful tips and tricks that you come to know after
|
||||
working on/with LLVM that aren't obvious at first glance. Instead of
|
||||
letting everyone rediscover them, this section talks about some of these
|
||||
issues.
|
||||
|
||||
Implementing portable offsetof/sizeof ## {#offsetofsizeof}
|
||||
----------------------------------------------------------
|
||||
|
||||
One interesting thing that comes up, if you are trying to keep the code
|
||||
generated by your compiler "target independent", is that you often need
|
||||
to know the size of some LLVM type or the offset of some field in an
|
||||
llvm structure. For example, you might need to pass the size of a type
|
||||
into a function that allocates memory.
|
||||
|
||||
Unfortunately, this can vary widely across targets: for example the
|
||||
width of a pointer is trivially target-specific. However, there is a
|
||||
`clever way to use the getelementptr
|
||||
instruction <http://nondot.org/sabre/LLVMNotes/SizeOf-OffsetOf-VariableSizedStructs.txt>`_
|
||||
that allows you to compute this in a portable way.
|
||||
|
||||
Garbage Collected Stack Frames ## {#gcstack}
|
||||
--------------------------------------------
|
||||
|
||||
Some languages want to explicitly manage their stack frames, often so
|
||||
that they are garbage collected or to allow easy implementation of
|
||||
closures. There are often better ways to implement these features than
|
||||
explicit stack frames, but `LLVM does support
|
||||
them <http://nondot.org/sabre/LLVMNotes/ExplicitlyManagedStackFrames.txt>`_,
|
||||
if you want. It requires your front-end to convert the code into
|
||||
`Continuation Passing
|
||||
Style <http://en.wikipedia.org/wiki/Continuation-passing_style>`_ and
|
||||
the use of tail calls (which LLVM also supports).
|
||||
Loading…
Add table
Add a link
Reference in a new issue