Author: David Beazley (beazley@cs.uchicago.edu)
June 2, 2002
With the release of SWIG-1.3.12, I thought I'd take a few moments of everyone's time to talk about the past, the present, and the future of SWIG development. I'm really quite excited about the current release because I think it represents a huge turning point in SWIG's development. Furthermore, it is only the beginning of bigger and better things to come. However, we definitely need your help.
To put a little perspective on the discussion, I'd start with a few development statistics. In the last 12 months, there have been over 300 entries added to the CHANGES log and over 4000 CVS commits. Although that may not sound like a lot compared to a huge software project, it is significant in the context of SWIG. As a point of comparison, there has been more SWIG development this year than in any other year of the project and more than in the previous three years combined. This even includes the first few years of development in which there was also a lot of activity. Furthermore, many of the recent changes have been extremely non-trivial (e.g., templates, namespaces, type system, operators, etc.). As a result, SWIG is more capable than I ever imagined possible.
Regrettably, I must admit that I've been a little negligent in discussing the roadmap for where I thought this flurry of SWIG development was actually headed. In part, this is because I've been buried in work. However, the real reason is that I didn't really know where we were going---except that in a time far far far away in the future, we might arrive at some kind of "stable" release with a version number other than "1.3.x". Needless to say, that's not a very compelling story.
That said, I've spent a lot of time thinking about SWIG and trying to wrap my brain around it. Specifically, just what is (or should be) the primary focus of this project and what are we really trying to do? That's what the rest of this message is about.
The first version of SWIG was written in 1995. The original system was developed to help with some software wrapping problems I encountered while writing molecular dynamics software at Los Alamos. Later that year, I became interested in extending the wrapper generator to support other scripting languages so it was rewritten in C++ and modified with multiple backends (Tcl, Perl, and Guile). This led to a first public release in February, 1996. Feedback from this release led to a series of enhancements and the release of SWIG 1.0 in September 1996. Throughout this process, my intent was to create a tool that I would want to use---I never viewed the project as an CS experiment in programming languages or software engineering.
To deal with the design problem, there were at least four attempts to completely rewrite SWIG, some of which were attempted in parallel with the work on SWIG-1.1. Unfortunately, none of these were ever completed. The primary problem was a strong "second system" effect and a desire to make SWIG do everything that one might conceivably want to do with a wrapper generator (somehow). Clearly, this was a recipe for disaster. In fact, all such attempts to rewrite SWIG were eventually abandoned several years ago. In hindsight, I think the real problem was that these rewrite efforts focused far too much attention on implementation technique rather than principles. In short, the failure of these efforts was due to a lack of clarity in understanding how SWIG ought to work (regardless of how it was actually implemented).
Having languished for several years, the SWIG1.1p5 release had a growing pile of maintenance issues. It didn't work for newer versions of certain language modules and a lot of minor bug reports and feature requests had been building up. With a lot of help from Loic Dachary and Thien-Thi Nguyen, we put together the 1.3a1 release (February, 2000). This was mostly a bug fix release to 1.1p5 except that the preprocessor module from SWIG1.2 was added and a lot of minor enhancements were added.
For the next six months, a massive effort was made to rewrite all of SWIG's internal data structures (strings, lists, hashes, etc.). This work was all going on underneath the covers while we tried to keep SWIG in an operational state. The primary focus of this work was really one of cleanup. Having given up on a total rewrite, perhaps we could settle with making the implementation incrementally better than before. In addition this, Matthias Koppe jumped on board to reimplement the Guile module and to make other improvements to the system.
An important aspect of these releases was that many parts of the system not directly related to wrapper code generation were removed. This included the documentation system and Objective-C support. These were not removed because they weren't useful. Instead, the documentation system was removed because it accounted for nearly half of the special SWIG directives, yet had no direct bearing on what SWIG actually did. Obective-C support was removed because it was tangled with C++ support in a way that was very difficult to understand and maintain. The removal of these features was seen as a way to vastly simplify cleanup--and to buy some time where we could rethink their role in a future release.
The general goal of this development could best be described as an attempt to reduce SWIG to an easily described set of general "ideas" about how it should operate. Special SWIG directives have been eliminated or combined with others. Different parts of the system have been better integrated. Features not directly related to wrapper code generation have been removed and the system has become more focused. Changes in internal data structures and APIs have allowed SWIG to capture more information from interface files and to resolve hard corner cases. More often than not, these are things that you never notice unless you are an old user and you suddenly realize that a problem you had several years back has disappeared.
Along with the destruction of old code, this work has quietly introduced a new core--the most significant features of which are a new C++ type system and multi-pass compilation. More importantly, this work has really tried to provide a more principled foundation for future SWIG development. However, just what is this "more principled foundation?"
To be honest, I don't think that the emphasis on types is entirely obvious. In fact, a great deal of work in past SWIG rewrites has focused on the problem of C++ parsing. For instance, modifying the parser to handle more advanced C++ code or representing parse trees as XML. Furthermore, if one looks at the SWIG mailing list, you can find a *lot* of messages related to issues of C++ parsing whereas almost no one ever talks about types (well, other than typemaps). Even other wrapper generation tools seems to spend a lot of time dealing with the parsing issue. Although parsing is clearly important, I don't think it has ever been the real problem in SWIG. This is because even though a parser can tell you what's in a header file, it doesn't tell you anything about how the different pieces of the system behave or how they might interact. To do that, you need to do a lot more than just parsing--and that's really the whole point.
Although earlier 1.3 releases have made big improvements to the type system, SWIG-1.3.12 is the first release that really tries to tackle the type-system issue in a major way. We have patched nearly all remaining holes in the type system and we have added full support for C++ namespaces. Not only that, we have completely reimplemented C++ template support in a way that supports templates, member templates, and template specialization. Luigi and I are currently using this to work on proper SWIG library support for parts of the C++ standard library and the Standard Template Library (STL). Although some crusty C programmers (present company excepted), might balk at such apparent insanity, this work has impacted all parts of SWIG at all levels. Even a variety of subtle errors in C support have been fixed by this work.
In addition to the type system work, SWIG-1.3.12 contains continued reduction in the implementation. Directives have been removed, refined, renamed, or consolidated. We're still narrowing the focus of the system and working towards some kind of convergence. "Convergence to what?!?", you ask.
Typemaps. Typemaps are rules that define the process by which data is converted between languages. They are fully integrated with the C++ type system and they are applied using a type-based pattern matching mechanism. All data conversion SWIG is defined by typemaps and is fully reconfigurable.
Declaration annotation. There are special directives that modify the wrapping behavior of individual declarations. Declarations can be selectively identified and decorated with arbitrary attributes that affect wrapper generation. Like typemaps, declaration matching is fully integrated with the C++ type system. Almost all SWIG customization directives are a form of declaration annotation.
Class extension. The ability to extend classes and structures with new methods/attributes when building wrappers. Classes are part of the type system so class extension is naturally integrated with the C++ type system as well (big surprise).
And that's it--this is the end-game of SWIG-1.3.x development. When stabilized and documented it will become SWIG-2.0.
It's also worth noting that there are some rather interesting CS connections at work here. Extensions to the type system and typemaps have some interesting relations to work in programming languages. The SWIG declaration annotation system and class extension feature seem oddly similar to work in the emerging area of Aspect Oriented Programming (AOP). There are undoubtedly connections to other areas of software engineering and architecture.
The key point is that SWIG isn't going to connect to anything if no-one can quite describe what it is or how it works.
Over the summer, a few more 1.3.x releases may appear but the current plan is to aim for a SWIG-2.0 release in September. This release is really moving towards the design principles described above and will be a very major enhancement over SWIG-1.1.
As for the future, a number of interesting ideas are on the table. I want to add support for contracts/assertions in order to solve some reliability issues that arise when retrofitting legacy codes with a scripting interface. Support for an extension language has been promoted by David Fletcher and was suggested by someone else on the mailing list rather recently. I have a graduate student working on SWIG support for the Microsoft CLR and .NET languages. Other work might include support for alternative parsers, dynamically loadable language modules, and so forth. However, none of this is really going to materialize if we can't get the 2.0 release stablized. In fact, I see SWIG-2.0 as a necessary step for moving forward with these ideas.
If you like SWIG and find it useful, we need you to try new versions. We want you to torture test the releases and to break them. We need bug reports. No bug is too obscure or unimportant---we *will* fix it if we can. We also need feedback about things that are annoying or compatibility features that might help in going from 1.1 to 2.0.
We need help with documentation, examples, testing, libraries, and all sorts of other aspects of the release. Even if you have never written a SWIG language module or dived into its implementation, there are many ways that you can help. Consider writing a case study about how you wrapped a big library. Contribute tests that break the implementation in horrible ways. Help with the web page or FAQ.
Most of the SWIG-1.3.x work has focused on the SWIG core. However, as the 2.0 release nears, we will be working on a variety of enhancements to the language modules. If there are things you would like to see in any of the language modules, you need to let us know.
There are SWIG language modules that have not made it into the distribution. Examples that I know about include ITCL, Swig-Eiffel, and Swig-Lua. We would gladly make these part of the standard SWIG distribution. However, we also need help to do it. Porting from SWIG-1.1 is no easy task, but we're more than willing to help. It's not as bad as one might imagine.
We are always looking for developers. Subscribe to to the swig-devel mailing list or send me email to get involved.
Dave Beazley (beazley@cs.uchicago.edu)
June 2, 2002