Pyston 0.2: better than Pyston 0.1

We’re very excited to announce the release of Pyston 0.2, the latest version of Dropbox’s high-performance Python implementation, which contains substantial improvements over the initial 0.1 version.  Version 0.2, while still very much a work in progress, sports much-improved language compatibility, and is able to run many of the standard libraries — and some standard extension modules — without modifications.

New features

When we announced Pyston 0.1 in April, we had built the core Python-to-LLVM JIT infrastructure and were able to run simple Python programs.  Since then, we have made significant progress in supporting more language features and libraries.  Some of the features we added are:

  • Exceptions, using C++-style exception handling
  • Inheritance and metaclasses (no multiple inheritance, yet)
  • Basic native C API support
  • Closures, generators, lambdas, generator expressions
  • Full argument handling behavior
  • Longs, and integer promotion
  • Multithreading support

We are particularly excited about the native C API support: the implementation is a work in progress, but we are able to natively provide a subset of the CPython API and run existing extension modules with a recompile.  Among other techniques, we disable all refcounting operations, and use our conservative GC to handle the memory management.

We currently use a GIL to protect threaded code.  While inelegant, it is simple and efficient when there is only a single thread.  We have an experimental configuration option to use a read-write-lock variant, which “removes the GIL” and allows multiple threads to use multiple cores, but the potential parallelism is highly limited by the semantics of the C API.

There are still many features missing (such as finalizers, unicode, and C API exceptions), but we are at the point that we can run a number of completely unmodified benchmarks from the PyPy benchmark suite.  The benchmarks we run are themselves simple, but they use a harness that uses the optparse standard library, which in turn pulls in numerous other libraries including the entire regex system.  For example, the chaos.py benchmark is under 300 lines itself, but ends up running over 3k non-whitespace Python lines of code and requires the operation of over 10k lines of extension code.

Moving forward

Our strategy has been to first validate the overall JIT architecture (v0.1), then to improve language compatibility (v0.2), and is now to focus on improving performance (v0.3).  The 0.1 release demonstrated that we have the ability to produce high performance code using LLVM, but running real benchmarks has shown that our performance is currently hamstrung by our simple garbage collector and naive SSA-transform implementation.  In the coming months, we’ll be making improvements to these areas, and adding a new tiering framework to enable more advanced type speculation as well as support Python’s advanced frame introspection features.

Despite the improvements we’ve made, Pyston is still early in its life, and should be considered in early alpha.  It is still easy to crash Pyston (hopefully with an informative error message), and there are a number of known memory leaks.  Although we’ve made progress in the past five months, full language and builtin-types compatibility is a target that can only be approached and never 100% reached, and will be something we will have to continue working on.

We couldn’t have reached this state without the numerous open source contributions we’ve received: the 0.2 release includes non-documentation commits from 10 non-Dropbox-employees, and we would like to thank all of them for their contributions!  We still have a long way to go, and the repository remains open.  If building a high-performance Python JIT interests you, please feel free to browse the codebase or email our mailing-list.

9 thoughts on “Pyston 0.2: better than Pyston 0.1

  1. Really Cool! Is Pyston likely to ever run the whole scientific python stack at any point and thus replace Cpython for data science? (Scikits, statsmodels, plotting libraries numpy, scipy etc)

    Like

  2. […] Представлен второй тестовый выпуск проекта Pyston, в рамках которого развивается реализация языка Python, созданная с использованием наработок проекта LLVM, примечательная применением современных технологий JIT-компиляции и нацеленная на достижение высокой производительности, близкой к производительности традиционных системных языков, таких как C++. Код Pyston написан на языке C++ и распространяется под лицензией Apache. Проект развивается компанией Dropbox, в которой работает Гвидо ван Россум, создатель языка Python. […]

    Like

    • We have a number of places that fall into the category of “currently slow but we believe can be made fast”, so we’re choosing to not give numbers right now because we think there’d be more noise than signal. For example, in a benchmark I profiled, more than 75% of the time was spent in the garbage collector. This doesn’t really reflect on the potential of Pyston, but just shows that our quickly-thrown-together GC implementation needs to be replaced.

      But yes, these things need to actually be made fast at some point, which is what we’ll be doing with the 0.3 release; you can expect to see performance numbers when that comes out.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s