Python: The Full Monty

A Tested Semantics for the Python Programming Language

Artifact Evaluation Submission

This Document

This documentat is intended to assist the OOPSLA Artifact Evaluation Committee (AEC) and any other interested parties reproduce and extend the results from our OOPSLA 2013 paper.

This document is available at http://cs.brown.edu/research/plt/dl/lambda-py/ae/index.html, and in the tarball that we submitted to the AEC. The tarball is also available from http://cs.brown.edu/research/plt/dl/lambda-py/ae/lambda-py.tgz.

This document covers:

Contact

If you have any problems or questions that require consulting the authors during the review process, email joe@cs.brown.edu

Getting Started

The virtual machine is in the directory lambda-py-server. The file lambda-py-server.vbox is runnable by double-clicking on many platforms with VirtualBox installed. You can also create a new virtual machine, choosing 'Linux' and 'Ubuntu' (not 64-bit), and use lambda-py-server.vmdk as the disk image. If you take this route, provide at least 1G of memory for the VM. In the first run wizard, choose lambda-py-server.vmdk as the disk image to use.

Once you start the server, it may take a few minutes to start up on a screen that looks like:

There is one useful user on the virtual machine, reviewer, with sudo privileges and password reviewer. When prompted for a login, enter these credentials and you should be presented with a desktop that looks like:

The files for review are all in the directory /home/reviewer/lambda-py. The implementation and tests for λπ at artifact submission time are in lambda-py-artifact-submission. We focus on the current state of the implementation, which enjoys some significant improvements over the implementation at submission time. It passes all the same tests as the submission-time version and more (we include the version from submission time with instructions (at the end of this document) for completeness).

Our first goal should be to simply verify that lambda-py runs. Open a terminal ("Applications Menu" at the top left, then "Terminal Emulator"), and let's first run a very small test so we can find out if anything goes wrong right away (if you have the opportunity to ask the authors for help, we're putting this here for quick diagnosis):

$ cd lambda-py/lambda-py-artifact-submission/base
$ echo "print('lambda-py works')" | racket python-main.rkt --interp
lambda-py works
    

This will take a few seconds to run, and confirms that the λπ interpreter is up and running for you.

Step-by-Step Instructions

We've added new features since submission time, documented in CHANGES.txt. The new test cases are alongside the old in the tests/python-reference subdirectory of lambda-py-artifact-submission. So if you run the following, you will see more tests passing than the 175 we report in the paper:

$ cd ~/lambda-py/lambda-py-artifact-submission/base
$ racket python-main.rkt --test ../tests/python-reference/
205 tests succeeded
0 tests failed
    

We provide a script called line_count.sh that uses CLOC to count the lines of code in each directory of python-reference. This compares the current state of the repository to the numbers reported in the paper in Figure 12 (the "Built-in Datatypes" count is the combination of the directories function, bool, builtin, tuple, lists, and dict, "Iteration" is the combination of directories iter and range, and "(Multiple) Inheritance" is the combination of directories super and multiple-inheritance). So, you can do:

$ cd ~/lambda-py/lambda-py-artifact-submission/tests
$ ./line_count.sh 
Feature			# of tests	LOC
Built-in Datatypes	81		902
Scope			39		455
Exceptions		25		247
Multiple Inheritance	16		303
Properties		9		184
Iteration		13		214
Generators		9		129
Modules			6		58

Total			205		2636
    

The repository is also set up to run all the same tests using Python. We've included an installation of Python 3.2.3 (the version we are modelling) in the VM, and these tests can be run with:

$ cd ~/lambda-py/lambda-py-artifact-submission/base
$ racket python-main.rkt --python-path ~/install-stuff/Python-3.2.3/python --test-py ../tests/python-reference
    

Note: We drive the Python tester with Racket, passing the Python path as an argument. The implementation of our test harness performs the same directory traversal as when running with λπ, but runs with the Python executable, and checks standard out and standard error the same way.

To go back in time and run all the tests that we report in the paper, you can visit the other pre-installed directory that contains the code from that time:

$ cd ~/lambda-py/lambda-py-28-march-2013/base
$ racket python-main.rkt --python-path ~/install-stuff/Python-3.2.3/python --test ../tests/python-reference/
175 tests succeeded
0 tests failed
    

(We pass the Python path because we originally used Python to get the original Python AST for desugaring, and it has since been replaced. Also, the --test-py option, when run on the submission time code, reports 3 failures in the modules tests, this is an error in the test harness setting the working directory for Python. We have since fixed it.)

That's enough to verify the test results reported in the paper, and some of what we've worked on since then.

Looking Around

Code from the paper

Many of the discussions in the paper directly correspond to a few places in the code; it is useful to see how they translate. These are presented as links to Github sources, since they have a nice viewer for the files.

As noted in the paper, λπ isn't quite to the point of running Python's full unittest library, so these tests use a limited language of assertions. For example, if we look at tests/python-reference/multiple-inheritance/methods.py, we see lines like:

___assertEqual(Foo.getx(foo), 1)
    
The implementation of ___assertEqual is Python code that simply checks that the arguments are equal according to Python's == operator, and its implementation is in base/py-prelude.py.

Playing around with λπ

It's easy to tweak the Python-implemented libraries of λπ to see what their effects are on Python programs. Since these libraries implement so much built-in behavior, tweaking them can have interesting effects. You can run individual Python programs through λπ by passing them through standard input and using the --interp option:

$ racket python-main.rkt --interp < some-python-file.py
    

Some things you might try:

Building from Scratch

These instructions are included so reviewers can see the build process for λπ on their own machines. Reviewers should be able to evaluate the artifact's adequacy relative to the paper from the VM, but a manual installation hopefully demonstrates that the artifact is easy to install and extend.

The code for λπ is available at https://github.com/brownplt/lambda-py.

We have two points in the repository's history that are interesting for this review: One at the time of our OOPSLA submission (28 March, 2013), and one at the time of our artifact submission (1 June 2013). The former is tagged in the repository as oopsla2013, and the latter is the HEAD of the artifactevaluation branch (we'll push minor bugfixes here if any come up in the evaluation process, it contains this index file which isn't on the master branch, and if we do work and change master during the review process, we won't make breaking changes for the reviewers).

Getting Racket

Both require that you install Racket; the main download link at http://racket-lang.org/download/ has installers for many platforms. We have tested on several different Ubuntus from 11.04 to 12.10, and on OSX, with both Racket 5.3.3 and Racket 5.3.4. For review, we recommend Racket 5.3.4. Download the installer and run it; you can pick any of the options for installing Racket (for example, you can install UNIX-style in /usr/, or keep the installation in your home directory). In the instructions that follow, we assume that the $PATH environment variable is pointing to the bin/ directory of the Racket installation that holds the raco and racket commands (this may be done for you automatically depending on which options you choose in the Racket installer).

Example:

$ wget http://download.racket-lang.org/installers/5.3.4/racket/racket-5.3.4-bin-i386-linux-ubuntu-karmic.sh
$ sh racket-5.3.4-bin-i386-linux-ubuntu-karmic.sh
    
Getting the Source of λπ

Github is the easiest way to get a copy of the code:

$ git clone https://github.com/brownplt/lambda-py
    

BE AWARE: If you fork the repository on Github, it will leak your identity to us (it's up to you if that's a problem). Just using git clone (as above) won't.

Artifact Submission-time Build

Paper Submission-time Build

We recommend using the provided virtual machine to review the code as it was at submission time, since it involves additional build steps that we have since made much easier.