Moderate Realism

There’s work afoot by Mark Hammond with encouragement from others to produce a Python launcher for Windows. The work’s being specified under PEP 397 (not yet in the PEP index) and a reference implementation is being tracked under issue11629 on the Python issue tracker.

The need arises from the difficulties of having an easy way to associate Python .py/.pyw files with the appropriate version of Python on Windows, exacerbated by the advent of Python 3. The idea is that rather than having .py associate with c:\pythonxy\python.exe, it would associate with \py.exe which would attempt to read a shebang line (#! python.exe) from the top of the file dropping through various levels of default to choose a meaningful version of the Python executable to run.

From what sounds like a simple requirement there are lots of details to iron out. The (draft) PEP explains the levels of drop-through which the launcher will employ to decide what to run. There’s what seems to be a sort of awkward stand-off between not caring about cross-platformability and honouring common conventions such as /usr/bin/python. For my part I rarely if ever write code which needs to run on *nix — it’s so Windows specific that it wouldn’t even import all its dependencies, let alone do anything useful. So that side of things matters less to me.

The other detail which has caused much discussion is the mechanism by which the launcher process starts the Python interpreter. Ignoring such esoterica as COM instantiation, you have basically two options: have the launcher load the appropriate Python .DLL; or have the launcher call CreateProcess on the appropriate python .exe, passing through the parameters, open handles, etc. Each has its pros and cons but Mark’s currently plumping for creating a child process and using the Windows Job Object API to ensure the child dies with its parent.

If you’re at all invested in developing or deploying Python under Windows, have a look at the PEP and try out the Python reference implementation.

I’m fairly new to github; to git, in fact. Truth be told, I wouldn’t even have bothered with it if it weren’t for the fact that ntoll organises the code for the London Python Dojo around it. And that the game engine we had decided to use was hosted there. While I’m all in favour of DVCSes (you just *know* what’s coming when someone says “I’m all in favour…”, don’t you?), the problem is that there’s only one Subversion, but there are at least two if not three — depending on your view on bzr — DVCSes with fairly widespread support, and they’re all subtly or radically different. I more-or-less committed to learning Mercurial, because it’s what Python’s moving to, but that doesn’t help me when other people are using git. Oh well…

Previously on London Dojo: we cloned the crawlr engine from axedcode’s github repo. And found, ironically, a maze of twisty passages all alike. We went north, south, east and west in search of treasure but found only darkness… Fortunately, the man himself spotted that his game engine was flavour of the month and got in touch to ask why, and to offer a newer, better version: axengine2. Which is a lot cleaner, even if there are some questionable design choices…

So we did that. And I spent half an hour at the beginning of last Thursday’s Dojo going over key pygame concepts — rects, surfaces, sprites, and sprite groups — and giving a whistle-stop tour of the codebase which, slightly surprisingly, reimplements the Python class inheritance model in JSON, backed up by Python classes. Sort of. There was half a chance we might even have a live Skype connection to the author at the beginning of the session — before he started work over in the States — but timing didn’t work out so it didn’t come off. Shame.

We set to work again, in teams as usual, each team having a “Dojo Quest” and 90mins to complete it. Naturally, everyone was still frantically coding at the 89-minute mark. (I know we were, having made a critical breakthrough in our attempt to introduce a joystick-controlled second player [*]). We had the usual show-and-tell of: a menu system, some background sound, the two-player mode, hidden treasure, and something else I’ve forgotten. By which time it was getting on for 10.30pm and there was only just time to pull a name out of the hat for the winner of the O’Reilly Book-of-the-Month on XMPP, which we hope to use for the inter-dojo gameplay.

The Dojos are always fun; they’re also nothing like the disciplined Dojos which others favour. Basically, it’s a social coding exercise with a fair bit of laughter and passenger-seat driving. We seem to be nudging the 30 mark every month at the moment, so watch the Python UK mailing list for the announcement that tickets are available and get in there quick.

[*] We delayed ourselves a little by confusing the KEYUP / KEYDOWN constants — which indicate depression and release of keys — with the constants indicating the up and down-arrow keys :(

What technical stuff am I trying to pick up at the moment? I suppose many techy types are like me in that they keep a more-or-less close eye on the various technologies and libraries just to know what’s around. Without an immediate need; and without the time or perhaps the inclination to dive into any one of them. But from time to time some kind of incentive or motivation (or need (or job)) comes along and you start to pick something up to see if it’s worth pursuing further.

At the moment, then, I’m reading up on and trying out:

• Jabber / XMPP - for a future London Python Code Dojo
• C# - for possible work and for career prospects (and just because…)
• BI / Data Warehousing - for work
• Django - just because

Things I want to find the time for (and the money to buy the books):

• Windows Internals - to have a better understanding when I’m helping people out
• New aspects of WMI & AD - for work and to help other people

All of which adds up to a fair amount of reading and so on. And that’s on top of everything I’m trying to do. Good thing I’ve got a 50-minute train journey in the morning.

There were sounds of fear and screaming courtesy of Team 1 at the Dojo last night, hosted as ever at Fry IT offices in Southwark — an area whose eclectic mixture of shiningly-new and uncertainly-old architecture I continue to enjoy. We kicked off with the idea, promoted by Bruce (aka @otfrom), of producing the basis of a game to which other London-based Dojos could submit bots. Nicholas (@ntoll) had tweaked an existing PyGame-based game and proposed specific “Dojo Quests” for each team to work on independently.

Which was fine, except for two things. the sheer number of people — just below 30, a record I think, and leaving people perched on tables or even on the floor for want of chairs; and the relative complexity of the code. Very much to its credit, the modules were well-structured, consistently named and fairly well laid-out. But there were layers of complexity in there which meant that, in less than 90 minutes, certain of the teams were struggling to understand enough to start the task, let alone complete it. Including ours.

Fortunately — and with the help of Rene (@renedudfield) — most teams managed to get at least something to show in the “show & tell” session at the end. I think Nicholas was relieved that it came off as well as it did: we’d had a previous bad experience in trying to get to grips with an existing library in the course of a Dojo. Team 1’s Dojo Quest involved adding music & sound effects, which they did with gusto. The end result was a character wandering around a rural landscape with randomly-generated screams and cries of fear filling the air. Never has a walk in a forest been so terrifying.

In addition to helping out generally, Rene also generously provided a bunch of game controllers. And O’Reilly continued their “sponsorship” of the Dojo by providing their XMPP title for our customary raffle… which I won! (The idea is to use XMPP to connect these hypothetical Dojo-generated gamebots).

It was great to see so many people, although it does pretty much illustrate the physical limitations of our venue. I enjoyed it as always, and people I chatted to afterwards did, too. Looking forward to next month.

UPDATE: Forgot to mention that we *really* kicked off with a truly lightning talk from Tom (@tomviner) showcasing Michael Foord’s recent e and oo modules.

If I might essay a generalisation: Windows-based users of Python are less likely to have become involved in the development of Python itself. This might be for the same reasons that other people don’t get involved: a lack of time, of inclination, or of confidence in their ability to help. But it might be because Windows users on the whole aren’t so accustomed to delving into the source code of the things they use, nor are they used to checking out from source control, testing fixes and rebuilding. It’s not so much part of the culture.

I’m keen to see more Windows users & developers active in core CPython development. Python has been supported on Windows for a long time by the core development team but it can sometimes feel a little bit like a poor cousin. The majority of Python users (and of key Python developers) are Unix-based: traditionally Linux & BSD, these days often OS X. Even if they’re not unsympathetic to the needs of Windows users, they may not be be in a position to help. But Python is a volunteer effort and if Windows-savvy developers aren’t available, Windows issues will remain unfixed and untested and better support for Windows-specific issues will not be forthcoming.

There are practicalities involved: Subversion or Mercurial, Visual Studio, being able to build Python from scratch and so on, but let’s start from the other end: you’re well-intentioned and motivated, but what should you do? I’m focusing on Windows-specific issues, but some of what I say here applies to anyone who’s interested in helping Python’s core development.

How should I start?

You don’t have to be an über-hacker or even a developer at all to contribute to Python. (Although you can do more if you are). You can start by submitting bug reports or requests for improvements to the issue tracker:

• Good: Submit a bug report: send an email to bugs@python.org detailing what version of Python you’re using and on which version of Windows, what you did and what went wrong. Or what you think should happen. If you’re able to re-test with a newer version of Python that will help, in case the bug’s already been identified and fixed. If it’s a documentation issue you’re describing, be clear what alternative or additional text you’re proposing.
• Better: Submit a bug report or feature request and a failing test case: send an email as above, but attach the simplest possible self-contained script which will cause the failure you’re describing or which will illustrate what should happen.
• Best: Submit a bug report with a failing test and add a patch which you believe will solve the issue plus a patch to the relevant Python test module plus a doc patch if applicable. If this is a documentation issue, the ideal is a patch against the reStructuredText docs; at the least, a clear plain text version of what you’re proposing.

Things to bear in mind:

• Don’t be disappointed if no-one takes any action. Ideally – and often – someone quickly notices the new issue, does a little categorisation on it and assigns it to a developer who’s known to be the owner of the area where the problem occurred. At the least, one or more developers may be added to the nosy list, and one of them may take ownership. Even if that doesn’t happen, you’ve definitely done the right thing by reporting the issue and offering a patch. If it’s Windows-related, feel free to add me to the nosy list. But even if a developer is assigned, developers are volunteers with their own availability, pressures and interests.
• Don’t be surprised if the issue isn’t quite as straightforward as it appears at first blush, particularly if backwards compatibility is at stake. There may be some history of which you’re unaware, or some ramifications in other areas of the codebase which aren’t obvious.
• Don’t give up in despair if the issue is rejected. This isn’t a hard-hearted manoeuvre by the core developers to reject all outside help. It just means that things aren’t always as simple as they might appear and some bigger-picture issue might have put the kibosh on your bright idea. If you feel that a valid point has genuinely been misunderstood you’re free to ask that the issue be reopened; but please don’t do this frivolously.
• Not all versions of Python are under active development. At the time of writing, the 2.x series is at the end of its life with 2.7 the final release. Clear bugs may be fixed in that branch, but little else. It’s unlikely that anything categorised as a new feature will make it in there. The same goes for 3.1. The latest development branch is version 3.2 where new features will be targetted. Any earlier releases will either see no changes whatsoever or only critical security fixes.

I’m a Windows developer, but where are the problems?

Being a core Python developer on Windows can mean several things: doing any kind of Python core development when Windows is your primary platform; working on Windows-specific modules, such as msvcrt or winreg, or the Windows-specific aspects of other modules, such as os and subprocess; or assessing and maintaining platform-specific subtleties such as compiler-sensitive datatype sizes or line-ending issues.

A lot depends on your own inclinations, motivations and areas of expertise. If you’re an experienced Visual Studio user you might focus more on some of the compiler-specific issues or build file configurations. If you have experience of sysadmin-type work you might want to look at the os module which wraps some of the Win32 API in posix-like calls or the winreg module which exposes the registry functions. If you’re a general purpose C coder who just uses Windows as a platform then you could look at the core Python code or the parts of the stdlib written in C. If you’re an MSI guru, look at some of the outstanding issues in Python’s own MSI and the bdist_msi distutils subsystem.

Ultimately involvement probably comes from one of three things:

• Scratching your own itch: is there something in the docs which you’ve never really understood because it relies on knowledge of Unix-specifics? Or just because it’s confusing? Have you been tripped up by a stdlib function which assumes that Windows paths always act like Unix paths? Have you tried to use os.access only to discover that it’s essentially useless on Windows? Do you think you could reimplement the signal module using the Win32 API? Or sponsor the merge of a 3rd-party Windows port of the curses module?
• Scratching someone else’s itch: have a look at bugs.python.org and choose an issue which is unresolved. The issue tracker has recently had new quick-searches added to it down the left and you can easily build your own searches. This search, for example, looks for unclosed issues which include Windows in the component list. It can be a useful exercise if you simply confirm that an unclear issue still obtains on current versions of Python and, possibly, on recent versions of Windows. Or if you produce a failing test case for a merely textual or vague report. (Something which can take quite some time). But be wary of producing “me-too” noise as people following that particular issue will receive notification emails which don’t add usefully to the known state of affairs.
• Finding an itch to scratch: if you’re keen to contribute but haven’t experienced any problems which need fixing, and haven’t found any existing bugs which take your fancy, then why not just browse the source code seeing how things are done and looking for ways to improve. Be careful here since cosmetic, trivial or frivolous changes are generally frowned upon. But, for example, the subprocess and multiprocessing modules both involve some reasonably serious Windows-specific plumbing. The module maintainers aren’t necessarily Windows experts and even if they are, the more eyes the better. There are other modules which have Windows-specific elements if only in small ways.

Where should I be helping?

• The issue tracker: bugs.python.org hosts the Python issue tracker, an instance of roundup which keeps track of all bug reports and feature requests. It’s broadly followed by core developers, but a small team keeps track of incoming bugs and tries to categorise them appropriately and to assign them to a known developer if possible, or to add to the issue’s nosy list developers known to be interested in that area of the system. By helping out in this way, you get a fairly good idea of the issues being reported and the work being done on them. And on which core developers have expertise in which areas of the system. It’s also relatively easy to get higher privileges on the tracker than he default logged-in user has so that you can help classify and manage open issues.
• Documentation: The documentation is maintained as reStructuredText files within Python’s version control repository and is published using Sphinx. It’s very easy to change and patch and there’s a batch file to rebuild it on Windows although you may need to tweak it a little. Doc patches are welcome, whether for small fry such as typos or slight clarifications, or for weightier rewrite.
• stdlib Python: The Python-only parts of the standard library. A lot of the stdlib is written in Python itself; that’s what makes the whole package so portable. Even if you’re not a C developer, you can still debug and test issues in the stdlib itself. It’s especially possible to do this without a C compiler or even a Subversion checkout: you only need to put an altered version of the module in a test directory from which you run the Python interpreter and the interpreter will see the local version first.
• stdlib Python & C: Some of the standard library relies partly or wholly on extension modules, written in C against Python’s C-API. The C is reasonably readable and is often concerned only with moving Python objects to and from some system library, which makes it a fairly restricted subset of C. To do anything useful with this, you will need a C compiler and, ideally, a version-controlled checkout.
• Core Python: The core of CPython is written in C: the objects, the interpreter, the underlying data structures, key built-in modules. You’ll need to be a fairly good C programmer to work here, not least because the impact is high if a mistake is made.
• Infrastructure: CPython needs a certain amount of scaffolding to build, especially on Windows where the standard autoconf/make tools don’t apply. This scaffolding tends to be under-supported, not least because it’s not very public-facing and doesn’t solve most people’s problems. But it still needs to be done. This includes Visual Studio solution and proj files, msi build mechanisms, buildbot scripts and documentation make files.

This is Part I; Part II should include information about checkout out the source, building it, creating & applying patches and working within the development community.