I’m keen to see more Windows users & developers active in core CPython development. Python has been supported on Windows for a long time by the core development team but it can sometimes feel a little bit like a poor cousin. The majority of Python users (and of key Python developers) are Unix-based: traditionally Linux & BSD, these days often OS X. Even if they’re not unsympathetic to the needs of Windows users, they may not be be in a position to help. But Python is a volunteer effort and if Windows-savvy developers aren’t available, Windows issues will remain unfixed and untested and better support for Windows-specific issues will not be forthcoming.

There are practicalities involved: Subversion or Mercurial, Visual Studio, being able to build Python from scratch and so on, but let’s start from the other end: you’re well-intentioned and motivated, but what should you do? I’m focusing on Windows-specific issues, but some of what I say here applies to anyone who’s interested in helping Python’s core development.

How should I start?

You don’t have to be an über-hacker or even a developer at all to contribute to Python. (Although you can do more if you are). You can start by submitting bug reports or requests for improvements to the issue tracker:

• Good: Submit a bug report: send an email to bugs@python.org detailing what version of Python you’re using and on which version of Windows, what you did and what went wrong. Or what you think should happen. If you’re able to re-test with a newer version of Python that will help, in case the bug’s already been identified and fixed. If it’s a documentation issue you’re describing, be clear what alternative or additional text you’re proposing.
• Better: Submit a bug report or feature request and a failing test case: send an email as above, but attach the simplest possible self-contained script which will cause the failure you’re describing or which will illustrate what should happen.
• Best: Submit a bug report with a failing test and add a patch which you believe will solve the issue plus a patch to the relevant Python test module plus a doc patch if applicable. If this is a documentation issue, the ideal is a patch against the reStructuredText docs; at the least, a clear plain text version of what you’re proposing.

Things to bear in mind:

• Don’t be disappointed if no-one takes any action. Ideally – and often – someone quickly notices the new issue, does a little categorisation on it and assigns it to a developer who’s known to be the owner of the area where the problem occurred. At the least, one or more developers may be added to the nosy list, and one of them may take ownership. Even if that doesn’t happen, you’ve definitely done the right thing by reporting the issue and offering a patch. If it’s Windows-related, feel free to add me to the nosy list. But even if a developer is assigned, developers are volunteers with their own availability, pressures and interests.
• Don’t be surprised if the issue isn’t quite as straightforward as it appears at first blush, particularly if backwards compatibility is at stake. There may be some history of which you’re unaware, or some ramifications in other areas of the codebase which aren’t obvious.
• Don’t give up in despair if the issue is rejected. This isn’t a hard-hearted manoeuvre by the core developers to reject all outside help. It just means that things aren’t always as simple as they might appear and some bigger-picture issue might have put the kibosh on your bright idea. If you feel that a valid point has genuinely been misunderstood you’re free to ask that the issue be reopened; but please don’t do this frivolously.
• Not all versions of Python are under active development. At the time of writing, the 2.x series is at the end of its life with 2.7 the final release. Clear bugs may be fixed in that branch, but little else. It’s unlikely that anything categorised as a new feature will make it in there. The same goes for 3.1. The latest development branch is version 3.2 where new features will be targetted. Any earlier releases will either see no changes whatsoever or only critical security fixes.

I’m a Windows developer, but where are the problems?

Being a core Python developer on Windows can mean several things: doing any kind of Python core development when Windows is your primary platform; working on Windows-specific modules, such as msvcrt or winreg, or the Windows-specific aspects of other modules, such as os and subprocess; or assessing and maintaining platform-specific subtleties such as compiler-sensitive datatype sizes or line-ending issues.

A lot depends on your own inclinations, motivations and areas of expertise. If you’re an experienced Visual Studio user you might focus more on some of the compiler-specific issues or build file configurations. If you have experience of sysadmin-type work you might want to look at the os module which wraps some of the Win32 API in posix-like calls or the winreg module which exposes the registry functions. If you’re a general purpose C coder who just uses Windows as a platform then you could look at the core Python code or the parts of the stdlib written in C. If you’re an MSI guru, look at some of the outstanding issues in Python’s own MSI and the bdist_msi distutils subsystem.

Ultimately involvement probably comes from one of three things:

• Scratching your own itch: is there something in the docs which you’ve never really understood because it relies on knowledge of Unix-specifics? Or just because it’s confusing? Have you been tripped up by a stdlib function which assumes that Windows paths always act like Unix paths? Have you tried to use os.access only to discover that it’s essentially useless on Windows? Do you think you could reimplement the signal module using the Win32 API? Or sponsor the merge of a 3rd-party Windows port of the curses module?
• Scratching someone else’s itch: have a look at bugs.python.org and choose an issue which is unresolved. The issue tracker has recently had new quick-searches added to it down the left and you can easily build your own searches. This search, for example, looks for unclosed issues which include Windows in the component list. It can be a useful exercise if you simply confirm that an unclear issue still obtains on current versions of Python and, possibly, on recent versions of Windows. Or if you produce a failing test case for a merely textual or vague report. (Something which can take quite some time). But be wary of producing “me-too” noise as people following that particular issue will receive notification emails which don’t add usefully to the known state of affairs.
• Finding an itch to scratch: if you’re keen to contribute but haven’t experienced any problems which need fixing, and haven’t found any existing bugs which take your fancy, then why not just browse the source code seeing how things are done and looking for ways to improve. Be careful here since cosmetic, trivial or frivolous changes are generally frowned upon. But, for example, the subprocess and multiprocessing modules both involve some reasonably serious Windows-specific plumbing. The module maintainers aren’t necessarily Windows experts and even if they are, the more eyes the better. There are other modules which have Windows-specific elements if only in small ways.

Where should I be helping?

• The issue tracker: bugs.python.org hosts the Python issue tracker, an instance of roundup which keeps track of all bug reports and feature requests. It’s broadly followed by core developers, but a small team keeps track of incoming bugs and tries to categorise them appropriately and to assign them to a known developer if possible, or to add to the issue’s nosy list developers known to be interested in that area of the system. By helping out in this way, you get a fairly good idea of the issues being reported and the work being done on them. And on which core developers have expertise in which areas of the system. It’s also relatively easy to get higher privileges on the tracker than he default logged-in user has so that you can help classify and manage open issues.
• Documentation: The documentation is maintained as reStructuredText files within Python’s version control repository and is published using Sphinx. It’s very easy to change and patch and there’s a batch file to rebuild it on Windows although you may need to tweak it a little. Doc patches are welcome, whether for small fry such as typos or slight clarifications, or for weightier rewrite.
• stdlib Python: The Python-only parts of the standard library. A lot of the stdlib is written in Python itself; that’s what makes the whole package so portable. Even if you’re not a C developer, you can still debug and test issues in the stdlib itself. It’s especially possible to do this without a C compiler or even a Subversion checkout: you only need to put an altered version of the module in a test directory from which you run the Python interpreter and the interpreter will see the local version first.
• stdlib Python & C: Some of the standard library relies partly or wholly on extension modules, written in C against Python’s C-API. The C is reasonably readable and is often concerned only with moving Python objects to and from some system library, which makes it a fairly restricted subset of C. To do anything useful with this, you will need a C compiler and, ideally, a version-controlled checkout.
• Core Python: The core of CPython is written in C: the objects, the interpreter, the underlying data structures, key built-in modules. You’ll need to be a fairly good C programmer to work here, not least because the impact is high if a mistake is made.
• Infrastructure: CPython needs a certain amount of scaffolding to build, especially on Windows where the standard autoconf/make tools don’t apply. This scaffolding tends to be under-supported, not least because it’s not very public-facing and doesn’t solve most people’s problems. But it still needs to be done. This includes Visual Studio solution and proj files, msi build mechanisms, buildbot scripts and documentation make files.

This is Part I; Part II should include information about checkout out the source, building it, creating & applying patches and working within the development community.

To get involved you certainly don’t need to be a Windows guru, nor a Python guru. You don’t need to have oodles of time on your hands. (Although all of the above would help). You do need a willingness to chase down occasionally obscure failing code paths and propose and implement patches. Or workarounds. Or documentation improvements. At present there is a very small number of us watching and fixing win32 issues and, speaking for myself, I have very little time available to help. The more people there are, each of which may have very little time, the more time is available overall to help Python develop and improve.

If you’d like to get involved but haven’t bothered previously, please bother now.

If you’d like to get involved but aren’t sure what to do, ping me or drop into #python-dev on IRC or just go to the Python bug tracker and look for Windows-related problems. (That search isn’t exhaustive as it relies on someone setting the Component field, but it certainly brings some results back).

Guido’s recently expressed a wish to give more people commit privileges, and MSDN have in the past generously provided free licences to allow core developers to develop and test on different Windows and Visual Studio versions.

There’s documentation on python.org for developers, and a new in-progress document by Brian Curtin on the Python sprints site.

Even if your contribution is to confirm that a bug still applies, or has been fixed, or to add tests to an undertested module which you use yourself, or to clarify or add some documentation, it will still be a very useful addition.

Although I was reasonably clued-in on the changes from 2 to 3, I was caught out by a few things, which I list here in case they help anyone else:

• __nonzero__ is now __bool__: I knew this perfectly well, but it took me a while to find out why my tests were failing. 2to3 would have picked this up straightaway.
• __hash__ isn’t inherited if __eq__ is defined: this one surprised me and took a bit of research. (All right; I actually read the docs for __hash__).
• bytes vs str: well, obviously you knew this, but it’s bitten me in some slightly unexpected places when interfacing via pywin32. The error message speaks of expecting a buffer object, which threw me slightly.
• assertTrue doesn’t bool its param: UPDATE - of course this isn’t true at all; I was confusing this and the __bool__/__nonzero__ issue above. Thanks to Marius Gedminas for pointing this out.
• pywin32 generates tz-aware datetime objects: for a long time, pywin32 rolled its own datetime type (as Python had none). Last year, Mark H canvassed the python-win32 list about committing to the Python datetime type. And now it’s there. But it’s tz-aware. And the core Python type isn’t. Fortunately, pywin32 has for a while shipped with the win32timezone module which solves the issue.

I’m sure there’ll be more which I’ll add here as they come along.

Irmen de Jong has released a Python3-compatible version of the newly-rewritten Pyro4 (as well as maintaining for 2.x). The developers of pyodbc (a module I use extensively) have a Python3 branch which is due to be merged soon. pywin32 has been 3.x for quite a while now (thanks, Mark & Roger). Once I finish porting my own win32-specific stuff, I’ll be well-placed to see 3.x rolled out here at work.

The only drawback to this approach is that people are commenting on a post I made (and which someone else submitted to reddit) without my being aware that the discussing is taking place, and without my being able to engage in that discussion. Which is a shame. But probably for the best :)

Now, I don’t really mind futile discussions, and this clearly is one: Python 3 isn’t going to switch back, no matter how many people weigh in on the print-as-statement side. I’m in that camp myself, and when the matter was first up for discussion I had a to-and-fro with Alex Martelli on the subject which I backed away from fairly quickly. I continue to enjoy and benefit from Python regardless; I contribute to its development very slightly and very rarely; and even were I the most prolific contributor on the planet, I don’t believe that would give me any particular right to dictate design decisions of this sort. You take the rough with the smooth.

I think the side of the discussion which irks me the most is from those who are defending the print-as-function decision. Clearly there are cogent and persuasive reasons why print should be / should have been a function. No-one’s pretending that this was a decision taken randomly and for the sake of change. Or even for the sake of a foolish consistency. But if someone’s working practice involves using print a lot — perhaps in the interpreter rather than in a code editor — then the switch to the function version is clearly a burden. It may be more or less of a burden, but it involves having to do something which you didn’t have to do before. Being told that you should have been using logging or pdb or sys.stdout.write — all of which are valid tools to use in the proper place — doesn’t really make your life any easier.

So while Python 3 is now the default interpreter on my shortcut keys and I’m trying to write new code against 3.2, I still find it a pain to overcome the very long habit of, eg, print ad.find_user ().sAMAccountName. Obviously if Python 1.5.2 had had print () as a function originally, I wouldn’t find is so hard now. ;)

(As an aside I did wonder at first whether the American or some other non-UK keyboards had the brackets in some more convenient spot, but it doesn’t appear so…)

Well I had misgivings as to whether we could grasp the codebase and get a workable solution in place within an hour and a bit, but it was great. The idea is — within the rules of the game — to pit your agents against the other teams’, trying for a winning strategy combining eating, attacking, moving, and replicating, each of which need, lose or gain energy according to certain rules. The agents can also message others on the same team to communicate, eg, the location of food or enemies. We had four teams, some with agents based on the sample code, others with from-scratch algorithms.

The clear winners were the “white” agents — Team 2 — who’d based their strategy on one of the supplied examples plus an expansion phase which kicked in only once a certain amount of time had passed and involved replicating like mad. On Team 1, we’d gone for a pacifist approach, with some of our agents as “Buddhas”, staying still and eating, the rest “Hungry Ghosts”, eating, moving and replicating. None of them attacked. FWIW, we got wiped out pretty quickly, although that did depend on how far you started from the voracious white team.

The other teams had differing strategies: explorers and eaters; or tribal warfare — which looked hilarious on-screen, as all the yellows suddenly went into warfare mode, sending each other the location of nearby enemies and flocking towards them.

Next Dojo’s in September, but EuroPython’s later this month and I’m sure there’ll be quite a few us there.

PyMODINIT_FUNC
PyInit_fileutils (void)
{
  PyDateTime_IMPORT;
  return PyModule_Create (&fileutils_module);
}

This is — sort of — expressed in the current docs but because the markup uses the :cfunction: role, Sphinx automatically appends a pair of brackets to the text. Additionally there’s no indication of where to invoke the macro. I tried a couple of places (at the head of the file, somewhere in the code) before discovering that the code would at least compile without it, so I assumed that it wasn’t needed for my simple use of PyDateTime_FromDateAndTime. Until it crashed on me, at which point I went searching and found the answer (thanks, Martin).

I plan to patch the docs, at least to change the markup to use cmacro, possibly to indicate usage once I’ve got some input from more knowledgeable people.

I kicked off with a talk about Pyro, which has recently announced a major version alpha release with versions for Python 2.x and 3.x. We’d discussed the possibility of enhancing our desultory Adventure Game efforts with multiplayerness and I’d volunteered to show how Pyro might be used for the purpose. Of course, the last-minute neatening-up and bugfixes I’d made before leaving the office didn’t make it onto my laptop, leaving me scrambling to correct code with minutes to spare. (Let this be a lesson to you: hg commit && hg push). Although I’d prepared a fairly tight series of examples and didn’t hang around, I still ran for 40 minutes with a few questions thrown in. (Coincidentally, as I sat down again, I had an email from Irmen de Jong, the author of Pyro, asking if I’d tried the new Pyro 4.x on Python 3.x yet. As it happens, I’d intended to do a simple comparison as part of my presentation, but I ran out of time).

Dave Kirby followed from the same angle, with a presentation about Twisted. *His* talk was hampered by laptop-to-projector issues which gave us a green screen, then a red screen, then as we were preparing for a blue screen it went green-red before Gaultier kicked the projector and we went back to white. I certainly came out understanding more about Twisted (which isn’t saying much, admittedly) and I can see why people enthuse about it so much. I’m also aware, from comments on the main Python mailing lists, that the Twisted guys have done a lot of work to file away the rough edges which result from getting low-level cross-platform networking code running.

I think Tom Dunham’s presentation was the best of the night (altho’ I’d had a glass of wine by then, which probably helped). A few dojos ago, one team had used Divmod’s Reverend implementation of a Bayesian classifier as a way to have more flexible parsing / analysis of the user input. It had worked at first, but had soon fallen apart in spite of some ingenious (read: desperate) attempts to “trick” the classifier into giving the desired result. Tom had gone away to investigate how we’d misunderstood the way in which such a classifier works. He came back with a presentation which skipped the maths (fortunately for most of us, I suspect!) and focused on the internals of the classifier, showing how our approach was hopelessly overloading the classification pools to the point where it would classify “the moon is cheese” and “the” in exactly the same way.

Jon Ribbens followed with a demo of his — now quite mature — jon.py web framework, built before almost all the other ones in common use today but still offering remarkable value for (no) money.

In spite of the late hour (gone 10pm before he started) Nicholas Tollervey, our brave organiser, gave a Lightning talk on FluidBD, which answered at least some of the questions we had about it (such as: what is it?), although it left me with the impression that it could be a Brave New World or a Interesting Curiosity.

Next time we’ll probably go back into Dojo mode proper, and with a new project to replace our slightly tired Adventure Game.

I find exactly the issues that the author outlines on page: open source s/w is mostly *nix-oriented, naturally, and assumes the development setups which that implies. By and large, Windows ports are either best-endeavour bolt-ons or rely on Cygwin / MingW environments. Python performs rather better here with a maintained VS solution, but even there you have to do a little bit more jiggery-pokery with file locations than I’d like.

I’ve signed up for the CoApp announcments list, so let’s see what happens…