Beyond Python Basics

While travelling in Europe, I had the opportunity to dig a little deeper into the Python programming language — not just into its internals, but also play around with some of Python’s slightly esoteric features.

I’d recommend any Python student who’s completed a basic tutorial [1] and fiddled around with a project or two to have a look at the following. Don’t be deterred by the terminology, because the concepts are explained as you go:

Iterators, Generators and Coroutines

Generator Tricks for System Programmers (v 2.0)

• Starts with Python’s iteration mechanics (e.g. for item in list)
• Explains generators, objects which generate output values upon request (not in advance) and why they’re often useful, e.g. for efficiency purposes
• Demonstrates generators’ power by using them to analyse web server logs (including real-time processing as data is appended to the log) and shows how to extend these pipelines across processes and machines.

A Curious Course on Coroutines and Concurrency

• Compares generators (message producers), which are ‘pulled from’ in iteration, to coroutines (message consumers), which ‘push’ data to others
• Demonstrates why, for processing pipelines, coroutines provide more data routing options (e.g. for concurrency & distributed systems)
• Utilises coroutines’s ‘task’ nature to build a miniature, multitasking operating system (Yeah. This gets heavy.)

Liked these? David Beazley, author of the Python Essential Reference and creator of these presentations, has kindly provided a list of some of the talks and tutorials he’s given.

Metaclasses, Decorators and ‘Yield’

e-satis on Stack Overflow wrote three fantastic ‘conceptual understanding’ answers, which make sense as stand-alone explanations:

The Python yield keyword explained

• If you found David Beazley’s generators presentation too long-winded or complex, you’ll like this; it covers the same concepts.

What is a metaclass in Python?

• Not only are instances of classes objects, but classes themselves are too.
• In addition to its familiar use, the type function can return a class object.
• In fact, type is actually the class used to create all class objects (a metaclass), and you can create your own metaclasses by creating objects which subclass or use type.

Understanding Python decorators

(Decorators are the weird @name syntax you sometimes see before function definitions; I’ve only used them once before, in views.py for Django apps.)

• Because functions are objects, you can:
  • Make a function which returns a function (object), and
  • Pass a function as an argument to another function
• Say A is a function object. We can create another function (B) which:
  • Takes a function (e.g. A) as an argument
  • Defines a new function C which calls A before/after doing other stuff
  • Returns C
• If A is assigned to the return value of B, A is now C, i.e. a ‘wrapped’ version of what A used to be. Decorators are just a convenient syntax for this process.

Advanced Python or Understanding Python

This is an internal Google presentation on hardcore Python by Thomas Wouters, uploaded for the rest of the world to enjoy.

Something that never really clicked for me until this presentation: namespaces are just a dictionary mapping names to objects. I started seeing them everywhere, e.g. as the context used by Django to render HTML templates (references to ‘list_of_jobs’ in the template refer to this Python object, etc).

The presentation has lots of other cool (but deep) stuff too:

• Python design, execution model, scopes (1:28)
• Python objects (14:40)

(this stuff is hard, but makes slightly more sense after watching the PyPy video below (specifically re: its bytecode comments))

• Iterators/Generators (26:17)
• Decorators (33:22)

(these topics are easy to understand if you viewed their above tutorials)

• New-style class features (38:05)
• Descriptors, properties (40:25)
• Classmethods (43:28), staticmethods (45:46)

(most of this goes over my head, but it doesn’t seem to matter in a day-to-day practical sense anyway)

• Metaclasses (52:48), multiple inheritance (1:00:40), Unicode (1:04:27)

(the metaclass stuff isn’t too hard to understand once you’ve read the tutorial on them above, and there’s a nice description of how Django uses metaclasses for models.py function definitions; I’ve never had to use multiple inheritance stuff; Unicode makes sense and is important!)

Bonus: Thinking about Tinkering with PyPy (PyCon 2012)

Curious about PyPy? Python’s default implementation is written in C (CPython), but PyPy implements Python in a Python subset called RPython. Here's David Beazley’s attempt at explaining RPython and how (part of) PyPy works. (Here’s the beta version of the talk, if you’re curious.)

Bonus 2: ‘A Few of My Favourite Things’

Michael Pirnat’s CodeMash 2012 talk quickly runs through his favourite things about Python and its ecosystem. Though the slides’ material is probably more up-to-date than many of the above materials, it’s not a tutorial, so consider further research on anything which piques your interest!

[1] As for tutorials, I’d recommend the How to Think Like a Computer Scientist series; they’re the most helpful ones I’ve encountered. For Python 2.x and 3.x.