coroutines: basic building blocks for concurrency

This part of the series explains the basic building block that allow writing concurrent programs in python.

Later in the series I’ll show how to use different async paradigms using the new async syntax that was (finally) introduced in Python 3.5.

Prerequisites

  1. You’re using python 3.6.x
  2. You’re familiar with coroutines. otherwise, read - coroutines: Introduction.

A bit of history

How does async & await work?

I’m not a fan of duplicate work, and a colleague of mine told me that Brett Cannon, a Python core developer, already wrote a great post on the subject - How the heck does async/await work in Python 3.5?

I HIGHLY RECOMMEND READING ALL OF IT. It’s beautifully written and explains all the new things that were added to the language in the past few years.

What’s New In Python 3.6

Python 3.6 is considered by many the first release that makes sense to move over from Python 2.x.

There (no very long) list of features that were added in Python 3.6 can be found here.

You can also watch Brett’s talk on the subject:

Anyhow, the following are the new features related to async.

PEP 525: Asynchronous Generators

PEP 492 introduced support for native coroutines and async / await syntax to Python 3.5. A notable limitation of the Python 3.5 implementation is that it was not possible to use await and yield in the same function body. In Python 3.6 this restriction has been lifted, making it possible to define asynchronous generators:

async def ticker(delay, to):
"""Yield numbers from 0 to *to* every *delay* seconds."""
for i in range(to):
yield i
await asyncio.sleep(delay)

The new syntax allows for faster and more concise code.

PEP 530: Asynchronous Comprehensions

PEP 530 adds support for using async for in list, set, dict comprehensions and generator expressions:

result = [i async for i in aiter() if i % 2]

Additionally, await expressions are supported in all kinds of comprehensions:

result = [await fun() for fun in funcs if await condition()]

The death of Twisted and such

There are a bunch of async frameworks in the wild, the most noticeable being Twisted and Tornado. All of them use Python’s awesome generators & coroutines that were introduced into the language more than 15 years ago!

I’m not a fan of both. I’ve worked with Twisted a lot at my last work place. We wrote our generic crawler using Scrapy, which is based on Twisted.

These frameworks work great. until something breaks that is. Then you get the worst errors imaginable, and debugging them is a nightmare.

I was ecstatic when python introduced asyncio, then async & await. Many in the developer community, including myself, believed that was the end for Twisted, but we were wrong - The report of Twisted’s death was (and still is) an exaggeration.