Opnieuw: A simple and intuitive retrying library for Python

In this post, we want to introduce our new, easy to use, intuitive and simple retry package: Opnieuw, which is responsible for all the network retries in Channable’s Python code.

Opnieuw is a general-purpose retrying library that simplifies the task of adding retry behavior to both synchronous as well as asynchronous tasks.

Channable is a feed processing tool that imports products from webshops and exports those to marketplaces, price comparison sites, and advertisement platforms. At Channable we consume a lot of external APIs for our daily operations. If you make enough API calls, even the most robust API will occasionally fail. By retrying our request we can usually solve this problem. Therefore, having a good retry mechanism is important for making our operations run smoothly.

Why we wrote Opnieuw

Before writing our own retry package, we tried several other retry packages. Unfortunately we fell into multiple pitfalls:

Opnieuw is our take at avoiding those pitfalls, to make a retry library that is easy to use. Let’s look at an example in order to demonstrate the power of Opnieuw. Suppose we want to parse https://tech.channable.com/atom.xml, and we want to retry on ConnectionError and HTTPError. We can add Opnieuw to our network handler as follows:

import requests 
from requests.exceptions import ConnectionError, HTTPError

from opnieuw import retry


@retry(
    retry_on_exceptions=(ConnectionError, HTTPError),
    max_calls_total=4,
    retry_window_after_first_call_in_seconds=60, 
) 
def get_page() -> str: 
    response = requests.get('https://tech.channable.com/atom.xml') 
    return response.text 

As is clear from the above example, by decorating the function we add retry behavior. You should not need to read any documentation to understand the snippet:

In the above example, we would call get_page() at least once and at most 4 times, and the last call would be at most 60 seconds after the first call. On a ConnectionError or HTTPError we would retry, and the wait time after a failed attempt grows exponentially. Other exceptions would bubble up up and not be retried.

Opnieuw is a simple and easy way to add exponential backoff retries with jitter to a call. In addition to retry(), Opnieuw features retry_async() for retrying asynchronous tasks. All the parameters remain the same, the only difference is that it can be used with async functions.

About the name

“Opnieuw” means “again” in Dutch. We had an internal naming competition and Opnieuw won out over Perseverance and channable-retry.

Conclusion

At Channable, we saw an opportunity to improve upon existing retry packages. Unclear APIs were a source of bugs and counter-intuitive behavior. By prioritizing the parameters that users care about, Opnieuw makes it easy to reason about retry behavior of network operations.

Today we are releasing Opnieuw as open source.

Discuss this post on Reddit or on Hacker News