Python: Decorator Fundamentals#

Python has had the standard @decorator style decorator syntax since PEP 318 – Decorators for Functions and Methods was accepted, while some tweaks to the grammar have been made a long the way via PEP 3129 – Class Decorators and PEP 614 – Relaxing Grammar Restrictions On Decorators, their behavior has remained largely unchanged.

The most common misconception about decorators is that they are a function that takes a function and returns a function. While this does describe a common pattern for decorators, it ignores their generalized framework and misses strong usecases for decorators. Instead, I will say that a decorator is a callable that takes a class or function as an argument to encapsulate/manipulate some state and/or prepend/append some behavior to that class or function. While that definition is quite verbose, I think the following code snippets will help make my point.

When should I use a decorator?#

You only need a decorator when you have many functions/classes, and you want to apply a similar operation on all of them. As a broad generalization, can use a decorator whenever you have a piece of code that you would like to run when:

  1. A function/class is defined

  2. Prior each function/class call

  3. After each function/class call

These are the common entry points for code execution that decorators enable our code to tap into. By having programmatic access to these entry points, we can easily perform operations like registering functions¹, argument validation/injection², timing function execution²³, docstring appension¹, single dispatch¹², classmethods & staticmethods¹, and much more.

As you can see there are a huge variety to the types of patterns decorators help achieve in our Python code. While it is possible to write these ideas into our code, decorators enable us to do so with a concise and flexible language. Ensuring our code is not needlessly repetitive and easy to maintain/extend.

Let’s take some time to discuss how we can implement decorators in our code- going back to how we can implement them without using the @ syntax.

Decorators prior to @#

Before PEP 318, decorators had to be written in a much more verbose manner which often obscured code meaning because it separated the decorating-function from the decorated-function. Let’s look at a simplified version of the motivating example from the aforementioned PEP to demonstrate:

# 1) old syntax
def foo(cls):
    pass

foo = classmethod(foo)
foo
<classmethod(<function foo at 0x7f3691e53370>)>
# 2) new syntax
@classmethod
def foo(cls):
    pass

foo
<classmethod(<function foo at 0x7f3691e536d0>)>

In example 1, we allow for a separation between the decorating function classmethod to be separate from the decorated function foo. While this may not seem too bad, this syntax supports for arbitrary separation of the decorating and decorated functions, meaning that you may not know a function has been decorated until some time later reading the code.

By keeping the decorating function immediately on top of the decorated function such as with the @ style syntax, we can immediately notify someone reading this code that the currently defined function foo is being interacted with in some way by classmethod

Passing functions as arguments#

In order to understand decorators, we first need some foundational building blocks. The most important of these pieces is the idea of passing functions (and classes) as an argument to another callable.

Passing functions (and classes) as objects is the most basic concept you’ll need to know in order to understand decorators.

In a nut shell, this idea allows us to write code that looks like this:

from random import seed, random
seed(0)

def call_many_times(f, n_times):
    for i in range(n_times):
        print(f'call {i}{f() = :.3f}')

call_many_times(random, 3)
call 0 → f() = 0.844
call 1 → f() = 0.758
call 2 → f() = 0.421

In this case, I have passed the function random to my function call_many_times as argument f. I then call f 3 times and print out the result of each call using an f-string. While fairly simple, this demonstrates some of the power we tap into by being able to pass functions (and classes) as arbitrary Python obects.

Let’s take this idea a step further to motivate decorators. In this problem, I am maintaining a library that has 3 functions. Of these functions, I want to notify users that I am planning to deprecate (remove) the functions foo and bar.

def foo():
    return 5

def bar():
    return 7

def buzz():
    return 10

In order to notify my users about my deprecation plan, I decided to simply paste a print message into those functions like so:

def foo():
    return 5
print('foo will be deprecated sooon!')

def bar():
    return 7
print('bar will be deprecated soon!')

def buzz():
    return 10
foo will be deprecated sooon!
bar will be deprecated soon!

By writing my code this way, you’ll notice that I am repeating lines of code and I have introduced a possible dissociation between the true name of the function and the name I use in my print message. If I decide to change the name of one of my functions, I need to also remember to change it in the print message as well.

By using this idea of passing functions as objects, we can actually reduce the verbosity of the code eliminate this aforementiond dissociation while maintaining readability.

def deprecate_warning(f):
    print(f'{f.__name__} will be deprecated soon!')

def foo():
    return 5
deprecate_warning(foo)

def bar():
    return 10
deprecate_warning(bar)

def buzz():
    return 10
foo will be deprecated soon!
bar will be deprecated soon!

With this pattern, we no longer depend on writing an explicit deprecation message and instead use a new function deprecate_warning. However, I should point out that this is not yet a true decorator because we are not overwriting the decorated function with the pattern foo = deprecate_warning(foo) as this is what the @ style syntax performs under the hood.

To enable this pattern, we need to return f from deprecate_warning as a passthrough (meaning the decorated function is passed through unchanged).

def deprecate_warning(f):
    print(f'{f.__name__} will be deprecated soon!')
    
    # if we don't do this, we will run into
    #     errors when we try to call foo & bar later!
    return f 

def foo():
    return 5
foo = deprecate_warning(foo)

def bar():
    return 10
foo = deprecate_warning(bar)

def buzz():
    return 10
foo will be deprecated soon!
bar will be deprecated soon!

Once we have our code in the above format, where we overwrite foo with the decorated version of itself- we are ready to use the @ style syntax. Now we can simply do:

def deprecate_warning(f):
    print(f'{f.__name__} will be deprecated soon!')
    
    # if we don't do this, we will run into
    #     errors when we try to call foo & bar later!
    return f 

@deprecate_warning
def foo():
    return 5

@deprecate_warning
def bar():
    return 10

def buzz():
    return 10
foo will be deprecated soon!
bar will be deprecated soon!

In the above example, we actually replaced the function foo with deprecate_warning(foo) because that was what deprecate_warning returned! So with the @ style syntax, the decorated function becomes whatever the decorating function returns. This is the second fundamental concept to understanding decorators. By abstracting this pattern we can start talking about wrappers and how we can use them to prepend or append behaviors to decorated objects.

Function Wrappers#

def wrapper(f):
    print(f'before {f.__name__}')
    f()
    print(f'after {f.__name__}')
    
def foo():
    print(f'inside foo')
    
wrapper(foo)
before foo
inside foo
after foo

A wrapper is conceptually related to a decorator, such that they are both patterns that can be implemented by higher order functions. The above wrapper example is not readily appliable to decorators as the above example makes it seem. Let’s take a look:

def wrapper(f):
    print(f'before {f.__name__}')
    f()
    print(f'after {f.__name__}')

@wrapper
def foo():
    print('inside foo')
before foo
inside foo
after foo

Though the print-out appears to be what we expected, I want to note that I NEVER explicitly executed any function call here! Instead the decorator syntax executed wrapper when foo was defined. This is 1 of the 3 main areas where a decorator can be used to execute code. To put simply, a decorator enables us to run arbitrary code when:

entry point

attainable by

function/class definition

decorator only

prior to function/class call

decorator + wrapper

after function/class call

decorator + wrapper

  1. A function/class is defined = decoratory only

  2. Prior to a function/class being called = decorator + wrapper

  3. After a function/class was called = decorator + wrapper

Out of the above 3, we’ve explored number 1- executing code on function definition. What if I wanted to later call foo() and have something occur either before or after? This is where combining a decorator and wrapper come in handy.

def dec(f):
    print(f'{f} was just defined!')
    def wrapper():
        print(f'before {f} is executed')
        f()
        print(f'after {f} is executed')
    return wrapper

@dec
def foo():
    print(f'inside {foo}')
    
print('functions are done being defined', end='\n'*2)

foo()
<function foo at 0x7f3691e53b50> was just defined!
functions are done being defined

before <function foo at 0x7f3691e53b50> is executed
inside <function dec.<locals>.wrapper at 0x7f3691e53c70>
after <function foo at 0x7f3691e53b50> is executed

By returning a newly defined function wrapper from our decorator dec, we can prepand/append behaviors to our decorated function foo each time it is called. I should also note that it is very common to have your wrapper accept *args, **kwargs as arguments (meaning any number of positional and keyword arguments) and pass those back into the function it’s wrapping f. Behaviorally, this enables us to propagate the call signature of f into the newly defined wrapper function. Without this, if f took some arguments, wrapper would need to be able to take those same exact arguments.

The combination of returning a wrapper from a decorator is probably the most common pattern that we see for the use of decorators in code. However I hope you can see the 3 stages where one can execute code when using the decorator/wrapper pattern.

Summary#

By understanding:

  1. Passing function/class as arguments

  2. Callable wrappers

And the entry points these concepts are tied to

entry point

attainable by

function/class definition

decorator only

prior to function/class call

decorator + wrapper

after function/class call

decorator + wrapper

I hope to provide you with some insight on how to understand decorators from a conceptual level. With this level of understanding, we can begin to unravel some very advanced usages of decorators and what decorator patterns we encounter in the Python ecosystem- though I think I’ll save that for a future post. Talk to you all next week!