Parsing Narwhals Expressions

What's in a Narwhals Expression?

Under the hood, a Narwhals Expression is a composition of a function and some associated metadata. Taking a look at the call signature of the nw.Expr.__init__ method we can confirm that these two pieces are the only arguments involved in the instantiation of an Expression.

import narwhals as nw

help(nw.Expr.__init__)

Help on function __init__ in module narwhals.expr:

__init__(self, to_compliant_expr: '_ToCompliant', metadata: 'ExprMetadata') -> 'None'
    Initialize self.  See help(type(self)) for accurate signature.

How Does Narwhals Use Closures?

Let's take a look at an example of a narwhals.Expr object that stores a simple operation like selecting a single column and incrementing its values by 1.

import pandas as pd
import narwhals as nw

expr = nw.col('a') + 1

Then, we can have a particular backend evaluate this expression like so:

(
    nw.from_native(
        pd.DataFrame({'a': [0, 10, 100], 'b': [0, 10, 100]})
    )
    .with_columns(expr) # the backend evaluates the expression and returns the result
)

┌──────────────────┐
|Narwhals DataFrame|
|------------------|
|        a    b    |
|   0    1    0    |
|   1   11   10    |
|   2  101  100    |
└──────────────────┘

This code almost feels magical, but how does Narwhals track the fact that we've selected a specific column and then incremented it by a specific value? If you take a look at the narwhals.Expr object, you'll see no exposed parameters in sight!

print(
    # The __.*__ methods are not that interesting
    [attr for attr in dir(expr) if not attr.startswith('__')]
)

['_metadata', '_taxicab_norm', '_to_compliant_expr', '_with_aggregation', '_with_elementwise_op', '_with_filtration', '_with_orderable_aggregation', '_with_orderable_filtration', '_with_orderable_window', '_with_unorderable_window', 'abs', 'alias', 'all', 'any', 'arg_max', 'arg_min', 'arg_true', 'cast', 'cat', 'clip', 'count', 'cum_count', 'cum_max', 'cum_min', 'cum_prod', 'cum_sum', 'diff', 'drop_nulls', 'dt', 'ewm_mean', 'fill_null', 'filter', 'gather_every', 'head', 'is_between', 'is_duplicated', 'is_finite', 'is_first_distinct', 'is_in', 'is_last_distinct', 'is_nan', 'is_null', 'is_unique', 'len', 'list', 'map_batches', 'max', 'mean', 'median', 'min', 'mode', 'n_unique', 'name', 'null_count', 'over', 'pipe', 'quantile', 'rank', 'replace_strict', 'rolling_mean', 'rolling_std', 'rolling_sum', 'rolling_var', 'round', 'sample', 'shift', 'skew', 'sort', 'std', 'str', 'struct', 'sum', 'tail', 'unique', 'var']

If these values are not explicitly stored in the Expression object, then Python must have some implicit knowledge about the operation. The implicit solution used here lies within this week's focus: closures.

Opening a Closure

Before we take a look at some Narwhals source code, let's take a look at a very straightforward example of a closure and explore how we can parse variables back out of it.

Let's take a synthetic function named col which returns a function that can later be called:

from typing import Callable

def col(*names: str) -> Callable[[], list[str]]:
    def func():
        # func closes over the `names` variable
        return [*names]
    return func

f = col('a', 'b', 'c')

display(
    f,  # function object
    f() # returned value
)

<function __main__.col.<locals>.func()>

['a', 'b', 'c']

Just by reading the source code, we can deduce that the selected columns are 'a', 'b', and 'c'. But what if we couldn't read the source code? If all we have access to is the variable f, can we figure out what names it was passed without calling it? Clearly Python can access these strings, otherwise it wouldn't be able to return the list with the correct strings.

The tricky part is knowing where to look, for nonlocal variables (like those involved in a closure) we would look in the functions .__closure__ attribute. Then, we can also extract the variable name by inspecting the functions .__code__ attribute.

print(f.__code__.co_freevars[0], '⇒' , f.__closure__[0].cell_contents)

names ⇒ ('a', 'b', 'c')

Though for simplicity's sake, we can just use the inspect.getclosurevars function which also extracts other types of variables as well.

from inspect import getclosurevars

getclosurevars(f)

ClosureVars(nonlocals={'names': ('a', 'b', 'c')}, globals={}, builtins={}, unbound=set())

Inspecting Narwhals Expressions

As I mentioned earlier, a narwhals.Expr is a composition of a function and some associated metadata. If we look more closely at the narwhals.Expr.__init__ you'll notice that we don't immediately store the function that was passed in. Instead we wrap it and store the wrapped function in the _to_compliant_expr attribute.

import narwhals as nw

expr = nw.col('a') + 1

# Access the stored wrapped function
expr._to_compliant_expr

<function narwhals.expr.Expr.__init__.<locals>.func(plx: 'CompliantNamespace[Any, Any]') -> 'CompliantExpr[Any, Any]'>

Since we have access to this function, we can introspect what the original passed argument to_compliant_expr was even though it is not explicitly stored on the narwhals.Expr object.

from inspect import getsourcelines
from textwrap import dedent

lines, _ = getsourcelines(expr._to_compliant_expr)
print(dedent(''.join(lines)))

def func(plx: CompliantNamespace[Any, Any]) -> CompliantExpr[Any, Any]:
    result = to_compliant_expr(plx)
    result._metadata = self._metadata
    return result

We can simply call inspect.getclosurevars on this function object to find the value of to_compliant_expr just like we did in the earlier closure example!

from inspect import getclosurevars

getclosurevars(expr._to_compliant_expr).nonlocals['to_compliant_expr']

<function narwhals.expr.Expr.__add__.<locals>.<lambda>(plx)>

That's the addition operation which was the most recent (last) function call that is added to our "stack" in this expression. However, this is the tip of the iceberg, because now we need to also be able to figure out what function calls were performed before this addition operation.

To do this we need more introspection! Specifically, we need to locate the instance of the narwhals.Expr whose .__add__ method was called.

Thankfully each new Expression closes over the expressions that led up to it. So we can further inspect this <function narwhals.expr.Expr.__add__.<locals>.<lambda>(plx)> function to uncover any function calls that led to its creation.

# the leaf/child expression
expr = nw.col('a') + 1

# extract the `.__add__` function call
most_recent_func = getclosurevars(expr._to_compliant_expr).nonlocals['to_compliant_expr']
most_recent_func

<function narwhals.expr.Expr.__add__.<locals>.<lambda>(plx)>

# extract the parent narwhals.Expr that was closed over in the `.__add__` call
parent_expr = getclosurevars(most_recent_func).nonlocals['self']

# extract the function that exists in the parent narwhals.Expr
parent_func = getclosurevars(parent_expr._to_compliant_expr).nonlocals['to_compliant_expr']
parent_func

<function narwhals.functions.col.<locals>.func(plx: 'Any') -> 'Any'>

With the above, we've effectively traversed the call stack of the narwhals.Expr. We can make this stack explicitly accessible and traverse in the same way they are evaluated like so:

stack = [most_recent_func, parent_func]
while stack:
    print(stack.pop())

<function col.<locals>.func at 0x7f7a946b28e0>
<function Expr.__add__.<locals>.<lambda> at 0x7f7a946b3c40>

Simplified & Summarized

To better summarize this information, let's quickly write our own Expr class.

class Expr:
    def __init__(self, original_callable: Callable):
        def func(): # the arguments don't matter for our purposes
            return original_callable(...)
        self._to_compliant_expr = func

Here it becomes apparent that:

1. Expr takes a function original_callable

2. Expr.__init__.func closes over original_callable

3. Expr.__init__.func is stored as Expr(...)._to_compliant_expr

To then take a given instance of Expr and find the original passed in function to_compliant_expr we then traverse the above in reverse order.

1. Expr._to_compliant_expr → func

2. getclosurevars(func).nonlocals['original_callable'] → original_callable

Then if func closed over another Expr (likely due to method chained calls) we check

• getclosurevars(original_callable).nonlocals['self']

And now we can repeat this process to extract the callable associated with this Expr instance.

Wrap Up

That's all the time we have for this week. If I had to condense everything we discussed here about the Narwhals Expression system it would be this:

1. narwhals.Expr objects store closures that capture the prior expression state.

2. We can recover the original arguments and structure using inspect.getclosurevars.

3. Each new operation wraps the previous expression in a new closure, building a "stack" we can unwind.

Next week, we'll put all of this introspection work to good use: we'll build a parser that takes a narwhals.Expr and reconstructs an explicit call graph, allowing us to turn an opaque chain of closures into a clear, navigable structure.

By unpacking closures and tracing how expressions build on one another, we not only demystify how Narwhals expressions work, but also lay the groundwork for debugging, transforming, or even visualizing them in new ways. I am also coordinating to upstream these ideas to improve the debugability of Narwhals expressions and possibly introduce new features like serializable expressions!

If you've ever wanted to peel back the curtain on Narwhals or just geek out over Python internals, you won't want to miss it.

That's all from me this week. What do you think about Narwhals? Let me know on the DUTC Discord server! I'd love to hear your thoughts on Object Orientation in Python.