pandas Groupby: split-?-combine#

When choosing what groupby operations to run, pandas offers many options. Namely, you can choose to use one of these three:

agg or aggregate
transform
apply

This blog post takes the guesswork out of whether you should agg, transform, or apply your groupby operations on pandas objects.

Reducing vs Non-Reducing Functions#

To better grasp the differences between pandas groupby, agg, transform and apply, we first need to differentiate reducing functions from non-reducing functions.

from IPython.display import Markdown, display 
from pandas import Series, DataFrame, Index

from numpy.random import default_rng

x = Series([1, 2, 3, 4, 5])

x

  1
  2
  3
  4
  5
dtype: int64

Reducing functions N → 1

Reduces the size of the input array to a scalar output.
- sum([1, 2, 3]) → 6 is an example of a reduction. The input [1, 2, 3] is being reduced to a single value 6 by some operation sum
These are commonly aggregation functions (e.g. sum, mean, median)
- Can also be item selections (e.g., first, nth, last)

x.sum()

Non-Reducing functions N → N

Maintains the original size of the input array while transforming the original values
- log10([10, 100, 1000]) → [1, 2, 3] is an example of a non reducing transformation because the output has just as many values as the input
Functions operate “element-wise” (on each element) in an array and return a new value
- e.g., log, log10, pow, cumsum, add, subtract

x.pow(2)

   1
   4
   9
  16
  25
dtype: int64

Groupby Operations#

In pandas, groupby operations work on a “split → apply → combine” basis. This means that the data is first split into groups, some function is then applied to each group, and then those resultant pieces are recombined into the final DataFrame

But, if it’s as simple as the operation depicted above, then why does pandas have 3 options when performing a .groupby operation: .agg, .transform, and .apply?

The reason is in the speed of the operation. By laying out certain assumptions of the outputted shape of the combination step (e.g., N → N or N → 1), pandas can take some shortcuts to ensure that our operations are applied as quickly as possible. As you may have guessed, the resultant shape is very closely tied to whether the function we want to apply is a reducing or a non-reducing function!

Let’s make some data to demonstrate:

rng = default_rng(0)

df = DataFrame(
    index=(idx := Index(["a", "b", "c"], name='Group').repeat(3)),
    data={
        'values1': [34, 25, 20, 10, 12, 1, 3, 0, 7],
        'values2': [39, 41, 15, 20, 25, 31, 10, 9, 4],
    }
)

df

	values1	values2
Group
a	34	39
a	25	41
a	20	15
b	10	20
b	12	25
b	1	31
c	3	10
c	0	9
c	7	4

DataFrame.groupby(…).agg#

.agg only works with functions that reduce.
For user defined functions, .agg operates on each column independently this means that you cannot operate across columns with .agg

When working with multiple columns—as our data set contains—pandas will apply the function to each column within each group separately. This means it cannot use .groupby(…).agg() for values from multiple columns in the function passed to .agg.

There are many ways to pass functions into .agg. One is to pass multiple reduction functions as a list. Note that I receive an output for each of my original columns. To read on how else you can pass in functions to .agg, check out the pandas.core.groupby.DataFrameGroupBy.aggregate documentation

df.groupby('Group').agg([sum, 'mean', lambda s: False])

	values1			values2
	sum	mean	<lambda_0>	sum	mean	<lambda_0>
Group
a	79	26.333333	False	95	31.666667	False
b	23	7.666667	False	76	25.333333	False
c	10	3.333333	False	23	7.666667	False

If we pass a user defined function that does not reduce, we see this error:

df.groupby('Group')['values1'].agg(lambda s: s * 2)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [6], in <cell line: 1>()
----> 1 df.groupby('Group')['values1'].agg(lambda s: s * 2)

File ~/.pyenv/versions/dutc-site/lib/python3.10/site-packages/pandas/core/groupby/generic.py:297, in SeriesGroupBy.aggregate(self, func, engine, engine_kwargs, *args, **kwargs)
    294     return self._python_agg_general(func, *args, **kwargs)
    296 try:
--> 297     return self._python_agg_general(func, *args, **kwargs)
    298 except KeyError:
    299     # TODO: KeyError is raised in _python_agg_general,
    300     #  see test_groupby.test_basic
    301     result = self._aggregate_named(func, *args, **kwargs)

File ~/.pyenv/versions/dutc-site/lib/python3.10/site-packages/pandas/core/groupby/groupby.py:1682, in GroupBy._python_agg_general(self, func, raise_on_typeerror, *args, **kwargs)
   1678 name = obj.name
   1680 try:
   1681     # if this function is invalid for this dtype, we will ignore it.
-> 1682     result = self.grouper.agg_series(obj, f)
   1683 except TypeError:
   1684     if raise_on_typeerror:

File ~/.pyenv/versions/dutc-site/lib/python3.10/site-packages/pandas/core/groupby/ops.py:1081, in BaseGrouper.agg_series(self, obj, func, preserve_dtype)
   1078     preserve_dtype = True
   1080 else:
-> 1081     result = self._aggregate_series_pure_python(obj, func)
   1083 npvalues = lib.maybe_convert_objects(result, try_float=False)
   1084 if preserve_dtype:

File ~/.pyenv/versions/dutc-site/lib/python3.10/site-packages/pandas/core/groupby/ops.py:1109, in BaseGrouper._aggregate_series_pure_python(self, obj, func)
   1105 res = libreduction.extract_result(res)
   1107 if not initialized:
   1108     # We only do this validation on the first iteration
-> 1109     libreduction.check_result_array(res, group.dtype)
   1110     initialized = True
   1112 counts[i] = group.shape[0]

File ~/.pyenv/versions/dutc-site/lib/python3.10/site-packages/pandas/_libs/reduction.pyx:13, in pandas._libs.reduction.check_result_array()

File ~/.pyenv/versions/dutc-site/lib/python3.10/site-packages/pandas/_libs/reduction.pyx:21, in pandas._libs.reduction.check_result_array()

ValueError: Must produce aggregated value

DataFrame.groupby(…).transform#

.transform works with both non-reducing and reducing functions.

A groupby-transform operation ensures that the DataFrame does not change shape after the operation. The original Index remains intact and is used to index the resultant DataFrame.

Non-reducing function

df.groupby('Group').transform(lambda s: s.cumsum())

	values1	values2
Group
a	34	39
a	59	80
a	79	95
b	10	20
b	22	45
b	23	76
c	3	10
c	3	19
c	10	23

Reducing function

However, when a reducing function is used, the outputted value maps back to the shape of the original input to ensure an N → N grouped operation.

df.groupby('Group').transform(sum)

	values1	values2
Group
a	79	95
a	79	95
a	79	95
b	23	76
b	23	76
b	23	76
c	10	23
c	10	23
c	10	23

DataFrame.groupby(…).apply#

groupby(…).apply is the catch-all for pandas groupby operations. For user-defined functions, .apply works as follows:

N → M transformations (including N → 1 and N → N)
It works with entire sub-frames, instead of on a per column per group basis like .agg and .transform

The actual mechanics of .apply are similar to the above, except that pandas struggles to make the same assumptions as it can with the .agg and .transform methods. This enables maximum flexibility for passing user defined functions to a groupby operation, at the cost of some performance speed.

Wrap Up#

That takes us to the end of this post! This should help orient you towards how you can better leverage .agg and .transform in your work, and only use .apply when you really need it.

In summary, you can think of groupby operations with the following table:

Talk to you all next time!

Unconventional Pandas: Colormaps The Central Limit Theorem - Visualized

Recent Posts

Tags