Visualizing Dropped Video Frames#
Welcome back, everyone! This week on Cameron’s Corner, I’m going to get a bit creative. I want to take you through my process for optimizing the (many) Python instruction videos I make.
But, first, I want to let you know about my upcoming seminar, “Arrow & In-Memory Data”! This seminar is designed to provide attendees with a comprehensive understanding of Arrow and its interface with PyArrow, a Python library for working with Arrow data structures.
Now, let’s jump in!
With all of the seminars, courses, and consulting work we’ve had, I have a LOT of recorded video on my computer. Unfortunately, after some wear and tear on the hardware, it has come to my attention that my video was regularly skipping (dropping frames).
Being the data-enthusiast that I am, I wanted to do a quick analysis of these “frame drops”, creating different data visualizations along the way. I want to give you a look into my creative process. We’ll start with a simple chart, then transition into something that communicates a clear message: How many frames are being dropped? How far apart and how frequently do the drops occur? Is there an underlying pattern to them?
Data Sample#
To create the data I’ll be working from, I stepped through a video one frame at a time. I created a text file and typed a “D” when a frame was dropped and typed a “Y” when the frame was correct.
Below, is the results in a string. Wherever we encounter a “y,” we have an “Ok” frame. When there is a “d,” we have a dropped frame. Let’s convert this signal from a string into something easier to work with.
from textwrap import dedent
from pandas import Series
sample = dedent("""
yyydyyydyydydydydyydyyyyyyydyyydyydyyydyyydyyydyyydyydyyydyyyyd
yydydydydydydydydyydyyydyydyyyydyydydydyyydyy
""").strip().replace('\n', '')
sample = Series([letter != 'y' for letter in sample], dtype=bool)
sample.head()
0 False
1 False
2 False
3 True
4 False
dtype: bool
With the above Boolean representation, we can easily plot the signal as a stepwise line:
%config InlineBackend.print_figure_kwargs = {'bbox_inches':None}
%config InlineBackend.figure_formats = ['svg']
%matplotlib inline
from matplotlib.pyplot import subplots, rc
rc('figure', facecolor='white')
rc('axes.spines', top=False, right=False, left=False)
rc('font', size=16)
fig, ax = subplots(figsize=(10, 3))
ax.step(sample.index, sample, where='post')
ax.set_yticks([0, 1], labels=['Ok', 'Dropped'])
[<matplotlib.axis.YTick at 0x7f415a08a560>,
<matplotlib.axis.YTick at 0x7f415a089f00>]
The use of a stepwise interpolation is to indicate that the signal remains the same until it changes. If we have plotted a normal, linearly-interpolated line then we would see a much more jagged signal that does not correspond to the underlying world.
But, this chart doesn’t tell us much yet. We can see (if we look very closely) that we likely spend more time in the “Ok” state than the “Dropped” state. Or, in other words, we are dropping less frames than we are keeping. But, that is simply an educated guess from looking at the line.
Refine, Refine, Refine#
Take a close look at the above chart. What are your eyes drawn to? My eyes are drawn close to the densely-packed vertical lines (areas of rapid state change), but I’m not focused on the peak or the valleys but the banding of the lines themselves. The funny thing here is that this banding doesn’t quantitatively communicate any information. Qualitatively, we can say that, the tighter the vertical banding is, the more state changes there were. However it is nearly impossible to tell how quickly this was happening without a much more in-depth look at the chart.
Let’s take a moment and think about this representation—if my goal is to communicate information about the dropped frames, then I have completely missed my mark. Let’s take a second attempt at this chart and focus in on the dropped frames.
consecutive = (
sample.pipe(lambda s: s.groupby((s != s.shift()).cumsum()))
.agg(['first', 'count'])
.rename(columns={'first': 'state', 'count': 'duration'})
.replace({'state': {True: 'Dropped', False: 'Ok'}})
.assign(
start=lambda d: d['duration'].shift(fill_value=0).cumsum(),
stop=lambda d: d['start'] + d['duration'],
)
)
fig, ax = subplots(figsize=(10, 3))
ax.step(sample.index, sample, color='gainsboro', where='post')
ax.set_yticks([0, 1], labels=['Ok', 'Dropped'])
dropped = consecutive.query('state == "Dropped"')
ax.hlines([1] * len(dropped), dropped['start'], dropped['stop'], lw=3, color='tab:red');
The above does a good job at moving attention away from the vertical lines and the “Ok” frames (through the use of color), but we aren’t doing a great job of drawing attention to our dropped frames. This is because we haven’t chosen a great chart type for these data. Earlier, I mentioned that the line chart heavily displays the transitions without drawing attention to the total amount of time we spent in a given state.
Instead of using a line chart, we can reach for a Gantt chart to draw attention to the duration of each state, while still communicating information about the transitions.
from flexitext import flexitext
fig, ax = subplots(figsize=(10, 3))
colors = {'Dropped': 'tab:red', 'Ok': 'gainsboro'}
label_colors = colors | {'Ok': 'dimgray'} # darker gray for text
state_order = ['Ok', 'Dropped']
bar_height = .9
for i, s in enumerate(state_order):
group = consecutive.query('state == @s')
ax.broken_barh(
group[['start', 'duration']].to_numpy(),
(i, bar_height),
facecolor=colors[s]
)
ax.set_yticks([0 + bar_height / 2, 1 + bar_height / 2], state_order)
ax.margins(x=0)
ax.yaxis.set_tick_params(left=False)
for label in ax.get_yticklabels():
label.set_color(label_colors[label.get_text()])
title = '<size:large><color:tab:red, weight:bold>Dropped</> vs <color:gray>Ok</> frames</>'
flexitext(
0, 1, s=title, va='bottom',
ax=ax, xycoords='axes fraction'
);
Intermission#
Now this chart is on its way to better highlight how many frames were and were not dropped, but we’re not done yet! This is a great contextual chart, providing background on the dataset itself and its tendencies. Next, I’ll want to create some supporting statistical charts as well as some takeaway statements. So, make sure you stay tuned next week when we take this single chart and turn it into a compound figure by adding more visualizations!
Until next week!
Welcome Back!#
Hello, everyone! Welcome to part 2 of this data visualization journey where we’re analyzing some dropped frames I ran into while troubleshooting my video setup. When we left off last week, I mentioned that the chart I’d made does a good job at laying out the context of the problem. But now we need to do an analysis that answers the question at hand: “how many frames did we drop?” or “what proportion of frames did we drop?”
The aggregation is a single line of pandas:
frame_counts = consecutive.groupby('state')['duration'].sum()
frame_counts.head()
state
Dropped 33
Ok 75
Name: duration, dtype: int64
But, how do we present these data? Well, we can look at the values and emphasize their relationship to the above context by creating a chart that has a similar style:
fig, ax = subplots(figsize=(10, 3))
ax.barh(
state_order, frame_counts.loc[state_order],
color=[colors[s] for s in state_order]
)
ax.yaxis.set_tick_params(left=False)
ax.yaxis.set_tick_params(left=False)
for label in ax.get_yticklabels():
label.set_color(label_colors[label.get_text()])
ax.margins(x=0)
ax.set_label('Frames')
fig, (ax1, ax2) = subplots(2, 1, figsize=(10, 8))
colors = {'Dropped': 'tab:red', 'Ok': 'gainsboro'}
label_colors = colors | {'Ok': 'dimgray'} # darker gray for text
state_order = ['Ok', 'Dropped']
bar_height = .9
for i, s in enumerate(state_order):
group = consecutive.query('state == @s')
ax1.broken_barh(
group[['start', 'duration']].to_numpy(),
(i, bar_height),
facecolor=colors[s]
)
ax1.set_yticks([0 + bar_height / 2, 1 + bar_height / 2], state_order)
ax1.margins(x=0)
ax1.yaxis.set_tick_params(left=False)
title = '<size:large><color:tab:red, weight:bold>Dropped</> vs <color:gray>Ok</> frames</>'
flexitext(
0, 1, s=title, va='bottom',
ax=ax1, xycoords='axes fraction'
);
ax2.barh(state_order, frame_counts.loc[state_order], color=[colors[s] for s in state_order])
ax2.yaxis.set_tick_params(left=False)
ax2.yaxis.set_tick_params(left=False)
for label in ax.get_yticklabels():
label.set_color(label_colors[label.get_text()])
ax2.margins(x=0, y=.2)
ax2.set_xlabel('Frames')
for ax in [ax1, ax2]:
for label in ax.get_yticklabels():
label.set_color(label_colors[label.get_text()])
We’ve certainly added a visualization here, but the elements aren’t effectively playing off of one another. While we have carefully kept our coloring consistent, there are a lot of inconsistencies in this chart, such as the disagreement in their length. Additionally, we need a few more annotations to spruce this chart up, and consider the density of the information we’re communicating.
Welcome Back (again)#
Hello, everyone! Welcome to part 3 of this data visualization journey, where we’re analyzing some dropped frames I ran into while troubleshooting my video setup.
Last week, we managed to add a visualization, but the elements weren’t effectively playing off of one another. While we kept our coloring consistent, there are several inconsistencies in the chart, such as the disagreement in their length. Additionally, a few more annotations would spruce it up and consider the density of the information we’re communicating.
Before we jump into it, I want to let you know about our upcoming seminar series, “Python: How the experts do it!”! Join us for this unique seminar series presented in a “lab-style” format, wherein our experienced instructors will live-code solutions to pre-written problems. We’ll explore Python concepts using engaging games and practical applications, shedding light on the techniques used by Python masters. Up first are Tic-Tac-Toe & Connect Four, and I can’t wait!
Now, back to the topic at hand!
Final Improvements#
We want to make sure our entire visual is understandable and conveys a single, uniform message. In this case, we have axes conveying the same unit of frames but on different scales: the top axis having its data mapped to the underlying timescale and the bottom having a zero baseline for each type of frame. This “same unit, different scale” may confuse our audience and reduce the clarity of our message.
Additionally, we’re using a lot of space here because each of our frame types is separated by their vertical position. We can create a more compact visual by removing this separation. If we turn our lower axis into a horizontally stacked bar chart, we can fix the issue of our data existing on different scales.
Finally, we haven’t said anything specific just yet about the data, so let’s make sure we add a clear message about the degree to which we are dropping video frames!
fig, ax = subplots(1, 1, figsize=(10, 4))
colors = {'Dropped': 'tab:red', 'Ok': 'gainsboro'}
label_colors = colors | {'Ok': 'dimgray'} # darker gray for text
state_order = ['Dropped', 'Ok']
bar_height = .9
# Represent underlying data as closely as possible
for i, s in enumerate(state_order):
group = consecutive.query('state == @s')
ax.broken_barh(
group[['start', 'duration']].to_numpy(),
(1, bar_height),
facecolor=colors[s]
)
# Sort horizontal barchart to keep both charts on the same scale
bc = ax.barh(
[0, 0], frame_counts.loc[state_order],
left=frame_counts.loc[state_order].shift(fill_value=0).cumsum(),
color=[colors[s] for s in state_order],
align='edge', label=state_order,
)
# Annotate the width of each bar in the horizontal barchart
for rect in bc:
color = 'white' if rect.get_label() == 'Dropped' else 'black'
frames_label = ax.annotate(
f'{rect.get_width()} frames',
xy=(1, .5), xycoords=rect,
xytext=(-5, 0), textcoords='offset points',
ha='right', va='center', color=color,
)
ax.annotate(
f'({rect.get_width()/frame_counts.sum():.1%})',
xy=(1, 0), xycoords=frames_label,
ha='right', va='top', color=color,
size='x-small',
)
# Add the endpoint so audience can see how many frames we analyzed
ax.annotate(
consecutive['stop'].max(), xy=(1, .5), xycoords=ax.spines['bottom'],
xytext=(5, 0), textcoords='offset points',
va='center', size='small',
)
# Add a subtitle on top of our Axes
subtitle = flexitext(
s=(
'<size:small>'
'Our sample of 108 video frames revealed that <weight:semibold>31%</> '
'of frames are <color:tab:red>dropped</>.\n'
'meaning about <weight:semibold>1 out of every 3</> '
'frames are <color:tab:red>dropped</>.'
'</>'
),
x=0, y=1, va='bottom',
ax=ax, xycoords='axes fraction'
)
# Put the title right above the subtitle
title = flexitext(
s=(
'<size:large>'
'<color:tab:red, weight:bold>Dropped</> vs <color:gray>Ok</> '
'frames'
'</>'
),
x=0, y=1.1, va='bottom',
ax=ax,
)
title.xycoords = title.anncoords = subtitle # work around due to flexitext API
ax.set_yticks([.5, 1.5], labels=['Sorted', 'Raw'])
ax.margins(x=0)
ax.yaxis.set_tick_params(left=False)
# Inline our xlabel
ax.annotate(
'th frame', xy=(1, .5), xycoords=ax.get_xticklabels()[0],
va='center', size=ax.get_xticklabels()[0].get_size(),
)
fig.subplots_adjust(top=.8)
With that, we’ve created a visualization that communicates a specific point about our data! We have properly annotated our chart, removed any unnecessary “ink,” and distilled down the most important evidence in support of our message.
Wrap Up#
That’s how I think through my visualizations:
First, some quick charts to better understand my data.
Then I select an interesting feature and start highlighting it.
I add annotations to put my “bottom line up front,” and design my charts so that they support that message.
Additionally, I am careful to ensure that I don’t lose any important context in my charts so my audience is able to tie my analysis back to the raw data I worked with.
Hope you enjoyed the final chapter in this blog post. I’ll talk to you all again next week!