About

This notebook is a demonstration of some of capabilities of fastpages with notebooks.

With fastpages you can save your jupyter notebooks into the _notebooks folder at the root of your repository, and they will be automatically be converted to Jekyll compliant blog posts!

Front Matter

The first cell in your Jupyter Notebook or markdown blog post contains front matter. Front matter is metadata that can turn on/off options in your Notebook. It is formatted like this:

# "My Title"
> "Awesome summary"

- toc:true- branch: master- badges: true- comments: true
- author: Hamel Husain & Jeremy Howard
- categories: [fastpages, jupyter]

Setting toc: true will automatically generate a table of contents
Setting badges: true will automatically include GitHub and Google Colab links to your notebook.
Setting comments: true will enable commenting on your blog post, powered by utterances.

The title and description need to be enclosed in double quotes only if they include special characters such as a colon. More details and options for front matter can be viewed on the front matter section of the README.

Markdown Shortcuts

A #hide comment at the top of any code cell will hide both the input and output of that cell in your blog post.

A #hide_input comment at the top of any code cell will only hide the input of that cell.

The comment #hide_input was used to hide the code that produced this.

put a #collapse-hide flag at the top of any cell if you want to hide that cell by default, but give the reader the option to show it:

#collapse-hide
import pandas as pd
import altair as alt

put a #collapse-show flag at the top of any cell if you want to show that cell by default, but give the reader the option to hide it:

#collapse-show
cars = 'https://vega.github.io/vega-datasets/data/cars.json'
movies = 'https://vega.github.io/vega-datasets/data/movies.json'
sp500 = 'https://vega.github.io/vega-datasets/data/sp500.csv'
stocks = 'https://vega.github.io/vega-datasets/data/stocks.csv'
flights = 'https://vega.github.io/vega-datasets/data/flights-5k.json'

Interactive Charts With Altair

Charts made with Altair remain interactive. Example charts taken from this repo, specifically this notebook.

Example 1: DropDown

# single-value selection over [Major_Genre, MPAA_Rating] pairs
# use specific hard-wired values as the initial selected values
selection = alt.selection_single(
    name='Select',
    fields=['Major_Genre', 'MPAA_Rating'],
    init={'Major_Genre': 'Drama', 'MPAA_Rating': 'R'},
    bind={'Major_Genre': alt.binding_select(options=genres), 'MPAA_Rating': alt.binding_radio(options=mpaa)}
)
  
# scatter plot, modify opacity based on selection
alt.Chart(movies).mark_circle().add_selection(
    selection
).encode(
    x='Rotten_Tomatoes_Rating:Q',
    y='IMDB_Rating:Q',
    tooltip='Title:N',
    opacity=alt.condition(selection, alt.value(0.75), alt.value(0.05))
)

Example 2: Tooltips

alt.Chart(movies).mark_circle().add_selection(
    alt.selection_interval(bind='scales', encodings=['x'])
).encode(
    x='Rotten_Tomatoes_Rating:Q',
    y=alt.Y('IMDB_Rating:Q', axis=alt.Axis(minExtent=30)), # use min extent to stabilize axis title placement
    tooltip=['Title:N', 'Release_Date:N', 'IMDB_Rating:Q', 'Rotten_Tomatoes_Rating:Q']
).properties(
    width=600,
    height=400
)

Example 3: More Tooltips

# select a point for which to provide details-on-demand
label = alt.selection_single(
    encodings=['x'], # limit selection to x-axis value
    on='mouseover',  # select on mouseover events
    nearest=True,    # select data point nearest the cursor
    empty='none'     # empty selection includes no data points
)

# define our base line chart of stock prices
base = alt.Chart().mark_line().encode(
    alt.X('date:T'),
    alt.Y('price:Q', scale=alt.Scale(type='log')),
    alt.Color('symbol:N')
)

alt.layer(
    base, # base line chart
    
    # add a rule mark to serve as a guide line
    alt.Chart().mark_rule(color='#aaa').encode(
        x='date:T'
    ).transform_filter(label),
    
    # add circle marks for selected time points, hide unselected points
    base.mark_circle().encode(
        opacity=alt.condition(label, alt.value(1), alt.value(0))
    ).add_selection(label),

    # add white stroked text to provide a legible background for labels
    base.mark_text(align='left', dx=5, dy=-5, stroke='white', strokeWidth=2).encode(
        text='price:Q'
    ).transform_filter(label),

    # add text labels for stock prices
    base.mark_text(align='left', dx=5, dy=-5).encode(
        text='price:Q'
    ).transform_filter(label),
    
    data=stocks
).properties(
    width=700,
    height=400
)

Data Tables

You can display tables per the usual way in your blog:

movies = 'https://vega.github.io/vega-datasets/data/movies.json'
df = pd.read_json(movies)
# display table with pandas
df[['Title', 'Worldwide_Gross', 
    'Production_Budget', 'Distributor', 'MPAA_Rating', 'IMDB_Rating', 'Rotten_Tomatoes_Rating']].head()

Images

Local Images

You can reference local images and they will be copied and rendered on your blog automatically. You can include these with the following markdown syntax:

![](my_icons/fastai_logo.png)

Remote Images

Remote images can be included with the following markdown syntax:

![](https://image.flaticon.com/icons/svg/36/36686.svg)

Animated Gifs

Animated Gifs work, too!

![](https://upload.wikimedia.org/wikipedia/commons/7/71/ChessPawnSpecialMoves.gif)

Captions

You can include captions with markdown images like this:

![](https://www.fast.ai/images/fastai_paper/show_batch.png "Credit: https://www.fast.ai/2020/02/13/fastai-A-Layered-API-for-Deep-Learning/")

Other Elements

GitHub Flavored Emojis

Typing I give this post two :+1:! will render this:

I give this post two :+1:!

Tweetcards

Typing > twitter: https://twitter.com/jakevdp/status/1204765621767901185?s=20 will render this:

Altair 4.0 is released! https://t.co/PCyrIOTcvv
Try it with:

pip install -U altair

The full list of changes is at https://t.co/roXmzcsT58 ...read on for some highlights. pic.twitter.com/vWJ0ZveKbZ
— Jake VanderPlas (@jakevdp) December 11, 2019

Youtube Videos

Typing > youtube: https://youtu.be/XfoYk_Z5AkI will render this:

Boxes / Callouts

Typing > Warning: There will be no second warning! will render this:

Warning: There will be no second warning!

Typing > Important: Pay attention! It's important. will render this:

Important: Pay attention! It’s important.

Typing > Tip: This is my tip. will render this:

Tip: This is my tip.

Typing > Note: Take note of this. will render this:

Note: Take note of this.

Typing > Note: A doc link to [an example website: fast.ai](https://www.fast.ai/) should also work fine. will render in the docs:

Note: A doc link to an example website: fast.ai should also work fine.

Footnotes

You can have footnotes in notebooks, however the syntax is different compared to markdown documents. This guide provides more detail about this syntax, which looks like this:

For example, here is a footnote {% fn 1 %}.
And another {% fn 2 %}
{{ 'This is the footnote.' | fndetail: 1 }}
{{ 'This is the other footnote. You can even have a [link](www.github.com)!' | fndetail: 2 }}

For example, here is a footnote ¹.

And another ²

1. This is the footnote.↩

2. This is the other footnote. You can even have a link!↩

(Edited 03/01/2020, see 'Likelihood' section and end of article)

Introduction

Suppose you have a population of wild animals and you want to estimate the population size. It's impractical to catch all of them, so what do you do? A standard method is 'mark and recapture': catch some animals, mark them, release them, and catch some again. In the second capture, there will be some marked animals and some unmarked (i.e., not seen before). The ratio gives you some information about the population size.

The standard estimator for the population size is the Lincoln index, but it is only applicable in the case of two capture events. What if we want to repeatedly capture, mark, and release, and mark the animals such that we know how many times each has been captured? I'm by no means an expert in frequestist statistical methods, so I don't know what the approach would be in this iterated capture case, but I gather it's complicated. What I'll show in this notebook is how to do Bayesian inference for this problem. If you just want to see some code, skip to the bottom of this notebook.

An example of Bayesian inference of population size for iterated mark-and-recapture is shown in the following video.

The left hand side is a (crude) simulation of animals moving around. During a capture event, a square appears and the animals within are captured. An animal's colour indicates the number of times it has been captured. The right hand side shows the current state of belief about the total population size.

Assumptions

In the following we'll make these simplifying modelling assumptions:

The population size is constant. Animals don't leave or join the population between capture events.
Every animal has an equal probability of being captured, and this probability is independent between capture events.
The total number of animals captured in a given capture event does not depend on the total population size (apart from being upper-bounded by it). This assumption is actually false the animation above, but is often true in mark-and-recapture field work. Note, if the total number of animals captured at each stage did depend on the population size, the total number observed would give us further information about the population size.

Short Bayesian Inference Intro/Recap

In Bayesian inference we assign probabilities to hypotheses to represent our state of belief, and use Bayes' theorem to update the probability of a hypothesis using the probability of an observation given that hypothesis.

Bayes' theorem says that, for any two random variables $H$ and $X$:

$P(H\mid X) \propto P(X \mid H) \cdot P(H)$.

If we let $X$ stand for an observation, and $H$ for a hypothesis, then what this says is: "The probability that a hypothesis is true after making an observation (called the posterior) is proportional to its probability before that observation (called the prior) multiplied by the probability of making that observation if the hypothesis is true (called the likelihood). An important feature of Bayesian inference for our mark-and-recapture problem is that if you make a sequence of observations, you can iteratively apply Bayes' theorem to keep updating your posterior belief.

So, to do Bayesian inference you need to be able to:

Specify a prior distribution over hypotheses, in the absence of observational data.
For a given hypothesis and observation, compute the likelihood, i.e., the probability of having made that observation.
Re-normalize the posterior distribution, so that it sums to 1.

Our Prior Distribution, and Normalization

The normalization step is often the hard part. In our case, we're going to simplify things by using a prior distribution that assigns non-zero probability to only a finite set of hypotheses. Normalization, then, is just a matter of rescaling the posterior probability vector so that it always sums to 1.

E.g., the above animation uses a prior $p(k) = \frac{1}{350}$ for population size $k$ between 100 and 450, and assigns 0 probability otherwise. I.e., before making observations we believe it to be equally likely that the total population size is any value between 100 and 450.

The Likelihood

Suppose we've previously made several captures, and there are $10$ animals that have been captured once, $5$ that have been captured twice, and none that have been captured more than twice. We don't know how many have been captured zero times, but a hypothesis that there are $k$ animals in total says that there are $k-15$ such animals.

So, rephrasing the hypothesis: "The number of animals that have been captured zero, one, and two times is $k-15$, $10$, and $5$", and if we then capture again (sampling without replacement), the number of the newly captured animals that have previously been captured zero, one, or two times follows a multivariate hypergeometric distribution, so this gives us our likelihood.

(Quick probability distribution recap: suppose you have an urn with $N_1$ white balls and $N_2$ black balls, and you sample $M$ times with replacement. The binomial distribution is the probability distribution over the number of white balls you will observe. If you sample instead without replacement, the number of white balls sampled follows a hypergeometric distribution. The multivariate hypergeometric distribution is the generalization to more than two colours of balls, and is the distribution over the vector of counts for each colour that you will observe.)

03/01/2020 Edit: Since writing this article, I've realised that the number of animals captured that had previously been captured one, two, three, etc. times does not give any information about the population size. All of the information comes from the ratio of seen-before to never-seen-before. I'll explain why at the end of the article, but leave it unedited otherwise.

Some Code

First we need a way to compute the likelihood of a given observation for a multivariate hypergeometric distribution. Unfortunately, Scipy only implements the univariate case. Fortunately, we can build the multivariate case out of recursive univariate hypergeometric distributions, i.e., by first computing the likelihood of the number of zero vs greater than zero previous captures, then the number of one vs greater than one, etc.

import numpy as np
from scipy.stats import hypergeom

# The two input vectors here are the number of individuals in the population
# and in the latest capture that have been observed 0, 1, 2, etc. times.
# Each of these numbers is known, except for the population number of individuals
# that have been observed zero times. This is determined by our hypotheses.
def multi_hypergeom_likelihood(observation_vector, population_vector):
    obs_head, *obs_tail = observation_vector
    pop_head, *pop_tail = population_vector
    
    if len(obs_tail) == 0:
        return 1.0
    
    return hypergeom.pmf(
        obs_head,
        sum(population_vector),
        pop_head,
        sum(observation_vector)
    ) * multi_hypergeom_likelihood(obs_tail, pop_tail)

Now let's set up a population and a prior belief over its size.

true_population_size = 350

# A uniform prior distribution
candidate_population_sizes = np.arange(100, 450)
belief = np.ones_like(candidate_population_sizes, dtype=np.float32)
belief /= belief.sum()

# This is to keep track of how many times each member of the population has been seen.
# Note that for convenience we also record the counts for individuals that have never been
# captured before. This information is *not* used when computing likelihoods, since it is
# 'unknown' and implied by the hypothesis.
population_capture_counts = np.zeros(true_population_size, dtype=np.int32)

Here we crudely simulate the process of capturing animals.

def capture(true_population_size):
    number_captured = np.random.randint(30, 61)
    
    captured_indices = np.random.choice(
        true_population_size,
        replace=False,
        size=number_captured
    )
    
    return captured_indices

And here is some code for computing the likelihood for each hypothesis and updating the posterior over hypotheses.

def update_belief(candidate_pop_sizes, belief, population_capture_counts, captured_indices):
    max_capture_count = np.max(population_capture_counts)
    captured_prev_capture_counts = population_capture_counts[captured_indices]
    
    # Compute a vector of the number of individuals in the population
    # / latest capture that have been previously observed 0, 1, 2, etc.
    # times. Each of these is a known quantity, except for the
    # number of animals in the population that have previously
    # not been observed. This value will be filled in according to each
    # hypothesis before computing likelihoods.
    observation_vector = [sum(captured_prev_capture_counts == i) for i in range(max_capture_count + 1)]
    true_population_vector = [sum(population_capture_counts == i) for i in range(max_capture_count + 1)]
    
    for k_i, k in enumerate(candidate_pop_sizes):
        hypothesized_pop_vector = np.copy(true_population_vector)
        hypothesized_pop_vector[0] = k - np.sum(true_population_vector[1:])
        
        likelihood = multi_hypergeom_likelihood(observation_vector, hypothesized_pop_vector)
        belief[k_i] *= likelihood

    belief[~np.isfinite(belief)] = 0
    belief /= belief.sum()
    return belief

Now let's simulate eight rounds of capture, mark, and release, and plot the posterior belief over the population size at each step.

import matplotlib.pyplot as plt

# There will be 5 capture events
for observation_i in range(8):

    # Capture some animals
    captured_idx = capture(true_population_size)

    # Update posterior based on how many times each had been seen before
    belief = update_belief(
        candidate_population_sizes,
        belief,
        population_capture_counts,
        captured_idx
    )

    # Update the population counts
    population_capture_counts[captured_idx] += 1
    
    # Plot the posterior
    print(f'Capture event number {observation_i + 1}')
    plt.plot(candidate_population_sizes, belief)
    plt.xlabel('Population size'); plt.ylabel('Posterior belief'); plt.show()

Capture event number 1

Capture event number 2

Capture event number 3

Capture event number 4

Capture event number 5

Capture event number 6

Capture event number 7

Capture event number 8

03/01/2020 edit: The number of times an animal has been seen adds no further information

It turns out that during a capture event, the number of previously seen animals that had been previously seen once, twice, three times, etc., does not give any additional information about the total population size. In retrospect, I'm not sure why I believed that it would; it seems intuitive that it wouldn't.

To simplify the discussion, let's forget the capture-recapture example and think of the conceptually simpler problem of sampling coloured balls from urns.

Consider an urn containing balls of $c$ different colours, where there are $K_i$ balls of colour index $i$, and the total number of balls is $N = \sum_{i=1}^c K_i$.

Consider the case that the $K_i$ are known for $i > 1$, and $K_1$ is unknown.

Suppose we sample without replacement $n$ times, and in our sample there are $k_i$ balls of colour $i$, and so $n = \sum_{i=1}^c k_i$.

If we have a prior over $K_1$, we can use the probability of having seen this sample given different values of $K_1$ to update our state of belief over its value. This is analogous to what we did in the article above.

This likelihood comes from a multivariate hypergeometric distribution, and is given by:

$p_1(k_1, \ldots, k_c; K_1) = \frac{ \prod_{i=1}^c {K_i \choose k_i} }{ N \choose n } $.

What if instead of computing the likelihood of observing the given number of balls of each colour {$k_1, \ldots, k_c$}, we instead just computed the likelihood of having observed the given number of balls that are colour 1 in the sample? This likelihood comes from a univariate hypergeometric distribution, and is given by:

$p_2(k_1, \ldots, k_c; K_1) = \frac{ {K_1 \choose k_1} {N - K_1 \choose n-k_1} }{ N \choose n } $.

But the ratio of these two is

$\frac{p_2(k_1, \ldots, k_c; K_1)}{p_1(k_1, \ldots, k_c; K_1)} = \frac{ {K_1 \choose k_1} {N - K_1 \choose n-k_1} }{ \prod_{i=1}^c {K_i \choose k_i} } = \frac{ { \sum_{i=2}^c K_i \choose \sum_{i=2}^c k_i} }{ \prod_{i=2}^c {K_i \choose k_i} },$

which does not depend on $K_1$; the effect of using the multivariate likelihood over the univariate is just to multiply by a constant, having no effect on the posterior.

The Special Case, the Bernoulli Distribution

Suppose $\eta$ is the log odds, $\log\left(\frac{p}{1-p}\right)$ of some event that occurs with probability p, i.e., of a Bernoulli random variable with parameter $p$.

The local linearization of the softplus function, $s(\eta) = \log(1+\exp\eta)$ at any given value for $\eta$ has the following interesting properties:

its slope is the corresponding probability $p$.
its y-intercept is the entropy of the Bernoulli distribution that has probability $p$.

The first is true because $s^{\prime}(\eta) = \frac{1}{1 + \exp(-\eta)}$, the logistic sigmoid (inverse-log-odds) function.

The second can be found be found by expressing the entropy in terms of the log-odds: \begin{align} -p\log p - (1-p)\log (1-p) &= -p (\log p - \log (1-p)) - \log (1-p) \\ &= -p (\log \frac{p}{1-p}) - \log (1-p) \\ &= s(\eta) - s^{\prime}(\eta)\cdot \eta \quad . \end{align}

Plots

The following plots show the entropy, via the y-intercept of the linearization of the softplus function, of 5 Bernoulli distributions.

import matplotlib.pyplot as plt
import numpy as np

def softplus(eta):
    return np.log(1 + np.exp(eta))

def sigmoid(eta):
    return 1 / (1 + np.exp(-eta))

xs = np.linspace(-3, 3, 50)
etas = [-3, -2, 0.0, 2, 3]

fig, axes = plt.subplots(1, len(etas), figsize=(15, 4))
for plot_i, (eta, ax) in enumerate(zip(etas, axes)):
    ax.plot(xs, softplus(xs), label='softplus', c='C0')
    ax.plot(xs, sigmoid(eta) * xs + softplus(eta) - sigmoid(eta) * eta, label='linearization', c='C1')
    ax.axvline(x=0.0, c='k', alpha=0.3)
    ax.axhline(y=0.0, c='k', alpha=0.3)
    ax.axvline(x=0.0, ymin=0.3333, ymax=(softplus(eta) - sigmoid(eta) * eta + 1)/3, c='r')
    ax.scatter([0], [softplus(eta) - sigmoid(eta) * eta], c='r', s=80, label='entropy')
    ax.axhline(y=0.0, xmin=0.5, xmax=(eta + 3)/6, c='g')
    ax.scatter([eta], [0], c='g', s=80, label='log odds')
    ax.axvline(x=1.0, ymin=(softplus(eta) - sigmoid(eta) * eta + 1)/3, ymax=(softplus(eta + 1) - sigmoid(eta + 1) * (eta + 1) + 1)/3, c='r')
    ax.scatter([0], [softplus(eta) - sigmoid(eta) * eta], c='r', s=80, label='entropy')
    ax.set_xlim((-3, 3))
    ax.set_ylim((-1, 2))
    ax.set_xlabel('Log odds')
    if plot_i == 0:
        ax.legend()

The General Case, Exponential Families

More generally, consider an exponential family. This is a parameterized distribution of the form:

$$ p(x; \eta) = \exp\left( \eta\cdot T(x) + k(x) - A(\eta) \right) \quad . $$

Here, $\eta$ is a vector of parameters, $T(x)$ is a vector of sufficient statistics for the distribution, $k(x)$ is called the carrier measure, and $A(\eta)$ is a normalizer that makes the distribution integrate to 1.

For some popular distributions the carrier measure is zero. This is true of the Gaussian distribution and the Bernoulli distribution. Then we have

$$ p(x; \eta) = \exp\left( \eta\cdot T(x) - A(\eta) \right) \quad , $$

and the entropy of the distribution is given by

\begin{align} H(p) &= A(\eta) \int p(x; \eta) \text{dx} - \eta \int T(x) p(x; \eta) \text{dx} \\ &= A(\eta) - \eta \cdot \mathbb{E}_p\left[ T(x) \right] \quad . \end{align}

One useful property of exponential families is that the gradient of the log normalizer $A$ is equal to the expected value of the sufficient statistics: $\mathbb{E}_p\left[ T(x) \right] = \nabla A(\eta)$, so the entropy is equal to:

\begin{align} H(p) = A(\eta) - \eta \cdot \nabla A(\eta) \quad . \end{align}

This means that if we locally linearize, at some value of the parameters $\eta$, the log normalizing function $A$ of an exponential family with carrier measure zero, the resulting hyperplane has slope equal to the expected value (with respect to the distribution) of the sufficient statistics, and has a y-intercept equal to the entropy of the distribution.

In this post I'll show how we can train an ensemble of neural network classifiers cooperateively in a largely decoupled way, such that we can distribute the training of the ensemble over a potentially very large cluster with low communication overhead.

Introduction

The gist is to mix manual and automatic differentiation, as I described in a previous post, such that our framework can operate independently on each node without requiring any knowledge of the larger model/computational cluster. I made use of this trick in some recent work on the LAMBDA project (EPSRC reference EP/N035127/1) at the University of Manchester.

By training an ensemble cooperatively, I mean that we'll tune the parameters of the networks by minimizing the cross entropy between the ensemble predictions and the target, rather than training each network individually by minimizing the cross entropy between that network's predictions and the target. This is an idea that's gaining some traction. See for example:

Joint Training of Neural Network Ensembles, Webb, Reynolds et al. (arxiv.org/abs/1902.04422)
Coupled Ensembles of Neural Networks, Dutt, Pellerin, Quénot (arxiv.org/abs/1709.06053)
Born Again Neural Networks, Furlanello, Lipton et al. (arxiv.org/abs/1805.04770)

It might be more natural to think of the ensemble as being a single network with a modular structure, since the individual members of the ensemble are not encouraged to individually fit the data at all. The following figure shows the structure of the ensemble/modular network.

Suppose we have $M$ neural networks indexed by $j$ which, for input ${\bf{x}}$, have a a vector of logits $f^{(j)}({\bf x})$ and of prediction probabilities $q^{(j)}({\bf x}) = \sigma(f^{(j)}({\bf x}))$, where $\sigma$ is the softmax function. We'll use as our ensemble predictions $\bar{q}({\bf x})$ the normalized geometric mean of the individual model predictions, i.e., a Product of Experts: $$ \bar{q}_k({\bf x}) = \frac{1}{Z} \prod_{j=1}^M q_k^{(j)}({\bf x}) ^{1/M}\quad, $$ where $Z$ is a normalization term and the index $k$ indicates the $k$th class. Note that the ensemble prediction $\bar{q}({\bf x})$ is equal to the softmax function ${\sigma}$ applied to the arithmetic mean of the individual network logits: $$ \bar{q}({\bf x}) = \sigma\left(\frac{1}{M} \sum_{j=1}^M f^{(j)}({\bf x}) \right) \quad . $$

Now I'll show two equivalent methods to train such an ensemble. The first is the usual way, defining a single computational graph. For the second method we'll define a seperate computational graph for each ensemble member, and these can be distributed over computational nodes with minimal communication overhead. A seperate instance of PyTorch will run on each computational node, without any given instance knowing about the whole computational graph/model structure.

Here are the usual imports:

#collapse-show
import torch
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

Now let's set up a data loader for the training data from MNIST.

batch_size = 200
learning_rate = 0.5
momentum = 0.9

use_cuda = torch.cuda.is_available()
device = torch.device('cuda' if use_cuda else 'cpu')
kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}

train_loader = torch.utils.data.DataLoader(
        datasets.MNIST('./data', train=True, download=True,
                       transform=transforms.Compose([
                           transforms.ToTensor(),
                           transforms.Normalize((0.1307,), (0.3081,))
                       ])),
        batch_size=batch_size, shuffle=True, **kwargs)

Now let's define the invididual network architectures. Although they could be heterogeneous, here we'll use the same very small convolutional network architecture for each member of the ensemble.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, kernel_size=3, stride=2)
        self.conv2 = nn.Conv2d(20, 40, kernel_size=3, stride=2)
        self.fc1 = nn.Linear(40, 20)
        self.fc2 = nn.Linear(20, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.avg_pool2d(x, 6)
        x = x.view(-1, 40)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

Single Computational Graph Method

Below is the more typical, naive way to train our ensemble, by defining it as a single computational graph, defining a loss function, and then doing backpropagation in order to set the network parameters to minimize that loss function.

#collapse-show
def train_single_computational_graph(epochs=1,
                                     ensemble_size=1,
                                     seed=None):
    criterion = nn.CrossEntropyLoss()
    if seed is not None:
        torch.manual_seed(seed)

    # Initialize M models
    models = [Net().to(device) for i in range(ensemble_size)]
    # Initialize a single optimizer
    optimizer = optim.SGD(nn.ModuleList(models).parameters(),
                          lr=learning_rate,
                          momentum=momentum)

    losses = []
    for epoch in range(epochs):
        for data, target in train_loader:
            data, target = data.to(device), target.to(device)

            optimizer.zero_grad()
            ensemble_logits = torch.mean(
                torch.stack([model.forward(data) for model in models]),
                dim=0)
        
            # Define a loss function for the ensemble to be minimized
            loss = criterion(ensemble_logits, target)
            loss.backward()
            optimizer.step()
        
            losses.append(loss.item())
            
    return losses

Multiple Computational Graphs Method

An alternative approach is to do the first step of backpropagation manually (as discussed in a previous post), and note that $$ \frac{\partial L}{\partial f_k^{(j)}} = \bar{q}_k - p_k \quad, $$ where $L$ is the ensemble cross entropy loss and $p$ is a one-hot encoding of the target. We can see from this that all of the interaction between ensemble members during training can be achieved by distributing the ensemble prediction vector $\bar{q}$ to $M$ seperate PyTorch instances, each of which handles a seperate ensemble member, and then manually setting $\frac{\partial L}{\partial f_k^{(j)}}$ in each of them as above. This constitutes an alternative, equivalent training procedure which, in more detail, works as follows.

Set up a separate computational graph and optimizer for each network in the ensemble, possibly each on a seperate computational node.
For each iteration:
- Do a forward pass of each, computing the logits for that network.
- Take the arithmetic mean of the logits and apply the softmax function to get the ensemble predictions $\bar{q}$. If the ensemble members are on seperate computational nodes, this will require communication of the logits.
- Do a backward pass of each graph, starting at the logit layer, with the gradient explicitly set to $(\bar{q} - p) / (\text{ensemble_size} * \text{batch_size})$. If the ensemble members are on seperate computational nodes, this will require communication of the ensemble prediction probability vector.

This is done in the following code. I've defined a $\texttt{ComputationalNode}$ class to emphasize the fact that each sub-model and optimizer could be run on a separate node in a computational cluster running a separate instance of PyTorch; the PyTorch framework doesn't need to know anything about our wider computational cluster. We need to handle a small amount of communication: each node needs to send its $B\times K$ logits per iteration, where $B$ is the batch size and $K$ is the number of classes, and then needs to receive the $B\times K$ ensemble prediction probabilities.

#collapse-show
class ComputationalNode():
    
    # Initialize a model and an optimizer for this node
    def __init__(self, ensemble_size):
        self.model = Net().to(device)
        self.optimizer = optim.SGD(self.model.parameters(),
                                   lr=learning_rate,
                                   momentum=momentum)
        self.ensemble_size = ensemble_size
        
    # Do the forward pass for this node, returning the logits
    def forward_pass(self, data):
        self.optimizer.zero_grad()
        self.logits = self.model.forward(data)
        return self.logits.detach()
    
    # Do a backward pass starting at the logit layer using the ensemble predictions
    def backward_pass(self, target, ensemble_prediction):
        self.logits.backward(1/(batch_size * self.ensemble_size) * (ensemble_prediction - target))
        self.optimizer.step()


# Get a one-hot encoding of the target
def get_one_hot_(target, target_one_hot):
    target_one_hot.zero_()
    target_one_hot.scatter_(1, torch.unsqueeze(target.data, 1), 1)

    
def train_multiple_computational_graphs(epochs=1,
                                        ensemble_size=1,
                                        seed=None):
    criterion = nn.CrossEntropyLoss()
    softmax = nn.Softmax(dim=-1)
    target_one_hot = torch.FloatTensor(batch_size, 10).to(device)
    if seed is not None:
        torch.manual_seed(seed)

    # Setup the M computational nodes with a model and optimizer each
    nodes = [ComputationalNode(ensemble_size) for i in range(ensemble_size)]

    losses = []
    for epoch in range(epochs):
        for data, target in train_loader:
            data, target = data.to(device), target.to(device)
            get_one_hot_(target, target_one_hot)
            
            # Do the forward pass for each node
            # Gather the logits
            ensemble_logits = torch.mean(
                torch.stack([node.forward_pass(data) for node in nodes]),
                dim=0)

            # Compute the ensemble prediction
            ensemble_prediction = softmax(ensemble_logits)
            
            # Do the backward pass for each node
            for node in nodes:
                node.backward_pass(target_one_hot, ensemble_prediction)
        
            loss = criterion(ensemble_logits, target)
            losses.append(loss.item())
            
    return losses

Running the two training procedures from the same initial state and plotting the ensemble loss over time, we see that initially it's pretty much exactly the same. After a while, small numerical differences seem to accumulate.

losses_single_graph = train_single_computational_graph(epochs=1,
                                                       ensemble_size=10,
                                                       seed=900)
losses_multiple_graphs = train_multiple_computational_graphs(epochs=1,
                                                             ensemble_size=10,
                                                             seed=900)

xs = np.arange(len(losses_single_graph))
plt.plot(xs, losses_single_graph, label='Single computational graph')
plt.plot(xs, losses_multiple_graphs, label='Multiple computational graphs')
plt.xlabel('Iteration')
plt.ylabel('Loss')
plt.legend()

<matplotlib.legend.Legend at 0x7fe7cbeb20f0>

Summary

We've seen in this post a way to distribute the joint training of an ensemble over a computational cluster with low communication overheads, and without our deep learning framework having to know about the cluster and manage communication across it.

One obvious optimization to the distributed training procedure described above is to set the nodes up in a tree topology to minimize the total amount of communication in getting the individual network logits up to the aggregation node. Another optimization is to pipeline things, and have each node do the forward pass of the next minibatch while it waits for the ensemble predictions to come back allowing the computation of the backward pass of the previous minibatch, rather than blocking.

What's potentially very nice about this procedure is its simplicity and the small amount of communication required. Although it's comparing apples to oranges, compare to batch parallelism, where you duplicate a single model across the computational nodes and each node computes parameter gradients on a very small minibatch. The parameter gradients are then shared and the mean is taken. The communication requirements for this procedure are that each node must send and receive a gradient for every parameter. For, e.g., a Wide ResNet 28-12, this requires each node sending and receiving 52.5 million values per iteration. Batch parallelism also restricts our model size to what can fit in memory on a single node. The procedure described in this post allows us to train modular/ensemble models of unlimited size, with each computational node sending and receiving $B\times K$ values per iteration, where $B$ is the batch size and $K$ is the number of classes. E.g., for a 10 class problem with batch size of 100, we communicate only 1000 values per iteration.

Gavin Brown and I gave a talk today at the 2018 Arm Research Summit. This is a talk about our work with Mikel Luján, Henry Reeve, and Charles Reynolds in training modular ensembles of neural networks.

This was a machine learning talk aimed at a non-machine learning audience, and we touched on the robustness of the trained ensembles to dropping out models and the largely decoupled nature of the training, which allows parallel training of the ensemble with low communication overheads.

The purpose of this post is to demonstrate that sometimes while using modern deep learning frameworks such as PyTorch or Tensorflow it's useful to not rely wholly on automatic differentiation.

The example application I'll use is regression where the labels/targets, conditional on the input, are sampled from an exponential family distribution, and where we train the network by minimizing the negative log-likelihood of the data. I.e., we'll deal with non-linear Generalized Linear Models (GLMs), or GLMs with learned representations. This encompasses regression with squared loss, Poisson regression, and classification with cross-entropy loss, the three examples I'll use in this post.

I'll show that by doing part of the backpropagation manually, we can avoid explicitly specifying a loss function, and the only thing we'll have to do to switch between label distributions is change the activation function used on the final layer. I'll use PyTorch, but the following can be achieved in TensorFlow.

Setting up Synthetic Datasets

First we need some data. Inputs will be scalar. For regression with squared loss, we'll fit a simple sin wave (with Gaussian noise). For binary classification and Poisson regression we'll fit appropriate transformations of the same data with appropriate error distributions. (Don't worry too much about the code in this block; you can skip right ahead to the plots of the data immediately below.)

#collapse-show
import numpy as np
import matplotlib.pyplot as plt
import torch
%matplotlib inline

num_examples = 400

X = np.random.random(num_examples)
X1 = torch.unsqueeze(torch.tensor(X, dtype=torch.float32), 1)
y = np.sin(10 * X)

# Labels for regression with Gaussian noise
gaussian_regression_y = np.random.normal(loc=y, scale=0.2)

# Labels for binary classification (Categorical noise)
class_1_probabilities = 1 / (1 + np.exp(-3.5 * y))
classification_y = np.random.binomial(1, p=class_1_probabilities)
classification_y_one_hot = np.zeros((num_examples, 2))
classification_y_one_hot[np.arange(num_examples), classification_y] = 1

# Labels for Poisson regression
lambdas = 2 * np.exp(y)
poisson_regression_y = np.random.poisson(lam=lambdas)

from collections import OrderedDict

datasets = OrderedDict()
datasets['Gaussian regression'] = {'data': torch.unsqueeze(torch.tensor(gaussian_regression_y, dtype=torch.float32), 1), 'plotting_data': gaussian_regression_y}
datasets['Classification'] = {'data': torch.tensor(classification_y_one_hot, dtype=torch.float32), 'plotting_data': classification_y}
datasets['Poisson regression'] = {'data': torch.unsqueeze(torch.tensor(poisson_regression_y, dtype=torch.float32), 1), 'plotting_data': poisson_regression_y}

def plot_data(regression_type, X, y, predictions=None):
    plt.scatter(X, y, s=80, label="True labels", alpha=0.2)
    if predictions is not None:
        if regression_type == "Classification":
            predictions = np.argmax(predictions, axis=1)
        plt.scatter(X, predictions, s=10, label="Predictions")
    plt.xlabel("x")
    plt.ylabel("y")
    plt.title("{} data".format(regression_type))
    plt.legend()

fig = plt.figure(figsize=(17,4.4))
for data_i, dataset_key in enumerate(datasets.keys()):
    data = datasets[dataset_key]['plotting_data']
    fig.add_subplot(1, 3, data_i + 1)
    plot_data(dataset_key, X, data)

Defining the Network

Now we'll define a simple, small feed-forward neural network with dense connectivity and ReLU activation functions. We'll use the same neural network for each of our regression problem types.

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    
    def __init__(self, output_dim=1):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1, 30)
        self.fc2 = nn.Linear(30, 20)
        self.fc3 = nn.Linear(20, output_dim)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

A Useful Property of Natural Exponential Family Distributions

The Gaussian, Categorical, and Poisson distributions are all instances of the natural exponential family (a subset of the exponential family) of distributions. This means that their probability functions can be expressed as $$q(y) = h(y) \exp{(\eta \cdot y - A(\eta))} \quad ,$$

where $\eta$ is called the natural parameter of the distribution and $A$, the log-partition function, simply normalizes the probability function such that it sums/integrates to $1$. For each function in the exponential family, there exists a canonical link function $f$ which gives the relationship between the natural parameter $\eta$ and the mean of the distribution: $$\mathbb{E}_q[y] = \sum y \cdot q(y) = f^{-1}(\eta) \quad .$$

For example, for labels following a Gaussian distribution, the inverse link function is the identity function. For the Categorical distribution, it's the softmax function (in which case $\eta$ is the vector of logits). For the Poisson distribution, it's the exponential function.

For each of the regression problems dealt with in this post (Gaussian, Categorical, Poisson), the label $y$ is, conditional on the input $x$, sampled from a natural exponential family distribution. I.e., there is some function $\eta(x)$ such that the label $y$ for input $x$ has probability function $$q(y \mid \eta(x)) \quad .$$

Often, what we want to estimate is, conditional on the input $x$, the expected value of the label $\mathbb{E}_q[y]$. Call this estimate $\hat{y}(x)$. This will be the (post-activation) output of our neural network. Suppose we use the inverse link function $f^{-1}$ as the activation function of the final layer of the network. In this case, the pre-activation final layer will be an estimate of the natural parameter, which we'll call $\hat{\eta}(x)$. (I.e., we're talking about fitting Generalised Linear Models, but where the natural parameter estimate $\hat{\eta}$ is a nonlinear function of the inputs.)

Suppose we use the negative log-likelihood of the true labels as a loss function $L$. For a single example with input $x$ and label $y$:

$$L = - \ln q(y \mid \hat{\eta}(x)) = - \ln h(y) - \hat{\eta}(x) \cdot y + A(\hat{\eta}(x)) \quad .$$

In order to do parameter updates by gradient descent, we need the derivatives of the loss with respect to the network parameters, which can be decomposed by the chain rule: $$\frac{\partial L}{\partial \theta} = \frac{\partial L}{\partial \hat{\eta}} \frac{\partial \hat{\eta}}{\partial \theta} \quad, $$

where $\theta$ is a particular network parameter. For every natural exponential family label distribution, the derivative of this loss with respect to the natural parameter is the same:

$$\frac{\partial L}{\partial \hat{\eta}} = \mathbb{E}_\hat{\eta}[y] - y = \hat{y} - y \quad . $$

The upshot of this is that instead of explicitly defining the loss function $L$ to be the negative log-likelihood function for the relevant label distribution and doing backpropagation from the loss, we can instead define $\partial L / \partial \hat{\eta} = \hat{y} - y$ (implicitly defining the loss by our choice of activation function on the final layer) and start backpropagation from the natural parameter estimate layer. Essentially we're doing one step of the backpropagation manually, and relying on auto-differentation for the rest.

An Example with Gaussian Distributed Labels

In the following code, we fit the Gaussian distributed data by explicitly specifying and minimising a mean-squared error loss function (equivalent up to irrelevant constants to the negative log-likelihood for a Gaussian target distribution). We won't worry about evaluating on a validation set.

import torch.optim as optim

torch.manual_seed(500)

net = Net()

y = datasets['Gaussian regression']['data']

optimizer = optim.SGD(net.parameters(), lr=0.2)
loss_function = nn.MSELoss()

for i in range(5000):
    optimizer.zero_grad()
    eta_hat = net(X1)
    y_hat = eta_hat
    loss = 0.5 * loss_function(y_hat, y)
    
    loss.backward()
    optimizer.step()
    if i % 500 == 0:
        print("Epoch: {}\tLoss: {}".format(i, loss.item()))
        
plot_data("Gaussian regression", X, y, y_hat.detach())

Epoch: 0	Loss: 0.23857589066028595
Epoch: 500	Loss: 0.13250748813152313
Epoch: 1000	Loss: 0.07796521484851837
Epoch: 1500	Loss: 0.047447897493839264
Epoch: 2000	Loss: 0.032297104597091675
Epoch: 2500	Loss: 0.02540348283946514
Epoch: 3000	Loss: 0.02224355936050415
Epoch: 3500	Loss: 0.02245643362402916
Epoch: 4000	Loss: 0.022122113034129143
Epoch: 4500	Loss: 0.01919456571340561

Compare the above to the result of running the following code, in which instead of doing backpropagation from the loss, we do backpropagation from the natural parameter prediction $\hat{\eta}$ ($\texttt{eta}$ in the code), while setting the accumulated backprop gradient explicitly to

$$\frac{1}{\text{batch_size}} * (\hat{y} - y) \quad.$$

Note that we don't need to specify a loss function at all in the following, and we do so only so that the loss can be reported. For optimisation purposes, the loss function has been implicitly set to the negative log-likelihood for the Gaussian distribution by choosing the appropriate inverse link function (the identity function, in this case).

torch.manual_seed(500)

net = Net()

optimizer = optim.SGD(net.parameters(), lr=0.2)
loss_function = nn.MSELoss()

for i in range(5000):
    optimizer.zero_grad()
    eta_hat = net(X1)
    y_hat = eta_hat
    
    # Specifying the loss function is not strictly necessary; it's done here so that the value can be reported
    loss = 0.5 * loss_function(y_hat, y)
    
    eta_hat.backward(1.0/num_examples * (y_hat - y))
    optimizer.step()
    if i % 500 == 0:
        print("Epoch: {}\tLoss: {}".format(i, loss.item()))
        
plot_data("Gaussian regression", X, y, y_hat.detach())

Epoch: 0	Loss: 0.23857589066028595
Epoch: 500	Loss: 0.13250748813152313
Epoch: 1000	Loss: 0.07796521484851837
Epoch: 1500	Loss: 0.047447897493839264
Epoch: 2000	Loss: 0.032297104597091675
Epoch: 2500	Loss: 0.02540348283946514
Epoch: 3000	Loss: 0.02224355936050415
Epoch: 3500	Loss: 0.02245643362402916
Epoch: 4000	Loss: 0.022122113034129143
Epoch: 4500	Loss: 0.01919456571340561

We achieve exactly the same results as when explicitly specifying the loss function.

The General Case

The following code demonstrates how easy it is to switch between different types of regression in this way. We pass through the main loop three times, once for regression with Gaussian distributed labels, once for classification, and once for regression with Poisson distributed labels. The only differences between these cases (marked "# ***" in the code) are:

Loading the appropriate data
Setting the network output dimension (2 for binary classification, 1 for the regression examples)
Setting the final layer activation function to be the appropriate inverse canonical link function, which implicitly sets the loss to be minimised to be the negative log-likelihood for the corresponding distribution

datasets['Gaussian regression'].update({'final layer activation': lambda x: x, 'output_dim': 1})
datasets['Classification'].update({'final layer activation': nn.Softmax(dim=1), 'output_dim': 2})
datasets['Poisson regression'].update({'final layer activation': torch.exp, 'output_dim': 1})

fig = plt.figure(figsize=(17,4.4))

for regression_type_i, regression_type in enumerate(datasets.keys()):
    # *** Difference 1: data loading
    y = datasets[regression_type]['data']
    plotting_y = datasets[regression_type]['plotting_data']
    
    # *** Difference 2: setting the network output dimension
    net = Net(output_dim = datasets[regression_type]['output_dim'])
    
    optimizer = optim.SGD(net.parameters(), lr=0.2)
    
    for i in range(5000):
        optimizer.zero_grad()
        eta_hat = net(X1)
        
        # *** Difference 3: The inverse of the canonical link function for the
        # label distribution is used as the final layer activation function.
        y_hat = datasets[regression_type]['final layer activation'](eta_hat)
    
        # Using the appropriate activation above means that the following results in
        # implicitly minimizing the negative log-likelihood of the true labels
        eta_hat.backward(1.0/num_examples * (y_hat - y))
        optimizer.step()
        
    fig.add_subplot(1, 3, regression_type_i + 1)
    plot_data(regression_type, X, plotting_y, y_hat.detach())

Why?

So what are the advantages of this approach?

In the example given here, it means not having to worry about the implementation of the loss function; while modern frameworks such as PyTorch and TensorFlow have efficient and numerically stable implementations of loss functions that are (in terms of optimisation) equivalent to the negative log-likelihood of the most common label distributions, if you want to do regression with a less common label distribution you'll have to write the loss function yourself. For many distributions, there'll be numerical/precision issues along the way.

If your unusual label distribution is a member of the natural exponential family (and there's a good chance that it is), then you can take the approach described above. You'll still need to implement the appropriate inverse link function, but you won't need to worry about its gradient being well behaved.

More generally, it's useful to keep in mind that while auto-differentiation is extremely useful, we don't need to use it all the time and there can be advantages in doing parts of the backpropagation manually.

I've also found the approach described in this post to be a useful trick to simplify the training of an ensemble of neural networks, but I'll cover that in a future post.

Suppose you want to sample from the multinomial distribution, that is, the binomial distribution generalized to more than two outcomes.

The distribution is specified with an integer parameter $N$, specifying the number of independent trials, and a discrete probability vector $\bf p$ of length $d$ specifying the probability of each outcome in each trial. It is the probability distribution of the number of each of the $d$ possible outcomes.

The sequential solution

Typically, you'd sample from the multinomial by doing repeated binomial sampling; sample from the binomial to determine the number of trials resulting in outcome number 1, i.e., sample $n_1 \sim \text{Bin}(N, p_1)$. Then subtract $n_1$ from the number of trials $N$ and sample from the binomial for the second outcome, $n_2 \sim \text{Bin}(N-n_1, p_2)$, and so on, as follows.

def multinomial_sample(N, ps):
    N1 = N
    ns = np.zeros_like(ps)

    for p_i in range(len(ps)):
        n = np.random.binomial(N1, ps[p_i] / ps[p_i:])
        ns[p_i] = n
        N1 -= n

    return ns

How long is that going to take? If you sample from a binomial using NumPy or R, which algorithm is used depends on the parameters. If $N p \le 30$, then an inverse transformation algorithm is used, which takes $\text{O}(N \text{min}(p, 1-p))$. If $N p > 30$, an algorithm called BTPE, which is an approximate sequential acceptance/rejection algorithm, is used. This algorithm is sublinear time, and is "approximately constant time" for $N p > 30$ ¹.

Let's say that binomial sampling is $\text{O}(1)$. Then multinomial sampling by sequential binomial sampling is $\text{O}(d)$, where $d$ is the number of possible outcomes.

1. Binomial random variate generation, Kachitvichyanukul and Schmeiser.↩

Can we do better?

What about parallelising the work? Our sequential binomial sampling method doesn't seem very parallelisable, so let's think of a way that does. Another approach to determine the number of trials resulting in the $i$th outcome is to repeatedly bisect the possible outcomes, using binomial sampling each time to sample the number of trials resulting in an outcome in the same half as the outcome we're interested in.

E.g., suppose we have four possible outcomes, with probabilities [0.4, 0.1, 0.2, 0.3], and we want the number of trials resulting in the second outcome. First we sample from the binomial to determine how many trials result in any of the first half of outcomes (with probability 0.5), then we sample from the binomial again to determine the number of those that result in the second outcome (with probability $0.1/(0.4+0.1) = 0.2$). To sample the number of outcomes $i$, for any single $i$, in this way takes $\text{O}(\log d)$.

Consistency

Now it's just a matter of assigning each outcome to a separate node/computer and doing the work in parallel, right? Not quite. There's quite a bit of shared computation going on between nodes, and they'd better agree on their answers. For example, all nodes sampling for the first half of the possible outcomes first sample from a binomial to determine the number of trials that lead to outcomes in that half, and they should all get the same answer. If they don't, we won't be collectively sampling from a multinomial and our total number of trials, when we sum over the outcomes, won't sum to $N$.

There are two solutions: communication or ensuring consistent random number generator (RNG) states. The first solution would have a single node sample the number of trials leading to one of the first half of outcomes. It would then share this number with a second node, and the two of them would bisect the outcomes again, communicate the results to another two nodes, and so on.

I'll focus here on the consistent RNG solution. The idea is that when any two nodes are asking exactly the same question then their RNGs should be in the same state. Since in my examples I'll be using NumPy's RNG, I'll be explicitly setting its seed algorithmically. This is probably a bad idea. Ideally, you'd be using an RNG that allows you to skip ahead by $n$ states in constant time.

#collapse-show
# Sample the number of N trials resulting in outcome leaf_index with probability ps[leaf_index]
# by repeatedly bisecting the possible outcomes and doing binomial sampling
def multinomial_sample2(N, ps, leaf_index, branch_index=0, base_seed=0):
    if ps.size == 1:
        return N

    # Split the outcomes in half
    split_index = int(np.ceil(0.5 * ps.size))
    # Compute the probability that a trial results in the left half of the outcomes
    left_p = np.sum(ps[:split_index])

    # Set the RNG such that the RNG state depends on where we are in the outcome bisection tree
    # i.e., using the current branch index
    # Algorithmically setting the NumPy RNG state is **probably a bad idea**
    np.random.seed(base_seed + branch_index)

    # Sample from the binomial to determine the number of trials resulting in one of the outcomes on the left
    left_N = np.random.binomial(N, left_p)

    # Recursively call this function, with parameters depending on if the leaf_index outcome is on the left or the right
    # Branches are numbered depth first, so that if we take the left branch then the next branch index is the current + 1
    # and if we take the right branch then the next branch index is the current + 1 + #branch indices in the left subtree
    if leaf_index >= split_index:
        return multinomial_sample2(N - left_N, ps[split_index:]/(1-left_p), leaf_index - split_index, branch_index + split_index, base_seed)
    else:
        return multinomial_sample2(left_N, ps[:split_index]/left_p, leaf_index, branch_index + 1, base_seed)

The following shows the algorithm in action. The boxplots indicate the count for each outcome over a number of trials, while the coloured dots indicate the expected values.

N = 1000
d = 10
ps = np.random.random(size=d)
ps /= np.sum(ps)
trials = 1000

outcome_counts = np.array([[multinomial_sample2(N, ps, i, j) for i in range(d)] for j in range(trials)])

figure = plt.figure(figsize=(12,8))
plt.boxplot(outcome_counts);
plt.scatter(np.arange(1, d+1, 1), N*ps, c="r", s=50);
plt.xlabel("Outcome");
plt.ylabel("Count");

print("Total number of events counted = {} per trial".format(int(np.sum(outcome_counts) / trials)))

Total number of events counted = 1000 per trial

Equal probability case

One problem with the above is that it requires us to compute the sum of the probabilities on the left hand branch each time we bisect the possible outcomes. This effectively adds an $\text{O}(d)$ operation for each node. For the special case that each of the $d$ possible outcomes has the same probability, then we don't have to perform this sum and can do the following instead. (This is also useful if the majority of outcomes have the same probability.)

#collapse-show
# Sample the number of N trials resulting in outcome leaf_index, where each outcome has the same probability,
# by repeatedly bisecting the possible outcomes and doing binomial sampling
def multinomial_sample3(N, d, leaf_index, branch_index=0, base_seed=0):
    if d == 1:
        return N

    # Split the outcomes in half
    split_index = int(np.ceil(0.5 * d))
    # Compute the probability that a trial results in the left half of the outcomes
    left_p = float(split_index) / d

    # Set the RNG such that the RNG state depends on where we are in the outcome bisection tree
    # i.e., using the current branch index
    # Algorithmically setting the NumPy RNG state is **probably a bad idea**
    np.random.seed(base_seed + branch_index)

    # Sample from the binomial to determine the number of trials resulting in one of the outcomes on the left
    left_N = np.random.binomial(N, left_p)

    # Recursively call this function, with parameters depending on if the leaf_index outcome is on the left or the right
    # Branches are numbered depth first, so that if we take the left branch then the next branch index is the current + 1
    # and if we take the right branch then the next branch index is the current + 1 + #branch indices in the left subtree
    if leaf_index >= split_index:
        return multinomial_sample3(N - left_N, d-split_index, leaf_index - split_index, branch_index + split_index, base_seed)
    else:
        return multinomial_sample3(left_N, split_index, leaf_index, branch_index + 1, base_seed)

N = 1000
d = 10
trials = 1000

outcome_counts = np.array([[multinomial_sample3(N, d, i, j) for i in range(d)] for j in range(trials)])

figure = plt.figure(figsize=(12,8))
plt.boxplot(outcome_counts);
plt.scatter(np.arange(1, d+1, 1), N/float(d) * np.ones(d), c="r", s=50);
plt.xlabel("Outcome");
plt.ylabel("Count");

print("Total number of events counted = {} per trial".format(int(np.sum(outcome_counts) / trials)))

Total number of events counted = 1000 per trial

When might this be useful?

The above lets you replace an $\text{O}(d)$ time sampling algorithm with a $\text{O}(d \log d)$ one that allows the work to be distributed amongst $d$ independent nodes, allowing the sampling to be done in $\text{O}(\log d)$ time.

Is this likely to be practically useful? If you need to sample from a multinomial where $d$, the number of possible outcomes, is so large that an algorithm linear in $d$ is too slow, well, you're unlikely to have a $d$-processor machine or a $d$-machine cluster available to you.

If you have $m$ processing nodes available, $m < d$, then you can split the outcomes into $m$ blocks of $d/m$, and use the above algorithm to sample the number of trials leading to an outcome in each of the blocks in $\text{O}(\log m)$ time. You can then use repeated binomial sampling within each node to sample the number of trials for each outcome in that block in $\text{O}(d/m)$ time. Since for $d >> m$, the work is going to be dominated by the $\text{O}(d/m)$ part, this effectively divides the sampling time by the number of nodes.

I lately ironed out a little confusion I had about the correspondence between cross entropy and negative log-likelihood when using a neural network for multi-class classification. I'm writing this mostly so I have a handy reference in future.

Suppose we have a neural network for multi-class classification, and the final layer has a softmax activation function, i.e.,

$\hat{\mathbf{y}} = \sigma(\mathbf{z})$,

where

$\sigma(\mathbf{z})_j = \frac{\exp(z_j)}{\sum_{k=1}^{M}\exp(z_k)}$.

The vector $\hat{\boldsymbol{y}}$ is of length $M$ and can be interpreted as the probability of each of $M$ possible outcomes occuring, according to the model represented by the network.

The model is discriminative (or conditional), meaning that it models the dependence of the unobserved variable $y$ on the observed variable $\boldsymbol{x}$. The model is parameterised by the parameters of the network, $\boldsymbol{\theta}$. I.e., the network represents a conditional probability distribution $\mathrm{p}(y \mid \boldsymbol{x}, \boldsymbol{\theta})$.

Log likelihood

Ignoring any issues with generalisation, suppose we want to choose the model (within the family of models the network architecture represents) that maximizes the likelihood of the observed data. I.e., we want to find the value of the parameters $\boldsymbol{\theta}$ that maximizes the likelihood of the data. We'll do this by something like stochastic gradient descent, using the negative log-likelihood as a cost function.

How do we compute the likelihood of the data? If we make a single observation, and we observe outcome $j$, then the likelihood is simply $\hat{y}_j$.

If we represent the actual observation as a vector $\boldsymbol{y}$ with one-hot encoding (i.e., the $j$th element is 1 and all other elements are 0 when we observe the $j$th outcome), then the likelihood of the same single observation can be represented as

$\prod_{j=1}^{M}\hat{y}_j ^ {y_j}$, since each term in the product except that corresponding to the observed value will be equal to 1.

The negative log likelihood is then

$-\sum_{j=1}^{M} y_j \log{\hat{y}_j}$.

Now, we know that the vector $\hat{\boldsymbol{y}}$ represents a discrete probability distribution over the possible values of the observation (according to our model). The vector $\boldsymbol{y}$ can also be interpreted as a probability distribution over the same space, that just happens to give all of its probability mass to a single outcome (i.e., the one that happened). We might call it the empirical distribution. Under this interpretation, the expression for the negative log likelihood above is also equal to a quantity known as the cross entropy.

Cross entropy

If a discrete random variable $X$ has the probability mass function $f(x)$, then the entropy of $X$ is

$\mathrm{H}(X) = \sum_{x}f(x)\log \frac{1}{f(x)} = -\sum_{x}f(x)\log f(x)$.

It is the expected number of bits needed to communicate the value taken by $X$ if we use the optimal coding scheme for the distribution.

Imagine arranging the possible values of $X$ on a line, with each outcome occupying an area proportional to its probability.

If we take the base-2 logarithm, you can think of $\log \frac{1}{f(x)}$ as being the number of yes/no questions you need to ask (if you ask the right questions) to narrow yourself down to the region of the line containing the outcome $x$. For example, using the above probability space we would first ask "Is the outcome in the first half of the line?", so would only ask one question to determine that the outcome was $x_1$. If the outcome is not $x_1$ then we have to ask a second question. The entropy is just the expected number of yes/no questions you'll need to ask; it's the sum of the number of questions for each possible outcome, each weighted by the probability of that outcome. (We don't literally ask questions about where we are in the probability line. Instead, we assign strings of bits to possible outcomes. So in the above example, we assign the string '0' to the outcome $x_1$, '10' to $x_2$, and '11' to $x_3$.)

The key in the above paragraph was the phrase "if you ask the right questions".

If we choose our series of yes/no questions to minimize the average number of questions we'd have to ask if the probability mass function over the outcomes was $f(x)$, but in reality the probability mass function is $g(x)$,

then we're going to have to ask more yes/no questions to determine the outcome than if we used the optimal series of questions for $g(x)$. It's like if you'd played the game '20 questions' with your friend Alice so many times that you've got to learn the kind of objects she chooses, and tailored the sequence of questions to her. When you come to play the game with Bob, the questions aren't quite a perfect fit, and so you have to ask more questions on average.

The expression for the cross entropy is

$\mathrm{H}(g, f) = \sum_{x}g(x)\log \frac{1}{f(x)} = -\sum_{x}g(x)\log f(x)$.

(I don't really like the standard notation here. $\mathrm{H}(X, Y)$, where $X$ and $Y$ are random variables, is taken to be the joint entropy of $X$ and $Y$, and $\mathrm{H}(f, g)$, where $f$ and $g$ are probability mass functions or probability density functions over the same space of events is taken to be the cross entropy of $f$ and $g$.)

One important thing to note is that $H(p, q) \neq H(q, p)$. So going back to our example of using the cross entropy as a per-example loss function, how do we remember which of the distributions takes which role? I.e., how do we remember, without re-deriving the thing from the negative log likelihood, whether we should we computing

$-\sum_{j=1}^{M} y_j \log{\hat{y}_j}$

or

$-\sum_{j=1}^{M} \hat{y}_j \log{y_j}$.

The way I remember it is using the intuition from above. In the expression for cross entropy, the distribution that we take the element-wise logarithm of is the one that we used to generate our coding scheme, i.e., it is the distribution that we think the data follows. We can remember this by remembering the idea that the base-2 log of the (inverse) probability for each possible outcome measures the number of yes/no questions we have to ask (each time bisecting the probability space) in order to determine that that outcome occurred. To calculate the average number of questions we have to ask, we just weight each number by the true probability of each outcome. Clearly $\hat{\boldsymbol{y}}$ represents the distribution that the network/model believes the data follows, and $\boldsymbol{y}$ is the actual data, and so is the true distribution.

K-L divergence

The Kullback-Leibler (K-L) divergence is the number of additional bits, on average, needed to encode an outcome distributed as $g(x)$ if we use the optimal encoding for $f(x)$. Using the above definitions for cross entropy and entropy we see that the K-L divergence is

$\mathrm{D}_{KL}(g\mid \mid f) = \mathrm{H}(g, f) - \mathrm{H}(g) = -(\sum_{x}g(x)\log f(x)-\sum_{x}g(x)\log g(x))$.

The K-L divergence is often described as a measure of the distance between distributions, and so the K-L divergence between the model and the data might seem like a more natural loss function than the cross-entropy.

In our network learning problem, the K-L divergence is

$-(\sum_{j=1}^{M} y_j \log{\hat{y}_j} - \sum_{j=1}^{M} y_j \log{y_j})$.

What if we were to use the K-L divergence as the loss function? We can see that the $\sum_{j=1}^{M} y_j \log{y_j}$ term depends only on the (fixed) data, not on the likelihood $\hat{\boldsymbol{y}}$, and so not on the parameters of the model $\boldsymbol{\theta}$. In other words, the value of $\boldsymbol{\theta}$ that minimizes the Kullback-Leibler divergence is the same value that minimizes the cross entropy and the negative log likelihood.

Multiple observations

If we have $N$ independently sampled examples from a training data set, the joint likelihood of the data is just the product of the likelihoods of the individual examples. The joint likelihood is

$\prod_{i=1}^{N}\prod_{j=1}^{M}{\hat{y}_{j}^{(i)}} ^ {y_{j}^{(i)}}$,

where $y_j^{(i)}$ is the target or outcome of the $i$th example, and $\hat{y}_j^{(i)}$ is the likelihood of that outcome according to the model.

The negative log likelihood is

$-\sum_{i=1}^{N}\sum_{j=1}^{M} y_j^{(i)} \log{\hat{y}_j^{(i)}}$.

One source of confusion for me is that I read in a few places "the negative log likelihood is the same as the cross entropy" without it having been specified whether they are talking about a per-example loss function or a batch loss function over a number of examples. As we saw above, the per-example negative log likelihood can indeed be interpreted as cross entropy. However, the negative log likelihood of a batch of data (which is just the sum of the negative log likelihoods of the individual examples) seems to me to be not a cross entropy, but a sum of cross entropies each based on a different model distribution (since the model is conditional on a different $\boldsymbol{x}^{(i)}$ for each $i$).

Edit (19/05/17): I think I was wrong that the expression above isn't a cross entropy; it's the cross entropy between the distribution over the vector of outcomes for the batch of data and the probability distribution over the vector of outcomes given by our model, i.e., $\mathrm{p}(\boldsymbol{y}\mid \boldsymbol{X}, \boldsymbol{\theta})$, with each distribution being conditional on the batch of observed values $\boldsymbol{X}$.

	Title	Worldwide_Gross	Production_Budget	Distributor	MPAA_Rating	IMDB_Rating	Rotten_Tomatoes_Rating
0	The Land Girls	146083.0	8000000.0	Gramercy	R	6.1	NaN
1	First Love, Last Rites	10876.0	300000.0	Strand	R	6.9	NaN
2	I Married a Strange Person	203134.0	250000.0	Lionsgate	None	6.8	NaN
3	Let's Talk About Sex	373615.0	300000.0	Fine Line	None	NaN	13.0
4	Slam	1087521.0	1000000.0	Trimark	R	3.4	62.0

Andrew M. Webb

Interactive Earth

Fastpages Notebook Blog Post

About

Front Matter

Markdown Shortcuts

Interactive Charts With Altair

Example 1: DropDown

Example 2: Tooltips

Example 3: More Tooltips

Data Tables

Images

Local Images

Remote Images

Animated Gifs

Captions

Other Elements

GitHub Flavored Emojis

Tweetcards

Youtube Videos

Boxes / Callouts

Footnotes

An Example Markdown Post

Example Markdown Post

Basic setup

Basic formatting

Lists

Boxes and stuff

Images

Code

Tables

Tweetcards

Footnotes

Iterated Mark and Recapture

Introduction

Assumptions

Short Bayesian Inference Intro/Recap

Our Prior Distribution, and Normalization

The Likelihood

Some Code

03/01/2020 edit: The number of times an animal has been seen adds no further information

Cool Softplus Function Properties

The Special Case, the Bernoulli Distribution

Plots

The General Case, Exponential Families

Distributed Joint Training of Large Ensembles with Low Communication Overhead

Introduction

Single Computational Graph Method

Multiple Computational Graphs Method

Summary

Arm Research Summit 2018 Talk

Who Needs Loss Functions? Mixing Manual and Automatic Differentiation

Setting up Synthetic Datasets

Defining the Network

A Useful Property of Natural Exponential Family Distributions

An Example with Gaussian Distributed Labels

The General Case

Why?

Distributed Multinomial Sampling

The sequential solution

Can we do better?

Consistency

Equal probability case

When might this be useful?

Cross Entropy and Log Likelihood

Log likelihood

Cross entropy

K-L divergence

Multiple observations