Path: blob/master/notebooks/book1/13/bnn_hierarchical_pymc3.ipynb
1192 views
Illustration of hierarchial Bayesian neural network classifiers. Code and text is taken directly from This blog post by Thomas Wiecki. Original Notebook. Reproduced with permission.
The data set we are using are our battle tested half-moons as it is simple, non-linear and leads to pretty visualizations. This is what it looks like:
This is just to illustrate what the data generating distribution looks like, we will use way fewer data points, and create different subsets with different rotations.
As you can see, we have 18 categories that share a higher-order structure (the half-moons). However, in the pure data space, no single classifier will be able to do a good job here. Also, because we only have 50 data points in each class, a NN will likely have a hard time producing robust results. But let's actually test this.
Classify each category separately
The code for the NN below is explained in my previous blog post on Bayesian Deep Learning.
Next, we have our function to create the BNN, sample from it, and generate predicitions. You can skip this one.
Next, we loop over each category and fit a different BNN to each one. Each BNN has its own weights and there is no connection between them. But note that because we are Bayesians, we place priors on our weights. In this case these are standard normal priors that act as regularizers to keep our weights close to zero. 
We use NUTS sampling here because the hierarchical model further below has a more complex posterior (see Why hierarchical models are awesome, tricky, and Bayesian for an explanation) and I wanted the results to be comparable. All simulations in here work with ADVI as well but the results don't look quite as strong.
OK, that doesn't seem so bad. Now let's look at the decision surfaces -- i.e. what the classifier thinks about each point in the data space.
That doens't look all that convincing. We know from the data generation process as well as from the previous blog post Bayesian Deep learning using the same data set that it give us a "Z"-shaped decision surface. So what happens is we don't have enough data to properly estimate the non-linearity in every category.
Hierarchical Bayesian Neural Network
Can we do better? You bet!
It's actually quite straight-forward to turn this into one big hierarchical model for all categories, rather than many individual ones. Let's call the weight connecting neuron in layer 1 to neuron in layer 2 in category (I just omit the layer index for simplicity in notation). Rather than placing a fixed prior as we did above (i.e. ), we will assume that each weight comes from an overarching group distribution: . The key is that we will estimate and simultaneously from data.
Why not allow for different per connection you might ask? Mainly just to make our life simpler and because it works well enough.
Note that we create a very rich model here. Every individual weight has its own hierarchical structure with a single group mean parameter and 16 per-category weights distributed around the group mean. While this creates a big amount of group distributions (as many as the flat NN had weights) there is no problem with this per-se, although it might be a bit unusual. One might argue that this model is quite complex and while that's true, in terms of degrees-of-freedom, this model is simpler than the unpooled one above (more on this below).
As for the code, we stack weights along a 3rd dimenson to get separate weights for each group. That way, through the power of broadcasting, the linear algebra works out almost the same as before.
Great -- we get higher train and test accuracy. Let's look at what the classifier has learned for each category.
Awesome! By (partially) pooling the data for each individual category we actually manage to retrieve the non-linearity. This is the strength of hierarchical models: we model the similarities of individual categories and their differences, sharing statistical power to the degree it's useful.
Of course, as we are in a Bayesian framework we also get the uncertainty estimate in our predictions:
Further analysis
There are a couple of things we might ask at this point. For example, how much does each layer specialize its weight per category. To answer this we can look at the group standard-deviation which informs us how much each weight is allowed to deviate from its group mean.
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-28-b1a514d711f0> in <module>()
----> 1 pm.traceplot(trace_hier, varnames=['w_in_1_grp_sd', 'w_1_2_grp_sd', 'w_2_out_grp_sd']);
/usr/local/lib/python3.6/dist-packages/pymc3/plots/__init__.py in wrapped(*args, **kwargs)
40 warnings.warn('Keyword argument `{old}` renamed to `{new}`, and will be removed in pymc3 3.8'.format(old=old, new=new))
41 kwargs[new] = kwargs.pop(old)
---> 42 return func(*args, **kwargs)
43 return wrapped
44
/usr/local/lib/python3.6/dist-packages/pymc3/plots/__init__.py in __call__(self, *args, **kwargs)
20 def __call__(self, *args, **kwargs):
21 raise ImportError(
---> 22 "ArviZ is not installed. In order to use `{0.attr}`:\npip install arviz".format(self)
23 )
24
ImportError: ArviZ is not installed. In order to use `plot_trace`:
pip install arviz
---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.
To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------
Interestingly, it seems that the specialization of the individual category sub-models is happening at the last layer where weights change most strongly from their group mean (as group variance is highest). I had assumed that this would happen at the first layer, based on what I found in my earlier blog post Random-Walk Bayesian Deep Networks: Dealing with Non-Stationary Data where the first layer acted as a rotation-layer.
Another interesting property of hierarchical models reveals itself here. As the group standard deviation is small for the weights in layers 1 and 2, it means these weights will be close to their group mean, reducing the effective number of parameters (degrees of freedom) of this model. This is different from the separate models where no similarities could be exploited to reduce the effective number of parameters. So from this perspective, the hierarchical model is simpler than the sum of the separate ones above.
Finally, I wondered what the group-model actually learned. To get at that, we can use the trace of from the hierarchical model and pass it to the non-hierarchical model as if we trained these weights directly on a single data set.
It seems like the group model is representing the Z-shape in a fairly noisy way, which makes sense because the decision surface for every sub-model looks quite different.
Correlations between weights
Usually we estimate NNs with MLE and BNNs with mean-field variational inference (like ADVI) which both ignore correlations between weights. As we used NUTS here, I was curious if there are meaningful correlations. Here, we look at the correlations in the first layer of the group distribution.
It indeed seems like point or mean-field estimates miss a lot of higher-order structure.
Informative priors for Bayesian Neural Networks
Informative priors are a powerful concept in Bayesian modeling. Any expert information you encode in your priors can greatly increase your inference.
The same should hold true for BNNs but it raises the question how we can define informative priors over weights which exist in this abstract space that is very difficult to reason about (understanding the learned representations of neural networks is an active research topic).
While I don't know how to answer that generally, we can nonetheless explore this question with the techniques we developed this far. The group distributions from our hierarchical model are providing structured regularization for the subnetworks. But there is no reason we can't use the group distributions only in a hierarchical network. We can just use the inferred group structure and reapply it in the form of informative priors to individual, flat networks.
For this, we must first estimate the group distribution as it looks to the subnetworks. The easiest approach is to draw sample from the group posterior distributions ( and ) and, using this sample, draw realizations from the resulting distribution: . This is essentially sampling from the group posterior predictive distribution.
While there is no guarantee that this distribution is normal (technically it is a mixture of normals so could look much more like a Student-T), this is a good enough approximation in this case. As the correlation structure of the group distributions seem to play a key role as we've seen above, we use MvNormal priors.
Note that this code just creates a single, non-hierarchical BNN.
Again, we just loop over the categories in our data set and create a separate BNN for each one. This is identical to our first attempt above, however, now we are setting the prior estimated from our group posterior of our hierarchical model in our second approach.
Drum roll
Holy mackerel, it actually worked!
As demonstrated, informed priors can help NNs a lot. But what if we don't have hierarchical structure or it would be too expensive to estimate? We could attempt to construct priors by deriving them from pre-trained models. For example, if I wanted to train an object recognition model to my own custom categories, I could start with a model like ResNet trained on the CIFAR data set, derive priors from the weights, and then train a new model on my custom data set which could then get by with fewer images than if we trained from scratch.