Understanding the Metropolis Hasting algorithm - A tutorial

The Problem Say you want to evaluate the expectation of a function over a random variable \(E[f(x)] = \int p(x)f(x)dx\), or perhaps find the max of a probability distribution, typically the posterior, \(\arg \max p(x|y)\), where calculating the derivate is intractable since it depends on some feature that comes from some algorithm or some unknown […]

A short tutorial on Kernel Density Estimation (KDE)

The aim of Kernel Density Estimation(KDE) is: Given a set of \(N\) samples from a random variable, \(\mathbf{X}\), possibly multivariate and continuous, estimate the random variables probability density function(pdf) The univariate case To get a rough idea how one can think about the problem, we start out with a set of samples, \(X=[x_1,x_2,...,x_N]\), of a […]

Partially fixing numerical underflow for mixture models

In a mixture model the probability of an event \(x\) is written \(P(x=X) =\sum_{i}\pi_{i}P_{i}(x=X) \), where \(\pi_{i}\) is the probability of the point belonging to mixture \(i\) and \(P_{i}(x=X) \) is the probability of the event \(x=X\) in the \(i\)-th mixture. The problem is usually that the \(P_i\) are small which makes underflow happen when […]

The Trace Trick for Gaussian Log Likelihood

Maybe you have seen something like this when observing the log likelihood derivations for multivariate Gaussians \( \ln p(X|\mu, \Sigma) = \frac{1}{2}\ln|\Sigma|- \frac{1}{2}X^{T}\Sigma^{-1}X + const = \frac{1}{2}\ln|\Sigma| - \frac{1}{2}Tr(\Sigma^{-1}XX^{T}) + const \) and you wondered where that \(Tr\) came from. Here you can find a great explanation but I thought I would write it down […]

Calculating the indices for n choose k

So if you want to calculate n choose k, or n over k or all k different combinations of n unique items or binomial coefficients in Matlab or in math language

\[\frac{n!}{(n-k)! k!}\]

you can write nchoosek(n,k) But lets say you want to generate a list of these possible combinations? Well then you just write […]