Changeset 12326

Timestamp:

07/04/2009 01:44:12 PM (14 months ago)

Author:

hogg

Message:

fitting a line: the discussion following the mixture model is still a mess, but I am making progress

File:

• trunk/documents/notes/straightline/straightline.tex (modified) (5 diffs)

trunk/documents/notes/straightline/straightline.tex

@@ -604 +604 @@
 an analysis of your prior knowledge or else be uninformative.
 
+The posterior probability distribution function, in general, is the
+likelihood times the prior, properly normalized.  In this case, the
+posterior we (tend to) care about is that for the line parameters
+$(m,b)$.  This is the full posterior probability distribution
+marginalized over the bad-data parameters $(\allq,\Pbad,\Ybad,\Vbad)$.
+The full posterior is
+\begin{equation}
+p(m,b,\allq,\Pbad,\Ybad,\Vbad|\ally,I) =
+ \frac{p(\ally|m,b,\allq,\Pbad,\Ybad,\Vbad,I)}{p(\ally|I)}
+ \,p(m,b,\allq,\Pbad,\Ybad,\Vbad|I)
+ \quad,
+\end{equation}
+where the numerator can---for our purposes here---be thought of as a
+normalization integral that makes the posterior distribution function
+integrate to unity.  The marginalization looks like
+\begin{equation}
+p(m,b|\ally,I)=\int \d\allq\,\d\Pbad\,\d\Ybad\,\d\Vbad
+ \,p(m,b,\allq,\Pbad,\Ybad,\Vbad|\ally,I) \quad,
+\end{equation}
+where by the integral over $\d\allq$ we really mean a sum over all of
+the $2^N$ possible settings of the $q_i$, and the integrals over the
+other parameters are over their entire prior-probable domains.
+Effectively, then this marginalization involves evaluating $2^N$
+different likelihoods and marginalizing each one of them over
+$(\Pbad,\Ybad,\Vbad)$!  Of course because very few points are true
+candidates for rejection, there are many ways to ``trim the tree'' and
+do this more quickly; the lazy can simply sample; the very lazy can
+loop over all $2^N$ and go on vacation while it executes.
+  
+Compared to the standard line-fitting of
+\sectionname~\ref{sec:standard}, once we imagine rejecting data, we
+are going to have to put some thought into computational methodology;
+the sum over $2^N$ settings should give any reader pause.  We will
+give, below, some advice about performing the marginalization and
+computing the posterior distribution functions, which can be extremely
+complex.  We will also say a few things about Bayesian
+methods---involving priors and marginalization---in general.  But
+first we will present an alternative to this exponentially complex
+solution to this problem.
+
+There is a much faster---meaning ``less computationally
+complex''---method for modeling the outliers: Instead of requiring a
+hard assignment $q_i$ for each data point to either the straight-line
+or outlier populations, every data point can be modeled as being drawn
+from a \emph{mixture} (sum with amplitudes that sum to unity) of the
+straight-line and outlier distributions.  That is, the generative
+model the data $\ally$ is changed to
+\begin{eqnarray}\label{eq:mixture}\displaystyle
+\like &\equiv& \frac{p(\ally|m,b,\Pbad,\Ybad,\Vbad,I)}{p(\ally|I)}
+ \nonumber\\
+\like &\propto&
+ \prod_{i=1}^N \left[\frac{1-\Pbad}{\sqrt{2\,\pi\,\sigma_{yi}^2}}
+ \,\exp\left(-\frac{[y_i-m\,x_i-b]^2}{2\,\sigma_{yi}^2}\right)\right.
+ \nonumber \\ & & \quad
+ \left.+ \frac{\Pbad}{\sqrt{2\,\pi\,[\Vbad+\sigma_{yi}^2]}}
+ \,\exp\left(-\frac{[y_i-\Ybad]^2}{2\,[\Vbad+\sigma_{yi}^2]}\right)\right]
+ \quad ,
+\end{eqnarray}
+where, once again, $\Pbad$ is the probability that a data point is bad
+(or, more properly, the amplitude of the bad-data distribution
+function in the mixture), and $(\Ybad,\Vbad)$ are the parameters of
+the bad-data distribution.
+
+Marginalization of the posterior generated by the likelihood in
+\equationname~(\ref{eq:mixture}) does \emph{not} involve exploring an
+exponential number of alternatives, and, again, when combined with a prior
+on $(m,b,\Pbad,\Ybad,\Vbad)$, can be marginalized over only three
+parameters to produce the posterior probability distribution function
+for the straight-line parameters $(m,b)$.  This scale of problem is
+very well-suited to simple multi-dimensonal integrators, in particular
+it works very well with a Markov-Chain Monte Carlo, which we advocate
+below.
+
+Whether to use the exponential binary assignment or mixture methods
+depends---in principle---on your generative model for the data: Are
+there really ``good'' and ``bad'' points, drawn from different
+distributions, or are all points equivalent and drawn from a single
+distribution with good and bad parts to it?  However, this question
+can rarely be answered (or even posed well)---in practice---in most
+problems of interest, so we strongly advise the latter, that is,
+fitting the distribution as a mixture of good and bad distributions,
+where there is no exponential complexity.
+
+% syntactical reference point
+
+Hogg...in both we have to choose priors on $(m,b,\Pbad,\Ybad,\Vbad)$...Hogg
+
 Hogg...we must discuss optimization and marginalization...Hogg
 
@@ -612 +699 @@
 necessary in all your inference.  Most data analysis platforms have
 optimizers that can handle problems like those presented here with
-little or no trouble.  The second piece of advice is to spend a bit of
-thought on initialization.  That is, do not start the optimizer in an
-arbitrary point in parameter space, but rather initialize it near
-where you think the answer is likely to be, using either your prior
-knowledge, or else a linear fit (which provides an objective starting
-point).  The third piece of advice is to check for multiple minima by
-randomly sampling parameter space in some significant region around
-the optimum you have, re-converge the nonlinear optimizer, and verify
-that you return to the same best optimum, or that if you come to a
-different one, it has a worse value for the objective.
+little or no trouble.  The second piece of advice---especially if you
+use a continuous optimizer, such as gradient descent---is to spend a
+bit of thought on initialization.  That is, do not start the optimizer
+in an arbitrary point in parameter space, but rather initialize it
+near where you think the answer is likely to be, using either your
+prior knowledge, or else a linear fit (which provides an objective
+starting point).  The third piece of advice is to check for multiple
+minima by randomly sampling parameter space in some significant region
+around the optimum you have, re-converge the nonlinear optimizer, and
+verify that you return to the same best optimum, or that if you come
+to a different one, it has a worse value for the objective.
 
 One attractive kind of optimizer is a sampler such as a Markov Chain
@@ -645 +733 @@
 that optimizes the scalar objective.  This method doesn't return the
 strict optimum, but it does very well in most circumstances.
-
-The posterior probability distribution is the (correctly normalized)
-product of the likelihood times the prior.  If all we care about are
-the parameters $(m,b)$ of the line, we have to marginalize the
-posterior probability distribution over all choices for the binary
-integers $\allq$, all values of the prior probability $\Pbad$, and the
-parameter space $(\Ybad,\Vbad)$ of the bad-point distribution.  This
-marginalization involves evaluating $2^N$ different likelihoods and
-marginalizing each one of them over $(\Pbad,\Ybad,\Vbad)$!  Of course
-because very few points are true candidates for rejection, there are
-many ways to ``trim the tree'' and do this more quickly; the lazy can
-simply sample; the very lazy can loop over all $2^N$ and go on
-vacation while it executes.  In any case, this bad \emph{scaling} is a
-strong mark against this method, and we will give a much faster
-alternative below.
 
 One unfortunate (though unavoidably correct) thing about any
@@ -696 +769 @@
 or standing in as a \emph{prior} for some subsequent inference.
 
-We promised a faster---less computationally complex---method for
-modeling the outliers; here it is: Instead of requiring a hard
-assignment $q_i$ for each data point to either the straight-line or
-outlier populations, every data point can be modeled as being drawn
-from a \emph{mixture} (sum with amplitudes that sum to unity) of the
-straight-line and outlier distributions.  That is, the generative
-model the data $\ally$ is changed to
-\begin{eqnarray}\label{eq:mixture}\displaystyle
-\like &\equiv& \frac{p(\ally|m,b,\Pbad,\Ybad,\Vbad,I)}{p(\ally|I)}
- \nonumber\\
-\like &\propto&
- \prod_{i=1}^N \left[\frac{1-\Pbad}{\sqrt{2\,\pi\,\sigma_{yi}^2}}
- \,\exp\left(-\frac{[y_i-m\,x_i-b]^2}{2\,\sigma_{yi}^2}\right)\right.
- \nonumber \\ & & \quad
- \left.+ \frac{\Pbad}{\sqrt{2\,\pi\,[\Vbad+\sigma_{yi}^2]}}
- \,\exp\left(-\frac{[y_i-\Ybad]^2}{2\,[\Vbad+\sigma_{yi}^2]}\right)\right]
- \quad ,
-\end{eqnarray}
-where, once again, $\Pbad$ is the probability that a data point is bad
-(or, more properly, the amplitude of the bad-data distribution
-function in the mixture), and $(\Ybad,\Vbad)$ are the parameters of
-the bad-data distribution.
-
-Optimization of the likelihood in \equationname~(\ref{eq:mixture})
-does \emph{not} involve exploring an exponential number of
-alternatives, and---when combined with a prior on
-$(m,b,\Pbad,\Ybad,\Vbad)$---can be marginalized over three rather than
-$N+3$ parameters to produce a posterior for the straight-line
-parameters $(m,b)$.  This scale of problem is very easy to solve with
-any simple optimizer, and in particular works very well with a
-Markov-Chain.
-
-Whether to use the binary assignment or mixture methods depends, in
-principle, on your generative model for the data: Are there really
-``good'' and ``bad'' points, drawn from different distributions, or
-are all points equivalent and drawn from a single distribution with
-good and bad parts to it?  However, this question can rarely be
-answered (or even posed well) in most problems of interest, so we
-strongly advise the latter, that is, fitting the distribution as a
-mixture of good and bad distributions, where there is no exponential
-complexity.
-
 \begin{comments}
 Hogg...marginalization of posteriors vs likelihoods and why,
 technically you have to be a Bayesian for this...Hogg
+
+Hogg...everyone gets in a tizzy about priors as assumptions, but they
+shouldn't...Hogg
 
 When flagging bad data, you might not want to give all points the same
@@ -811 +845 @@
 $\Pbad,\Ybad,\Vbad$) and plot the samples as a set of light grey or
 transparent lines.
+\end{problem}
+
+\begin{problem}\label{prob:badfraction}
+Solve \problemname~\ref{prob:mixture} but now plot the fully
+marginalized (over $m,b,\Ybad,\Vbad$) posterior distribution function
+for parameter $\Pbad$.  Is this distribution peaked about where you
+would expect, given the data?  Now repeat the problem, but dividing
+all the data uncertainty variances $\sigma_{yi}^2$ by 4 (or dividing
+the uncertainties by 2).  Again plot the fully marginalized posterior
+distribution function for parameter $\Pbad$.  Discuss.
 \end{problem}