A review of Latent Variable Models

class: center, middle, inverse, title-slide

# A review of Latent Variable Models
### Andrea Havron<br>NOAA Fisheries, OST

---

layout: true

.footnote[U.S. Department of Commerce | National Oceanic and Atmospheric Administration | National Marine Fisheries Service]

code.cpp{
  font-size: 14px;
}
code.r{
  font-size: 14px;
}

</style>

---
# Latent Variable Models

.p[
Data originate from a process with multiple sources of variability, not all of which are observable
- Causes:
  - Unobserved variability caused by grouping structure
  - Unobserved spatial and/or temporal process
- Statistical consequence: 
  - Data are not independent
- Includes: 
  - mixed-effects models
  - random-effects models
  - hierarchical models
  - state-space models
]

---
# Fixed versus Random Effects

.pull-left[
<br>
`$\eta = \beta0 + \beta_{1} temp + \beta_{tank}$`

<br>

|           |Linear                         |Generalized Linear               |
|-----------|-------------------------------|---------------------------------|
| **FE Model**  |$$y \sim Normal(\eta, \sigma)$$|$$y \sim Poisson(exp(\eta))$$|
| **Mixed <br>Model** | `$$\begin{align} y\lvert tank &\sim Normal(\eta, \sigma_y) \\ tank &\sim Normal(\eta, \sigma_{RE})\end{align}$$`| `$$\begin{align} y\lvert tank &\sim Poisson(exp(\eta)) \\ tank &\sim Normal(\eta, \sigma_{RE})\end{align}$$` |

]

.pull-right-narrow[
<img src="data:image/png;base64,#static/glm-glmm-table.png" width="60%" style="display: block; margin: auto auto auto 0;" />
]
---
## Joint, Conditional, and Marginal Probability
.pull-left[
<br>
**Joint Probability**: `$$P(A\cap B) = P(A,B)$$`

]

.pull-right[
![](data:image/png;base64,#LVM_files/figure-html/unnamed-chunk-2-1.png)
]

---
## Joint, Conditional, and Marginal Probability
.pull-left[
<br>
**Joint Probability**: `$$P(A\cap B) = P(A,B)$$`<br>

**Marginal Probability**: `$$P(B)$$`

]

.pull-right[
![](data:image/png;base64,#LVM_files/figure-html/unnamed-chunk-3-1.png)
]

---
## Joint, Conditional, and Marginal Probability
.pull-left[
<br>
**Joint Probability**: `$$P(A\cap B) = P(A,B)$$`<br>

**Marginal Probability**: `$$P(B)$$`<br>

**Conditional Probability**: `$$P(A|B)$$`<br>

`$$P(A|B) = \frac{P(A,B)}{P(B)}$$`

]

.pull-right[
![](data:image/png;base64,#LVM_files/figure-html/unnamed-chunk-4-1.png)
]
---
# Continuous examples
<br>

.pull-left[
**Joint density** `$$f(A,B)$$`
]

.pull-right[
<img src="data:image/png;base64,#LVM_files/figure-html/unnamed-chunk-5-1.png" width="90%" style="display: block; margin: auto auto auto 0;" />
]

---
# Continuous examples
<br>

.pull-left[
**Joint density** `$$f(A,B)$$`
**Marginal density** `$$f(A)$$`
]

.pull-right[
<img src="data:image/png;base64,#LVM_files/figure-html/unnamed-chunk-6-1.png" width="90%" style="display: block; margin: auto auto auto 0;" />
]

---
# Continuous examples
<br>

.pull-left[
**Joint density** `$$f(A,B)$$`
**Marginal density** `$$f(A)$$`
**Conditional density**
`$$f(A|B=1)$$`
]

.pull-right[
<img src="data:image/png;base64,#LVM_files/figure-html/unnamed-chunk-7-1.png" width="90%" style="display: block; margin: auto auto auto 0;" />
]
---
# Marginalization - Discrete
<br>
.pull-left[
`\begin{align}
P(Species = A) &= \sum^{3}_{i=1}P(Site_{i},A)\\
&= \sum^{3}_{i=1}P(A|Site_{i})P(Site_{i})
\end{align}`
<br>

<table class=" lightable-classic" style="margin-left: auto; margin-right: auto;">
 <thead>
<tr>
<th style="border-bottom:hidden;padding-bottom:0; padding-left:3px;padding-right:3px;text-align: center; " colspan="1"><div style="border-bottom: 1px solid #ddd; padding-bottom: 5px; ">Species</div></th>
<th style="border-bottom:hidden;padding-bottom:0; padding-left:3px;padding-right:3px;text-align: center; " colspan="3"><div style="border-bottom: 1px solid #ddd; padding-bottom: 5px; ">Site</div></th>
<th style="empty-cells: hide;border-bottom:hidden;" colspan="1"></th>
</tr>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:right;"> 1 </th>
   <th style="text-align:right;"> 2 </th>
   <th style="text-align:right;"> 3 </th>
   <th style="text-align:right;"> Sum </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> A </td>
   <td style="text-align:right;"> 41 </td>
   <td style="text-align:right;"> 98 </td>
   <td style="text-align:right;"> 64 </td>
   <td style="text-align:right;"> 203 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> B </td>
   <td style="text-align:right;"> 12 </td>
   <td style="text-align:right;"> 52 </td>
   <td style="text-align:right;"> 23 </td>
   <td style="text-align:right;"> 87 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Sum </td>
   <td style="text-align:right;"> 53 </td>
   <td style="text-align:right;"> 150 </td>
   <td style="text-align:right;"> 87 </td>
   <td style="text-align:right;"> 290 </td>
  </tr>
</tbody>
</table>

]

.pull-right[
<img src="data:image/png;base64,#static/discrete-marginal.png" width="100%" style="display: block; margin: auto auto auto 0;" />
]
---
# Marginalization - Continuous
<br>
.pull-left[
`\begin{align}
f(B) &= \int_{\mathbb{R}} f(A,B)dA\\
&\\
&= \int_{\mathbb{R}}f(B|A)f(A)dA
\end{align}`
]

.pull-right[
<img src="data:image/png;base64,#LVM_files/figure-html/unnamed-chunk-10-1.png" width="90%" style="display: block; margin: auto auto auto 0;" />
]
---

# Likelihood of a Hierarchical Model

Data: `$y$`<br>
Random effect: `$u$`<br>
Parameters: `$\Theta = (\theta_{y}, \theta_{u})$`

.pull-left[
**Bayesian Inference**
`\begin{align}
L(\Theta|y) &= \frac{f(y,u,\Theta)}{f(y)}\\
L(\Theta|y) &= \frac{f(y|u,\theta_{y})f(u|\theta_{u})f(\Theta)}{f(y)}\\
&\\
&= \frac{f(y|u,\theta_{y})f(u|\theta_{u})f(\Theta)}{\int_{\mathbb{R}}f(y,u,\Theta)dud\Theta}
\end{align}`

- Posterior density of `$\Theta$`
]

.pull-right[
**MLE Inference** 
`\begin{align}
L(\Theta) &= \int_{\mathbb{R}}f(y,u;\Theta)du\\
&\\
&= \int_{\mathbb{R}}f(y|u; \theta_{y})f(u;\theta_{u})du
\end{align}`
<br><br>

- Point estimate of `$\Theta$` with confidence intervals
]

---

# Likelihood of a Hierarchical Model

Data: `$y$`<br>
Random effect: `$u$`<br>
Parameters: `$\Theta = (\theta_{y}, \theta_{u})$`

.pull-left[
**Bayesian Inference**
`$$L(\Theta|y) = \frac{f(y,u,\Theta)}{f(y)}$$`

- Exact
 - Conjugate Priors
 - Numerical Integration
- Approximations:
 - MCMC Simulations - Metropolis Hastings, Gibbs Sampler, Hamiltonian Monte Carlo
 - Integrated Nested Laplace Approximation (INLA)
]

.pull-right[
**MLE Inference** 
`$$L(\Theta) = \int_{\mathbb{R}}f(y,u;\Theta)du$$`

- Exact
 - Newton-Rhapsom Algorithm
 - Expectation-Maximization Algorithm
 - Numerical integration
- Approximations:
 - Penalized Likelihood
 - Laplace approximation
 - Gauss-Hermite quadrature

]

---
# Laplace Approximation

.pull-left[
**Bayesian Inference**
`$$L(\Theta|y) = \frac{f(y,u,\Theta)}{f(y)}$$`
]

.pull-right[
**MLE Inference** 
`$$L(\Theta) = \int_{\mathbb{R}}f(y,u;\Theta)du$$`

]<br>
<img src="data:image/png;base64,#static/laplace-accuracy.png" width="50%" style="display: block; margin: auto;" />
<br>
Figure from [Albertsen, C. M. (2018), 2.3.1](https://backend.orbit.dtu.dk/ws/portalfiles/portal/157133664/Publishers_version.pdf)
]
---
# Laplace approximation<br>
<br>

.pull-left[
1. Random effects likelihood is minimized to find `$\hat{u}$`
2. 2nd derivatives of `$f(y,u|\theta)$` w.r.t. `$u$` is evaluated at `$\hat{u}(\theta)$`
3. Taylor Series expansion around `$\hat{u}$` to approximate the marginal
]
.pull-right[
`\begin{align}
\hat{u}(\theta) &= \underset{u}{argmin}f(u,\theta)\\
\mathbb{H}(\theta) &= f^{"}_{uu}(\hat{u}(\theta), \theta)\\
&\\
L^{*}(\theta) &= \sqrt{2\pi}^{n}det(\mathbb{H})^{-1/2}f(y,\hat{u}; \theta)
\end{align}`
]

<br>

Changes the problem from **integration** `$\rightarrow$` **optimization** <br>

`$$L(\theta) = \int_{\mathbb{R}}f(y,u;\theta)f(u;\theta)du \hspace{1cm}\rightarrow\hspace{1cm} L^{*}(\theta) = \sqrt{2\pi}^{n}det(\mathbb{H})^{-1/2}f(y,\hat{u}; \theta)$$`