Machine Learning笔记(9)

Factor Analysis Model

background

$x \in \mathbb{R}^n$, $S = {x^1, \ldots, x^m}$,当 $n » m$时,用高斯分布很难拟合
$\mu = \frac{1}{m} \sum_{i=1}^m x^i$
$\Sigma = \frac{1}{m} \sum_{i=1}^m (x^i - \mu)(x^i - \mu)^T$
$\Sigma$是奇异矩阵

1. 不对 $\Sigma$做限制,并且希望协方差阵不是奇异的,需要 $m \ge n + 1$
2. 限制 $\Sigma$为对角矩阵,那么每一维是独立的, $\Sigma_{jj}$就是jth-维度数据方差,拟合非奇异协方差阵需要 $m \ge 2$

准备知识

假设 $x = \begin{bmatrix} x_1 \ x_2 \end{bmatrix}$, $\mu = \begin{bmatrix} \mu_1 \ \mu_2 \end{bmatrix}$, $\Sigma = \begin{bmatrix} \Sigma_{11} & \Sigma_{12} \ \Sigma_{21} & \Sigma_{22} \end{bmatrix}$. 其中 $x_1 \;and\; \mu_1 \in \mathbb{R}^r$, $x_2 \;and\; \mu_2\in \mathbb{R}^s$, $x \;and\; \mu \in \mathbb{R}^{r+s}$, $\Sigma_{11} \in \mathbb{R}^{r \times r}$, $\Sigma_{12} \in \mathbb{R}^{r \times s}$, 并且 $x \sim N(\mu, \Sigma)$,由于协方差阵是对称阵,因此 $\Sigma_{21} = \Sigma_{21}^T$
$x_1, x_2$的联合分布服从多元高斯分布,多元正态分布的性质

• 正态随机向量的任意线性组合仍为正态分布
• 多元正态分布的边缘分布仍为正态分布(上一条性质的推论)
• 其子向量的条件分布仍为正态分布

$E[x_1] = \mu_1$
$Cov(x_1) = E[(x_1 - \mu_1)(x_1 - \mu_1)^T] = \Sigma_{11}$
$x_1 \sim N(\mu_1, \Sigma_{11})$
$x_1 \mid x_2 \sim N(\mu_{1\mid2},\Sigma_{1\mid2})$
$\mu_{1\mid2} = \mu_1 + \Sigma_{12}\Sigma_{22}^{-1}(x_2 - \mu_2)$
$\Sigma_{1\mid2} = \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$

Factor Analysis Model

• 隐变量 $z \sim N(0, I)$, $z \in \mathbb{R}^d \;(d < n)$
• $x \mid z \sim N(\mu+\Lambda z, \Psi)$

等价地,

• $x = \mu + \Lambda z + \epsilon$
• $z \sim N(0,I)$
• $\epsilon \sim N(0, \Psi)$, $\Psi$是对角阵
• $z$和 $\epsilon$是独立的 ($\Rightarrow E[z\epsilon] = E[z]E[\epsilon]$)

在上述假设下( $p(x,z) = p(z)p(x\mid z)$ 高斯分布相乘是高斯),
$\begin{bmatrix} z \\ x \end{bmatrix} \sim N(\mu_{zx}, \Sigma)$
由于
$E[z] = 0$
$E[x] = E[\mu+\Lambda z + \epsilon] \\ = \mu + \Lambda E[z] + E[\epsilon] \\ = \mu$
因此
$\mu_{xz} = \begin{bmatrix} \overrightarrow{0} \\ \mu \end{bmatrix}$
由于
$\Sigma_{11} = E[(z - E[z])(z - E[z])^T] = I = E[zz^T]$
$\Sigma_{12} = E[(z - E[z])(x - E[x])^T] \\ = E[z(\mu+\Lambda z + \epsilon - \mu)^T] \\ = E[zz^T \Lambda^T + z\epsilon^T] \\ = \Lambda^T + E[z]E[\epsilon^T] \\ = \Lambda^T$
$\Sigma_{22} = E[(x - E[x])(x - E[x])^T] \\ = E[(\Lambda^T z + \epsilon)(\Lambda^T z + \epsilon)^T] \\ = E[\Lambda^T zz^T \Lambda + \epsilon\epsilon^T + \epsilon z^T\Lambda + \Lambda^T z \epsilon] \\ = \Lambda^T\Lambda + \Psi$
因此 $\Sigma = \begin{bmatrix} I & \Lambda^T
\Lambda & \Lambda^T\Lambda + \Psi \end{bmatrix}$
因此 $x \sim N(\mu, \Lambda\Lambda^T + \Psi)$
$l(\mu, \Lambda, \Psi) = \sum_{i=1}^m\log{\frac{1}{(2\pi)^{n/2} \lvert \Lambda\Lambda^T+\Psi \rvert^{1/2}}}exp(-\frac{1}{2}(x^i-\mu)^T(\Lambda\Lambda^T+\Psi)^{-1}(x^i-\mu))$
对似然函数关于参数求偏导,没有闭式解,因此我们采用EM算法解决

EM:
E-step:
$Q_i(z^i) = P(z^i \mid x^i;\mu, \Lambda,\Psi)$
由准备知识得:
$z^i\mid x^i \sim N(\mu_{z^i\mid x^i},\; \Sigma_{z^i\mid x^i})$
其中
$\mu_{z^i\mid x^i} = \Lambda^T(\Lambda\Lambda^T+\Psi)^{-1}(x^i-\mu)$
$\Sigma_{z^i\mid x^i} = I-\Lambda^T(\Lambda\Lambda^T+\Psi)\Lambda$
因此我们选取
$Q_i(z^i) = \frac{1}{(2\pi)^{d/2} \lvert \Sigma_{z^i\mid x^i}\rvert^{1/2}}exp(-\frac{1}{2}(z^i-\mu_{z^i\mid x^i})^T\Sigma_{z^i\mid x^i}^{-1}(z^i-\mu_{z^i\mid x^i}))$
M-step:
关于参数最大化
$\sum_{i=1}^m \int_{z^i} Q_i(z^i)\log{\frac{p(x^i,z^i;\mu, \Lambda,\Psi)}{Q_i(z^i)}}\mathrm{d} z^i \\ =\sum_{i=1}^m E_{z^i \sim Q_i}[\log{\frac{p(x^i,z^i;\mu, \Lambda,\Psi)}{Q_i(z^i)}}] \\ =\sum_{i=1}^m E_{z^i \sim Q_i}[\log{p(x^i \mid z^i;\mu, \Lambda,\Psi)}] + E_{z^i \sim Q_i}[\log\frac{p(z^i)}{Q_i(z^i)}]$
相当于最大化
$W =\sum_{i=1}^m E_{z^i \sim Q_i}[-\frac{1}{2}(x^i - \mu - \Lambda z^i)^T\Psi^{-1}(x^i - \mu^i - \Lambda z)]$
???

• Previous
  Machine Learning笔记(8)
• Next
  Machine Learning笔记(10)