Exponential Family
假设 $P(y;\eta) = b(y)exp(\eta^{T}T(y) - a(\eta))$ ,
其中 $T(y):$ sufficient statistic,通常取$T(y) = y$
属于指数族分布的有:
Bernoulli Distribution
$P(y=1;\phi) = \phi$
$P(y;\phi) = \phi^{y}(1-\phi)^{1-y}
= exp(\log(\phi^{y}(1-\phi)^{1-y}))
= exp(y\log(\phi) + (1-y)\log(1-\phi))
= exp(\log(\frac{\phi}{1-\phi})y+\log(1-\phi))$
那么有
$b(y) = 1$
$\eta = \log(\frac{\phi}{1-\phi})$ 推出 $\phi = \frac{1}{1 + e^{-\eta}}$
$T(y) = y$
$a(\eta) = -\log(1-\phi) = \log(1 + e^{\eta})$
Gaussian Distribution
$P(y;\eta) = \frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(y-\eta)^{2}}{2\sigma^{2}}) \quad$ set $\sigma = 1$
$= \frac{1}{\sqrt{2\pi}}exp(-\frac{1}{2}y^{2})exp(-\frac{1}{2}\mu^{2}+\mu y)$
那么有
$b(y) = \frac{1}{\sqrt{2\pi}}exp(-\frac{1}{2}y^{2})$
$\eta = \mu$
$T(y) = y$
$a(\eta) = \frac{1}{2}\mu^{2} = \frac{1}{2}\eta^{2}$
Generalized Linear Models
假设
- $y \mid x;\theta$ ~ Exp($\eta$)
- 已知$x$, 我们目标是求出$h(x) = E[T(y) \mid x]$
- $\eta = \theta^{T}X$ 或者 $\eta_{i} = \theta_{i}^{T}x$ (当$n > 1$)
是一种 判别式 模型
Bernoulli
如果选取Bernoulli分布，那么
$h_{\theta}(x) = E[y|x;\theta]
= 1 * P(y=1|x;\theta) + 0 * P(y=0|x;\theta)
= \phi = \frac{1}{1 + e^{-\eta}}
= \frac{1}{1 + e^{-\theta^{T}x}}$
logistic Regression
Multinomial
$y \in \{1, \ldots , k\}$
$P(y = i) = \phi_{i} \quad \sum_{i=1}^k \phi_{i} = 1$
所以我们可以用 $\phi_{1}, \ldots ,\phi_{k - 1}$ 作为参数
因此
属于Exp Family
$T(y) \in R^{k-1}$
推出
因此
softmax Regression
可以看作logistic Regression的推广,求解参数通过最大化后验概率求解
