AI.doll

このブログは僕のためのメモです。

$$ \newcommand{\inner}[2]{\langle #1, #2 \rangle} \newcommand{\matr}[1]{\boldsymbol{#1}} \newcommand{\pdif}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\pdifn}[3]{\frac{\partial^{#1} #2}{\partial #3^{#1}}} \newcommand{\dif}[2]{\frac{d #1}{d #2}} \newcommand{\bracket}[1]{\left\langle #1 \right\rangle} $$

Cramér-Rao inequality

Cramér-Rao 不等式

\( X=(X_1, \ldots, X_n)\)の分布が母数\( \theta\)を用いて\( f(X_1, \ldots, X_n|\theta)\)と表せるとする.
\( \varphi(X)\)が\( \psi(\theta)\)の不偏推定量, つまり\( \left\langle\varphi(X)\right\rangle=\psi(\theta)\)のとき,

$$ \begin{align} \mathrm{Var}[\psi(X)] \geq \frac{(\partial_\theta \psi(\theta))^2}{\mathcal{I}(\theta)}. \end{align} $$

特に, \( X_1, \ldots, X_n\)がi.i.d. のとき,

$$ \begin{align} \mathrm{Var}[\psi(X)] \geq \frac{(\partial_\theta \psi(\theta))^2}{n\mathcal{I}(\theta)}. \end{align} $$

ただし\( \mathcal{I}(\theta)\)はFisher情報量で,

$$ \begin{align} \mathcal{I}(\theta) = \left\langle(\partial_\theta \ln f(X|\theta))^2\right\rangle \end{align} $$

証明

微分\( \partial_\theta\) と積分\( \int dx\)は交換可能とすると,

$$ \begin{align} \partial_\theta \psi(\theta) &= \partial_\theta \left\langle\phi(X)\right\rangle\\ &= \int dx \partial_\theta f(x|\theta)\varphi(x)\\ &= \int dx f(x|\theta) \frac{\partial_\theta f(x|\theta)}{f(x|\theta)} \varphi(x)\\ &= \int dx f(x|\theta) \partial_\theta \ln f(x|\theta) \varphi(x)\\ &= \left\langle\varphi(X)\partial_\theta\ln f(X|\theta)\right\rangle. \tag{1} \end{align} $$
$$ \begin{align} \left\langle\partial_\theta\ln f(x|\theta)\right\rangle &= \int dx f(x|\theta)\frac{\partial_\theta f(x|\theta)}{f(x|\theta)}\\ &= \int dx \partial_\theta f(x|\theta)\\ &= \partial_\theta \int dx f(x|\theta)\\ &= \partial_\theta 1\\ &= 0. \tag{2} \end{align} $$

(1)\(-\)(2)\( \times\left\langle\varphi(X)\right\rangle\)より,

$$ \begin{align} \partial_\theta \psi(\theta) = \left\langle(\varphi(X)-\left\langle\varphi(X)\right\rangle)\partial_\theta\ln f(X|\theta)\right\rangle. \end{align} $$

両辺を2乗して,

$$ \begin{align} (\partial_\theta \psi(X))^2 &= \left\langle(\varphi(X)-\left\langle\varphi(X)\right\rangle)\partial_\theta \ln f(X|\theta)\right\rangle^2\\ &\leq \left\langle(\varphi(X)-\left\langle\varphi(X)\right\rangle)^2\right\rangle\left\langle(\partial_\theta \ln f(X|\theta))^2\right\rangle\;(\because \mathrm{Cauchy-Schwartz\;不等式}). \end{align} $$

\( X_1, \ldots, X_n\)がi.i.d.のとき,

$$ \begin{align} \partial_\theta \ln f(X|\theta) = \sum_{i=1}^n \partial_\theta \ln f(X_i|\theta). \end{align} $$

\( i\neq j\)とすると,

$$ \begin{align} \left\langle\partial_\theta \ln f(X_i|\theta)\partial_\theta\ln f(X_j|\theta)\right\rangle &= \left\langle\partial_\theta \ln f(X_i|\theta)\right\rangle\left\langle\partial_\theta\ln f(X_j|\theta)\right\rangle \\ &= 0 \end{align} $$

であることに注意すれば,

$$ \begin{align} \left\langle(\partial_\theta\ln f(X|\theta))^2\right\rangle = n\left\langle(\partial_\theta\ln f(X_1|\theta))^2\right\rangle . \end{align} $$