Q33. Score, Hessian, and information identity
[6 marks]
For the model in Question 32, let S(\boldsymbol\beta) =\frac{\partial\ell(\boldsymbol\beta)}{\partial\boldsymbol\beta}.
(a) Derive the score vector S(\boldsymbol\beta).
(b) Show that the score has expectation zero under the assumed model.
(c) Calculate the covariance matrix of the score and the Hessian of the log-likelihood separately. Hence verify that \operatorname{Var}\!\left(S(\boldsymbol\beta)\mid X\right) =-\mathbb{E}\!\left[ \frac{\partial^2\ell(\boldsymbol\beta)} {\partial\boldsymbol\beta\,\partial\boldsymbol\beta^\top} \,\middle|\,X\right].
Definitions. Let \boldsymbol\beta\in\mathbb R^p and let \mathbf X_i be a p-dimensional column vector, so the ith row of X is \mathbf X_i^\top. Write \eta_i=\mathbf X_i^\top\boldsymbol\beta, \mu_i=e^{\eta_i}, and W=\operatorname{diag}(\mu_1,\ldots,\mu_n). The score is S=\partial\ell/\partial\boldsymbol\beta and the Hessian is H=\partial^2\ell/(\partial\boldsymbol\beta\,\partial\boldsymbol\beta^\top). Fisher information is -\mathbb{E}(H\mid X).
All expectations and covariances below are conditional on the fixed design X and evaluated under the same parameter \boldsymbol\beta as the score.
(a) Score vector. From Answer 32, \ell(\boldsymbol\beta) =\sum_{i=1}^n\{Y_i\eta_i-e^{\eta_i}\}+C. For component r, \frac{\partial\eta_i}{\partial\beta_r}=X_{ir}, \qquad \frac{\partial\mu_i}{\partial\beta_r}=\mu_iX_{ir}. Consequently, S_r(\boldsymbol\beta) =\sum_{i=1}^n(Y_i-\mu_i)X_{ir}, or in vector form, \boxed{ S(\boldsymbol\beta) =\sum_{i=1}^n\mathbf X_i(Y_i-\mu_i) =X^\top(\mathbf Y-\boldsymbol\mu). }
(b) Expected score. The Poisson model gives \mathbb{E}(Y_i\mid X)=\mu_i. Hence \boxed{ \mathbb{E}\{S(\boldsymbol\beta)\mid X\} =\sum_{i=1}^n\mathbf X_i\{\mathbb{E}(Y_i\mid X)-\mu_i\} =\mathbf0. } This is a statement about the expected score under the assumed model; the score from an arbitrary observed sample need not be zero at the true parameter.
(c) Covariance of the score. Set \varepsilon_i=Y_i-\mu_i. Then \mathbb{E}(\varepsilon_i\mid X)=0, \qquad \mathbb{E}(\varepsilon_i\varepsilon_j\mid X) =\begin{cases} \mu_i,&i=j,\\ 0,&i\ne j, \end{cases} using the Poisson variance and conditional independence. Since the score has expectation zero, \begin{aligned} \operatorname{Var}\{S(\boldsymbol\beta)\mid X\} &=\mathbb{E}\{S(\boldsymbol\beta)S(\boldsymbol\beta)^\top\mid X\}\\ &=\sum_{i=1}^n\sum_{j=1}^n \mathbf X_i\mathbb{E}(\varepsilon_i\varepsilon_j\mid X)\mathbf X_j^\top\\ &=\sum_{i=1}^n\mu_i\mathbf X_i\mathbf X_i^\top =X^\top WX. \end{aligned}
Hessian, calculated separately. Differentiate the rth score component with respect to \beta_s: H_{rs}(\boldsymbol\beta) =\frac{\partial S_r}{\partial\beta_s} =-\sum_{i=1}^n\mu_iX_{is}X_{ir}. Thus \boxed{ H(\boldsymbol\beta) =\frac{\partial^2\ell} {\partial\boldsymbol\beta\,\partial\boldsymbol\beta^\top} =-\sum_{i=1}^n\mu_i\mathbf X_i\mathbf X_i^\top =-X^\top WX. } For fixed X and \boldsymbol\beta, this Hessian does not depend on \mathbf Y. Therefore -\mathbb{E}\{H(\boldsymbol\beta)\mid X\}=X^\top WX, which matches the covariance calculated above: \boxed{ \operatorname{Var}\{S(\boldsymbol\beta)\mid X\} =-\mathbb{E}\left[ \frac{\partial^2\ell} {\partial\boldsymbol\beta\,\partial\boldsymbol\beta^\top} \,\middle|\,X\right] =X^\top WX. }
Key result. The information identity is verified by two calculations: the covariance of the score and the negative expected Hessian. Simply asserting that they are equal does not supply the requested verification.
The zero expected score uses the correct conditional mean. The information equality additionally uses Poisson variances and independence: overdispersion or correlation changes the score covariance even if the mean remains correct.