In this post, I’ll cover some of the main ideas about inference functions, which are kind of like a parent concept to estimating equations and estimates. This is by no means exhaustive or even complete; see the end of the post for a couple of other sources on the topic.
Set-Up
Let’s start with a quick overview of what inference functions are. Chapter 3 of Song (2007) has a very good one, which we follow here.
Inference functions provide a method of parameter estimation without the hassle of dealing with the specifics of a probability distribution. We let $\mathcal{X}$ denote our sample space, and we will consider the class, $\mathcal{P}$, of distributions parametrized by $\boldsymbol{\theta} \in \boldsymbol{\Theta}$ with $\boldsymbol{\Theta} \subseteq \mathbb{R}^p$.
An inference function is just a special function of our data and the parameter upon which the sample depends.
Definition (Inference Function). An inference function (or estimating function) is any function $\boldsymbol{\Psi}: \mathcal{X} \times \boldsymbol{\Theta} \rightarrow \mathbb{R}^p$ such that:
$\boldsymbol{\Psi}(\cdot; \boldsymbol{\theta})$ is measurable for any $\boldsymbol{\theta} \in \boldsymbol{\Theta}$
$\boldsymbol{\Psi}(\mathbf{x}; \cdot)$ is continuous in a compact subspace of $\boldsymbol{\Theta}$ containing the true parameter $\boldsymbol{\theta}_0$ for any $\mathbf{x} \in \mathcal{X}$
Clearly, $\boldsymbol{\Psi}$ must have at least $p$ linearly independent components if $\boldsymbol{\theta}$ is $p$-dimensional. However, it has $q > p$ such components, then we say that $\boldsymbol{\theta}$ is over-identified. For inference function $\boldsymbol{\Psi}$, the corresponding estimating equation</strong> is given by:
The solution to the above equation, $\hat{\boldsymbol{\theta}}$, is the estimate. Two inference functions are called equivalent (denoted as $\boldsymbol{\Psi} \sim \boldsymbol{\Phi}$) if they yield the same estimate for any sample $\mathbf{x} \in \mathcal{X}$.
Characteristics
We can describe inference functions with several different nice properties.
Definition (Unbiased). An inference function $\boldsymbol{\Psi}$ is called unbiased if it satisfies, for all $\boldsymbol{\theta} \in \boldsymbol{\Theta}$: $$ \mathbb{E}_{\boldsymbol{\theta}}\left[ \boldsymbol{\Psi}(\mathbf{X}; \boldsymbol{\theta}) \right] = \mathbf{0}_p $$
An unbiased inference function will have expectation zero for any value of the parameter $\boldsymbol{\theta}$ (where the expectation is with respect to $\mathbf{X} \sim P_{\boldsymbol{\theta}}$). We can also define an additive inference function.
Definition (Additive). Let $\mathbf{x} = (x_1, \dots, x_K)^\top$ be a sample where $x_i \overset{iid}{\sim} p_{\boldsymbol{\theta}}$ which are random variables or vectors (in the case of vectors, we permit the coordinates to be correlated). An additive inference function is given by: $$ \boldsymbol{Psi}_K(\mathbf{x}; \boldsymbol{\theta}) = \sum_{i = 1}^K \boldsymbol{\Psi}(x_i; \boldsymbol{\theta}) $$ $\boldsymbol{\Psi}_K(\cdot; \cdot)$ is the kernel inference function.
An inference function can also be described as being regular.
Definition (Regular). Let $\boldsymbol{\Psi}(\mathbf{x}; \boldsymbol{\theta})$ be a $p$-dimensional inference function, and let $\phi_i(\cdot, \cdot)$ denote its $i$-th component. We call $\boldsymbol{\Psi}(\mathbf{x}; \boldsymbol{\theta})$ regular if it satisfies:
$\mathbf{E}_{\boldsymbol{\theta}}\left[ \boldsymbol{\Psi}(\mathbf{X}; \boldsymbol{\theta}) \right] = \mathbf{0}_p$, for all $\boldsymbol{\theta} \in \boldsymbol{\Theta}$
$\frac{\partial \boldsymbol{\Psi}(\mathbf{x}; \boldsymbol{\theta})}{\partial \boldsymbol{\theta}_j}$ exists, for all $\mathbf{x} \in \mathcal{X}$ and $j = 1, \dots, p$
For any $j = 1, \dots, p$ and bounded function, $f(\mathbf{x})$, that is independent of $\boldsymbol{\theta}$, $\frac{\partial}{\partial \boldsymbol{\theta}_j}\left[ \int_{\mathcal{X}} f(\mathbf{x}) \boldsymbol{\Psi}(\mathbf{x}; \boldsymbol{\theta}) p(\mathbf{x}; \boldsymbol{\theta}) d \mathbf{x} \right] = \int_{\mathcal{X}} f(\mathbf{x}) \frac{\partial}{\partial \boldsymbol{\theta}_j} \left[ \boldsymbol{\Psi}(\mathbf{x}; \boldsymbol{\theta}) p(\mathbf{x}; \boldsymbol{\theta}) \right] d\mathbf{x}$
$\mathbf{E}_{\boldsymbol{\theta}}\left[ \phi_j(\mathbf{X}; \boldsymbol{\theta}) \phi_k(\mathbf{X}; \boldsymbol{\theta}) \right]$ exists for all $j, k = 1, \dots, p$, and the $p \times p$ matrix (the variability matrix) $\mathbf{V}_{\boldsymbol{\Psi}}(\boldsymbol{\theta}) = \mathbb{E}_{\boldsymbol{\theta}}\left[ \boldsymbol{\Psi}(\mathbf{X}; \boldsymbol{\theta}) \boldsymbol{\Psi}^\top(\mathbf{X}; \boldsymbol{\theta}) \right]$ is positive-definite
The $p \times p$ matrix (the sensitivity matrix) $\mathbf{S}_{\boldsymbol{\Psi}}(\boldsymbol{\theta}) = \mathbb{E}_{\boldsymbol{\theta}}\left[ \frac{\partial \boldsymbol{\Psi}(\mathbf{X}; \boldsymbol{\theta})}{\partial \boldsymbol{\theta}} \right]$ is non-singular
Similarly, a statistical model is called regular if its score function is regular. We will use $\mathcal{G}$ to denote the class of $p$-dimensional, regular inference functions.
We will also define the Crowder class of regular inference functions, $\mathcal{G}_C$, as the subclass of $\mathcal{G}$ consisting of members satisfying:
where $\boldsymbol{\Psi}_i \in \mathcal{G}$ and $\mathbf{C}_i(\boldsymbol{\theta})$ is a non-random matrix function of $\boldsymbol{\theta}$ chosen such that the sequence of roots \(\{ \hat{\boldsymbol{\theta}}_K \}_{K \geq 1}\) to $\boldsymbol{\Psi}_C(\boldsymbol{\theta}) = \mathbf{0}_p$ is consistent.
Properties
In what follows, it will be helpful to define the quantity:
where $\boldsymbol{\theta}_0 \in \boldsymbol{\Theta}$ is the true parameter value. We now state a consistency theorem and an asymptotic normality theorem for additive inference functions.
Theorem 3.4 (Consistency). Let $\boldsymbol{\Psi}_K$ be an additive inference function that is unbiased at $\boldsymbol{\theta}_0$. If $\lambda(\boldsymbol{\theta})$ has a unique zero at $\boldsymbol{\theta}_0$, then there exists a sequence of roots, $\{ \hat{\boldsymbol{\theta}}_K \}$ to the estimating equation, $\boldsymbol{\Psi}_K(\mathbf{x}; \boldsymbol{\theta}) = \mathbf{0}_p$, such that: $$ \hat{\boldsymbol{\theta}}_K \overset{p}{\rightarrow} \boldsymbol{\theta}_0; \hspace{5mm} \text{under } P_{\boldsymbol{\theta}_0} $$
For asymptotic normality, we restrict our attention to regular inference functions. For a regular inference function, we can define the Godambe information:
Now we come to the normality result. We omit the proof, but it is similar to the proof of asymptotic normality for maximum likelihood estimators (i.e. relies upon a Taylor expansion and controlling higher order terms with the boundedness condition).
Theorem 3.11 (Asymptotic Normality). Let $$\{ \hat{\boldsymbol{\theta}}_K\}_{K \geq 1}$$ be a sequence of roots of the estimating equations: $$ \boldsymbol{\Psi}_K(\boldsymbol{\theta}) = \sum_{i = 1}^K \boldsymbol{\Psi}(\mathbf{x}_i; \boldsymbol{\theta}); \hspace{5mm} K \geq 1 $$ where $\boldsymbol{\Psi} \in \mathcal{G}$. Suppose $\hat{\boldsymbol{\theta}}_K$ is consistent and satisfies: $$ \left\rvert \left\rvert \frac{\partial^2 \boldsymbol{\Psi}(\mathbf{x}; \boldsymbol{\theta})}{\partial \boldsymbol{\theta} \partial \boldsymbol{\theta}^\top} \right\rvert \right\rvert < M(\mathbf{x}); \hspace{5mm} \boldsymbol{\theta} \in \mathbf{N}(\boldsymbol{\theta}_0) $$ for a neighborhood, $\mathbf{N}(\boldsymbol{\theta}_0)$, centered at the true parameter value $\boldsymbol{\theta}_0$ and for a $P_{\boldsymbol{\theta}}$-measurable function, $M(\mathbf{x})$, satisfying $\mathbb{E}_{\boldsymbol{\theta}_0}[M(\mathbf{X})] < \infty$. Then: $$ \sqrt{K}(\hat{\boldsymbol{\theta}}_K - \boldsymbol{\theta}_0) \rightsquigarrow \mathcal{N}\left(\mathbf{0}_p, \mathbf{J}^{-1}_{\boldsymbol{\Psi}}(\boldsymbol{\theta}_0)\right); \hspace{5mm} \text{under } P_{\boldsymbol{\theta}_0} $$
Let $\mathbf{U}(\mathbf{x}; \boldsymbol{\theta}) = \frac{\partial \log(p(\mathbf{x}; \boldsymbol{\theta}))}{\partial \boldsymbol{\theta}}$ denote the score function. If $\mathbf{U} \in \mathcal{G}$, then the Fisher information matrix (for a single observation) is given by:
Regular inference functions also satisfy the Godambe inequality.
Theorem 3.12 (Godambe Inequality). Let $\boldsymbol{\Psi} \in \mathcal{G}$. Then: $$ \mathcal{I}(\boldsymbol{\theta}) - \mathbf{J}_{\boldsymbol{\Psi}}(\boldsymbol{\theta}) $$ is positive semi-definite $\forall \boldsymbol{\theta} \in \boldsymbol{\Theta}$ and positive definite if, and only if, $\boldsymbol{\Psi} \sim \mathbf{U}$.
We also have the Crowder optimality theorem, which tells us which inference function will be optimal (in terms of efficiency) over $\mathcal{G}_C$.
Theorem 3.13 (Crowder Optimality). The optimal inference function in the Crowder class, $\mathcal{G}_C$, is $\boldsymbol{\Psi}_K(\boldsymbol{\theta}) \in \mathcal{G}_C$ with matrix $\mathbf{C}_i(\cdot)$ defined as: $$ \mathbf{C}_i(\boldsymbol{\theta}) = \mathbb{E}_{\boldsymbol{\theta}}\left[\left(\frac{\partial \boldsymbol{\Psi}_i(\mathbf{X}_i; \boldsymbol{\theta})}{\partial \boldsymbol{\theta}} \right)^\top \text{Var}_{\boldsymbol{\theta}}^{-1}\left(\boldsymbol{\Psi}_i(\mathbf{X}_i; \boldsymbol{\theta}) \right)\right] $$
There is a lot more to say about inference functions. See Heyde’s Quasi-Likelihood and Its Applications: A General Approach to Optimal Parameter Estimation and McLeish’s The Theory and Application of Statistical Inference Functions.