Appendix 1
Computations for JMAP estimation
This section presents the computation for the joint MAP estimation (Subsection 4.1). The estimation is done via alternate optimization. The criterion is \(\mathcal {L}\!\left (\boldsymbol {f},\boldsymbol {v}_{\boldsymbol {\epsilon }},\boldsymbol {v}_{\boldsymbol {f}}\right) = -\ln p\!\left (\boldsymbol {f},\boldsymbol {v}_{\boldsymbol {\epsilon }},\boldsymbol {v}_{\boldsymbol {f}}|\boldsymbol {g}\right)\), and p (f,v
ε
,v
f
|g) is defined in Eq. (15). ∙ With respect to f:
$${} {\fontsize{8.4pt}{9.6pt}{\begin{aligned} \frac{\partial \mathcal{L}\left(\boldsymbol{f}, \; {\widehat{\boldsymbol{v}_{\boldsymbol{\epsilon}}}}, \; {\widehat{\boldsymbol{v}_{\boldsymbol{f}}}} \right)}{\partial \boldsymbol{f}} = 0 & \Leftrightarrow \frac{\partial}{\partial \boldsymbol{f}} \left(\!\| \boldsymbol{V}_{\boldsymbol{\epsilon}}^{-\frac{1}{2}}\! \left(\boldsymbol{g} \,-\, \boldsymbol{H}\,\boldsymbol{f} \right) \|^{2} + \| \left(\boldsymbol{V}_{\boldsymbol{f}}\right)^{-\frac{1}{2}} \boldsymbol{f}\|^{2}\! \right) \!= 0 \\ & \Leftrightarrow - \boldsymbol{H}^{T} \boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f} \right) + \boldsymbol{V}_{\boldsymbol{f}}^{-1} \boldsymbol{f} = 0 \\ & \Leftrightarrow \left[ \boldsymbol{H}^{T} \boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1} \boldsymbol{H} + \boldsymbol{V}_{\boldsymbol{f}}^{-1} \right] \boldsymbol{f} = \boldsymbol{H}^{T} \boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1} \boldsymbol{g} \\ & \Rightarrow {\widehat{\boldsymbol{f}}}_{\text{JMAP}} = \left[ \boldsymbol{H}^{T} \boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1} \boldsymbol{H} + \boldsymbol{V}_{\boldsymbol{f}}^{-1} \right]^{-1} \boldsymbol{H}^{T} \boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1} \boldsymbol{g} \end{aligned}}} $$
∙ With respect to \(\phantom {\dot {i}\!}{v}_{{\epsilon }_{i}}\), i∈{1,2,…,N}:
$$\begin{aligned} \frac{\partial \mathcal{L}\!\left({\widehat{\boldsymbol{f}}}, {\boldsymbol{v}_{\boldsymbol{\epsilon}}}, {\widehat{\boldsymbol{v}_{\boldsymbol{f}}}}\right)}{\partial {v}_{{\epsilon}_{i}}} = 0 & \Leftrightarrow \frac{\partial}{\partial {v}_{{\epsilon}_{i}}} \left(\frac{1}{2} \ln \det \left(\boldsymbol{V}_{\boldsymbol{\epsilon}}\right) + \frac{1}{2} \|\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f}\right)\|^{2} + \left(\alpha_{\epsilon 0} + 1 \right) \ln {v}_{{\epsilon}_{i}} + \beta_{\epsilon 0} {v}_{{\epsilon}_{i}}^{-1} \right) = 0 \\[-2pt] & \Leftrightarrow \frac{\partial}{\partial {v}_{{\epsilon}_{i}}} \left(\left(\alpha_{\epsilon 0} + 1 + \frac{1}{2} \right) \ln {v}_{{\epsilon}_{i}} + \left[\beta_{\epsilon 0} + \frac{1}{2} \left({g}_{i} - \boldsymbol{H}_{i}\, \boldsymbol{f} \right)^{2} \right] {v}_{{\epsilon}_{i}}^{-1} \right) = 0 \\[-2pt] & \Leftrightarrow \left(\alpha_{\epsilon 0} + 1 + \frac{1}{2} \right) {v}_{{\epsilon}_{i}} - \left(\beta_{\epsilon 0} + \frac{1}{2} \left({g}_{i} - \boldsymbol{H}_{i}\, \boldsymbol{f}\right)^{2} \right) = 0 \\[-2pt] & \Rightarrow {\widehat{{v}_{{\epsilon}_{i}}}}_{JMAP} = \frac{\beta_{\epsilon 0} + \frac{1}{2} \left({g}_{i} - \boldsymbol{H}_{i}\, \boldsymbol{f} \right)^{2}}{\alpha_{\epsilon 0} + 1 + \frac{1}{2}} \end{aligned} $$
∙ With respect to v
f
, j∈{1,2,…,M}:
$$\begin{aligned} \frac{\partial \mathcal{L}\!\left({\widehat{\boldsymbol{f}}}, {\widehat{\boldsymbol{v}_{\boldsymbol{\epsilon}}}},{\boldsymbol{v}_{\boldsymbol{f}}}\right)}{\partial{v}_{{f}_{j}}} = 0 & \Leftrightarrow \frac{\partial}{\partial {v}_{{f}_{j}}} \left(\frac{1}{2} \ln \det \left(\boldsymbol{V}_{\boldsymbol{f}}\right) + \frac{1}{2} \|\left(\boldsymbol{V}_{\boldsymbol{f}} \right)^{-\frac{1}{2}} \boldsymbol{f} \|^{2} + \left(\alpha_{f 0} + 1 \right) \ln {v}_{{f}_{j}} + \beta_{f 0} {v}_{{f}_{j}}^{-1} \right) = 0 \\[-3pt] & \Leftrightarrow \frac{\partial}{\partial {v}_{{f}_{j}}} \left(\left[ \alpha_{f 0} + 1 + \frac{1}{2}\right] \ln {v}_{{f}_{j}} + \left[ \beta_{f 0} + \frac{{f}_{{j}^{2}}}{2} \right] {v}_{{f}_{j}}^{-1} \right) = 0 \\[-2pt] & \Leftrightarrow \left(\alpha_{f 0} + 1 + \frac{1}{2} \right) {v}_{{f}_{j}} - \left(\beta_{f 0} + \frac{{f}_{{j}^{2}}}{2} \right) = 0 \\[-3pt] & \Rightarrow {\widehat{{v}_{{f}_{j}}}}_{JMAP} = \frac{\beta_{f 0} + \frac{{f}_{{j}^{2}}}{2}}{\alpha_{f 0} + 1 + \frac{1}{2}} \end{aligned} $$
Appendix 2
Computations for PM estimation via VBA, partial separability
This section presents the computation for the PM estimation, via VBA, partial separability (Subsection 4.2). The analytical expression of the logarithm is as follows:
$$ \begin{aligned} \ln p\!\left(\boldsymbol{f}, \boldsymbol{v}_{\boldsymbol{\epsilon}}, \boldsymbol{v}_{\boldsymbol{f}}|\boldsymbol{g}\right) = & -\frac{1}{2} \ln \det \left(\boldsymbol{V}_{\boldsymbol{\epsilon}} \right) -\frac{1}{2}\| \boldsymbol{V}_{\boldsymbol{\epsilon}}^{-\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f} \right) \|^{2} -\frac{1}{2} \ln \det \left(\boldsymbol{V}_{\boldsymbol{f}} \right) -\frac{1}{2} \|\boldsymbol{V}_{\boldsymbol{f}}^{-\frac{1}{2}} \boldsymbol{f}\|^{2} \\ & -\sum\limits_{i=1}^{N} \left(\alpha_{\epsilon 0} + 1 \right) \ln {v}_{{\epsilon}_{i}} -\sum\limits_{i=1}^{N} \beta_{\epsilon 0} {v}_{{\epsilon}_{i}}^{-1} -\sum\limits_{j=1}^{M} \left(\alpha_{f 0} + 1 \right) \ln {v}_{{f}_{j}} -\sum\limits_{j=1}^{M} \beta_{f 0} {{v}_{{f}_{j}}}^{-1} + C \end{aligned} $$
((33))
∙ Expression of
q
1
(
f
):The proportionality relation concerning q
1(f) established in Eq. (22) refers to f, so in the expression of lnp (f,v
ε
,v
f
|g), all the terms free of f can be regarded as constants:
$$\left\langle \ln p\!\left(\boldsymbol{f}, \boldsymbol{v}_{\boldsymbol{\epsilon}}, \boldsymbol{v}_{\boldsymbol{f}}|\boldsymbol{g}\right)\right\rangle_{q_{2} \left(\boldsymbol{v}_{\boldsymbol{\epsilon}}\right) \; q_{3} \left(\boldsymbol{v}_{\boldsymbol{f}}\right)} = \left\langle C -\frac{1}{2} \| \boldsymbol{V}_{\boldsymbol{\epsilon}}^{-\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H}\, \boldsymbol{f} \right) \|^{2} -\frac{1}{2} \| \boldsymbol{V}_{\boldsymbol{f}}^{-\frac{1}{2}} \boldsymbol{f} \|^{2} \right\rangle_{\;q_{2} \left(\boldsymbol{v}_{\boldsymbol{\epsilon}}\right) \; q_{3} \left(\boldsymbol{v}_{\boldsymbol{f}}\right)} $$
leading to:
$$ \begin{aligned} \left\langle \ln p\!\left(\boldsymbol{f},\boldsymbol{v}_{\boldsymbol{\epsilon}},{v}_{{f}}|\boldsymbol{g}\right) \right\rangle_{q_{2}\left(\boldsymbol{v}_{\boldsymbol{\epsilon}}\right) \; q_{3}\left(\boldsymbol{v}_{\boldsymbol{f}}\right)} = C -\frac{1}{2} \left\langle \|\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f} \right)\|^{2} \right\rangle_{q_{2} \left(\boldsymbol{v}_{\boldsymbol{\epsilon}}\right)} -\frac{1}{2} \left\langle \|\boldsymbol{V}_{\boldsymbol{f}}^{-\frac{1}{2}} \boldsymbol{f}\|^{2} \right\rangle_{q_{3}\left(\boldsymbol{v}_{\boldsymbol{f}}\right)} \end{aligned} $$
((34))
Considering the notation introduced in (10) corresponding to V
ε
and denoting the ith line of the matrix H with H
i
, i∈{1,2,…,N}, we write:
$$ \boldsymbol{V}_{\boldsymbol{\epsilon}}^{-\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f}\right) = \left[ {v}_{{\epsilon}_{1}}^{-1/2} \left({g}_{1} - \boldsymbol{H}_{1} \,\boldsymbol{f} \right) \ldots {v}_{{\epsilon}_{i}}^{-1/2} \left({g}_{i} - \boldsymbol{H}_{i} \,\boldsymbol{f} \right) \ldots {v}_{{\epsilon}_{N}}^{-1/2} \left({g}_{N} - \boldsymbol{H}_{N} \,\boldsymbol{f} \right) \right]^{T} $$
((35))
so the norm is written as:
$$ \| \boldsymbol{V}_{\boldsymbol{\epsilon}}^{-\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H} \,\boldsymbol{f} \right)\|^{2} = \sum\limits_{i=1}^{N} {v}_{{\epsilon}_{i}}^{-1} \left({g}_{i} - \boldsymbol{H}_{i} \,\boldsymbol{f} \right)^{2} $$
((36))
Introducing the notations:
$${} \begin{aligned} {\widetilde{{v}_{{\epsilon}_{i}}^{-1}}} &= \!\int\! {v}_{{\epsilon}_{i}}^{-1} q_{2i}\!\left({v}_{{\epsilon}_{i}}\right) \text{d} {v}_{{\epsilon}_{i}} \;\; ; \; {\widetilde{\boldsymbol{v}_{\boldsymbol{\epsilon}}^{-1}}} = \left[ {\widetilde{{v}_{{\epsilon}_{1}}^{-1}}} \ldots {\widetilde{{v}_{{\epsilon}_{i}}^{-1}}} \ldots {\widetilde{{v}_{{\epsilon}_{N}}^{-1}}} \right]^{T} \; ; \\ {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} &= \text{diag} \left({\widetilde{\boldsymbol{v}_{\boldsymbol{\epsilon}}^{-1}}} \right) \end{aligned} $$
((37))
we can write:
$$ \begin{aligned} \left\langle \|\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f} \right) \|^{2}\right\rangle_{q_{2}(\boldsymbol{v}_{\boldsymbol{\epsilon}})} &= \sum\limits_{i=1}^{N} {\widetilde{{v}_{{\epsilon}_{i}}^{-1}}} \left({g}_{i} - \boldsymbol{H}_{i} \,\boldsymbol{f} \right)^{2}\\ &= \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\right)^{1/2} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f} \right)\|^{2} \end{aligned} $$
((38))
Introducing the notation
$${}\begin{aligned} {\widetilde{{v}_{{f}_{j}}^{-1}}} = \!\int\! {v}_{{f}_{j}}^{-1} q_{3j}\!\left({v}_{{f}_{j}}\right) \text{d} {v}_{{f}_{j}} \;\; ; \;\; {\widetilde{\boldsymbol{v}_{\boldsymbol{f}}^{-1}}} &= \left[ {\widetilde{{v}_{{f}_{1}}^{-1}}} \ldots {\widetilde{{v}_{{f}_{j}}^{-1}}} \ldots {\widetilde{v_{{f}_{M}}^{-1}}} \right]^{T} \; ; \\ {\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}} &= \text{diag} \left({\widetilde{{v}_{{f}_{j}}^{-1}}} \right) \end{aligned} $$
((39))
we can write:
$$ \left\langle \|\boldsymbol{V}_{\boldsymbol{f}}^{-\frac{1}{2}} \boldsymbol{f}\|^{2} \right\rangle_{q_{3}(\boldsymbol{v}_{\boldsymbol{f}})} = \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\right)^{\frac{1}{2}} \boldsymbol{f} \|^{2} $$
((40))
Finally from (34), (38), and (40), for the expression of \(\left \langle \ln p\!\left (\boldsymbol {f},\boldsymbol {v}_{\boldsymbol {\epsilon }},v_{f}|\boldsymbol {g}\right) \right \rangle _{q_{2}(\boldsymbol {v}_{\boldsymbol {\epsilon }}) \; q_{3}\left (\boldsymbol {v}_{\boldsymbol {f}}\right)}\), we have:
$${} {\fontsize{9.2pt}{9.6pt}{\begin{aligned} \left\langle \ln p\!\left(\boldsymbol{f},\boldsymbol{v}_{\boldsymbol{\epsilon}},\boldsymbol{v}_{\boldsymbol{f}}|\boldsymbol{g}\right) \right\rangle_{q_{2}(\boldsymbol{v}_{\boldsymbol{\epsilon}}) \; q_{3}\left(\boldsymbol{v}_{\boldsymbol{f}}\right)} = C& -\frac{1}{2} \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \right)^{1/2} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f} \right) \|^{2} \\&-\frac{1}{2} \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}} \right)^{\frac{1}{2}} \boldsymbol{f} \|^{2} \end{aligned}}} $$
((41))
and via the first proportionality from (22) and the notation:
$$ J(\boldsymbol{f}) = \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\right)^{1/2} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f} \right) \|^{2} + \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}} \right)^{\frac{1}{2}} \boldsymbol{f} \|^{2} $$
((42))
the probability q
1(f) can be expressed by the following proportionality:
$$ q_{1}(\boldsymbol{f}) \propto \left\{ -\frac{1}{2} J(\boldsymbol{f}) \right\} $$
((43))
The criterion J(f) introduced in Eq. (42) is quadratic in f. Equation 43 establishes a proportionality relation between q
1(f) and an exponential function having as argument a quadratic criterion. This leads to the following:
Intermediate conclusion 1.
The probability distribution function q
1(f) is a multivariate normal distribution.
Of course, the mean is given by the solution that minimizes the criterion J(f), i.e., the solution of the equation \(\frac {\partial J(\boldsymbol {f})}{\partial \boldsymbol {f}}=0\) (and in particular, this is the same criterion that arrived in the MAP estimation technique for f, with some formal differences):
$$ \begin{aligned} \frac{\partial J(\boldsymbol{f})}{\partial \boldsymbol{f}} = 0 & \Rightarrow {\widehat{\boldsymbol{f}}}_{\text{PM}} = \left(\boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \boldsymbol{H} + {\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}} \right)^{-1} \boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \boldsymbol{g} \end{aligned} $$
((44))
The corresponding covariance matrix is computed by identification. On the one hand, we have the following relation:
$${} {\fontsize{8.4pt}{9.6pt}{\begin{aligned} \mathcal{N}\left(\boldsymbol{f}|\,{\widehat{\boldsymbol{f}}}_{\text{PM}}, {\widehat{\boldsymbol{\Sigma}}}\right) \propto \left(\det ({\widehat{\boldsymbol{\Sigma}}}) \right)^{\frac{1}{2}} \!\exp\! \left\{ -\frac{1}{2} \left(\boldsymbol{f}-{\widehat{\boldsymbol{f}}}_{\text{PM}}\right)^{T} {\widehat{\boldsymbol{\Sigma}}}^{-1} \left(\boldsymbol{f}-{\widehat{\boldsymbol{f}}}_{\text{PM}}\right) \right\} \end{aligned}}} $$
((45))
One the other hand, we have the following proportionality, given by Eq. (43):
$$ \mathcal{N}\left(\boldsymbol{f} | \,{\widehat{\boldsymbol{f}}}_{\text{PM}}, {\widehat{\boldsymbol{\Sigma}}} \right) \propto q_{1}(\boldsymbol{f}) \propto \exp \left\{ -\frac{1}{2} J(\boldsymbol{f}) \right\} $$
((46))
So, the covariance matrix \({\widehat {\boldsymbol {\Sigma }}}\) must respect the following relation:
$$ \left(\boldsymbol{f} - {\widehat{\boldsymbol{f}}}_{\text{PM}}\right)^{T} {\widehat{\boldsymbol{\Sigma}}}^{-1} \left(\boldsymbol{f} - {\widehat{\boldsymbol{f}}}_{\text{PM}}\right) \equiv J(\boldsymbol{f}), $$
((47))
where the sign ≡ represents an equality between the two terms until a free f term. If we consider the covariance matrix
$$ {\widehat{\boldsymbol{\Sigma}}} = \left(\boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \boldsymbol{H} + {\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}} \right)^{-1} $$
((48))
we have the following equalities:
$${} {\fontsize{8.8pt}{9.6pt}{\begin{aligned} \left(\boldsymbol{f}-{\widehat{\boldsymbol{f}}}_{\text{PM}}\right)^{T} {\widehat{\boldsymbol{\Sigma}}}^{-1} \left(\boldsymbol{f} - {\widehat{\boldsymbol{f}}}_{\text{PM}}\right) & = \left(\boldsymbol{f} - {\widehat{\boldsymbol{\Sigma}}}\boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \boldsymbol{g}\right)^{T} {\widehat{\boldsymbol{\Sigma}}}^{-1}\\&\quad\, \left(\boldsymbol{f} - {\widehat{\boldsymbol{\Sigma}}}\boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \boldsymbol{g}\right) \\ & = \left(\boldsymbol{f}^{T} - \boldsymbol{g}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \boldsymbol{H} {\widehat{\boldsymbol{\Sigma}}}\right)\\&\quad\, \left({\widehat{\boldsymbol{\Sigma}}}^{-1} \boldsymbol{f} - \boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \boldsymbol{g}\right) \\ & = \boldsymbol{f}^{T} \left(\boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \boldsymbol{H} + {\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}} \right)\,\\&\quad\; \boldsymbol{f} - 2 \, \boldsymbol{f}^{T}\boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\boldsymbol{g} + C, \end{aligned}}} $$
((49))
where we have used the equality \(\boldsymbol {f}^{T} \boldsymbol {H}^{T} {\widetilde {\boldsymbol {V}_{\boldsymbol {\epsilon }}^{-1}}} \boldsymbol {g} = \boldsymbol {g}^{T} {\widetilde {\boldsymbol {V}_{\boldsymbol {\epsilon }}^{-1}}} \boldsymbol {H}\, \boldsymbol {f}\), as a consequence of the fact that one term is the transpose of the other and the term is a scalar. We also used the fact that \({\widehat {\boldsymbol {\Sigma }}} = {\widehat {\boldsymbol {\Sigma }}}^{T}\) and \(\boldsymbol {g}^{T} {\widetilde {\boldsymbol {V}_{\boldsymbol {\epsilon }}^{-1}}} \boldsymbol {H}\left (\boldsymbol {H}^{T} {\widetilde {\boldsymbol {V}_{\boldsymbol {\epsilon }}^{-1}}} \boldsymbol {H} + {\widetilde {\boldsymbol {V}_{\boldsymbol {f}}^{-1}}} \right)^{-1} \boldsymbol {H}^{T}{\widetilde {\boldsymbol {V}_{\boldsymbol {\epsilon }}^{-1}}} \boldsymbol {g}\) was viewed as a constant C. We also have the following equalities:
$${} \begin{aligned} J\left(\,\boldsymbol{f}\right) &= \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\right)^{1/2} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f}\right) \|^{2} + \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}}\right)^{\frac{1}{2}} \boldsymbol{f} \|^{2}\\ &= \left(\boldsymbol{g}^{T} - \boldsymbol{f}^{T}\boldsymbol{H}^{T}\right) {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \left(\boldsymbol{g} - \boldsymbol{H}\, \boldsymbol{f}\right) + \boldsymbol{f}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}}\, \boldsymbol{f} \\ &= \boldsymbol{f}^{T} \left(\boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \boldsymbol{H} + {\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}}\right)\, \boldsymbol{f} - 2 \, \boldsymbol{f}^{T} \boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \boldsymbol{g} + C. \end{aligned} $$
((50))
Equations 49 and (50) show that equality imposed in (47) is verified with the covariance matrix defined as in (48). So, for the normal distribution \(\mathcal {N}\left (\boldsymbol {f} | \,{\widehat {\boldsymbol {f}}}, {\widehat {\boldsymbol {\Sigma }}} \right)\) proportional to q
1(f), we have the following parameters:
$${} {\fontsize{8.4pt}{9.6pt}{\begin{aligned} q_{1}(\boldsymbol{f}) = \mathcal{N}\left(\boldsymbol{f} | \,{\widehat{\boldsymbol{f}}}_{\text{PM}}, {\widehat{\boldsymbol{\Sigma}}}\right), \left\{ \begin{array}{l} {\widehat{\boldsymbol{f}}}_{\text{PM}} = \left(\boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \boldsymbol{H} + {\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}}\right)^{-1} \boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \boldsymbol{g} \\{\widehat{\boldsymbol{\Sigma}}} = \left(\boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \boldsymbol{H} + {\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}} \right)^{-1} \end{array}\right. \end{aligned}}} $$
((51))
∙ Expression of
\(\boldsymbol {q_{2i}\!\left ({v}_{{\epsilon }_{i}}\right):}\)
The proportionality relation concerning \(\phantom {\dot {i}\!}q_{2i}\!\left ({v}_{{\epsilon }_{i}}\right)\) established in Eq. (22) refers to \(\phantom {\dot {i}\!}v_{{\epsilon }_{i}}\), so in the expression of lnp (f,v
ε
,v
f
|g), all the terms free of \(\phantom {\dot {i}\!}v_{{\epsilon }_{i}}\) can be regarded as constants:
$${} \begin{aligned} &\left\langle \ln p\!\left(\boldsymbol{f},\boldsymbol{v}_{\boldsymbol{\epsilon}},\boldsymbol{v}_{\boldsymbol{f}}|\boldsymbol{g}\right) \right\rangle_{q_{1}(\boldsymbol{f}) \; q_{2-i}\left(v_{{\epsilon}_{i}}\right) \; q_{3}\left(\boldsymbol{v}_{\boldsymbol{f}}\right)}\\ & = C -\frac{1}{2} \left\langle \ln \det \left(\boldsymbol{V}_{\boldsymbol{\epsilon}}\right) \right\rangle_{q_{2-i}\left({v}_{{\epsilon}_{i}}\right)} \\&\qquad\;\! - \left(\alpha_{\epsilon 0} + 1\right) \ln {v}_{{\epsilon}_{i}} \\ & \qquad -\frac{1}{2} \left\langle \| \boldsymbol{V}_{\boldsymbol{\epsilon}}^{-\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f} \right)\|^{2} \right\rangle_{q_{1}(\boldsymbol{f}) \; q_{2-i}\left({v}_{{\epsilon}_{i}}\right)} - \beta_{\epsilon 0} {v}_{{\epsilon}_{i}}^{-1} \end{aligned} $$
((52))
For the first integral, it is trivial to verify:
$$ \left\langle \ln \det \left(\boldsymbol{V}_{\boldsymbol{\epsilon}} \right)\right\rangle_{q_{2-i}\left({v}_{{\epsilon}_{i}}\right)} = C + \ln {v}_{{\epsilon}_{i}} $$
((53))
For the second integral, we have the following development:
$${} {\fontsize{8.8pt}{9.6pt}{\begin{aligned} \left\langle \|\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f} \right)\|^{2}\right\rangle_{q_{1}(\boldsymbol{f}) \; q_{2-i}\left({v}_{{\epsilon}_{i}}\right)} = \left\langle \|{\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon} -{i}}^{-1}}}^{\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f} \right)\|^{2} \right\rangle_{q_{1}(\boldsymbol{f})} \end{aligned}}} $$
((54))
where we have introduced the following notations:
$$ \begin{aligned} {\widetilde{\boldsymbol{v}_{\boldsymbol{\epsilon} -{i}}^{-1}}} &= \left[ {\widetilde{{v}_{{\epsilon}_{1}}^{-1}}} \; \ldots \; {\widetilde{{v}_{{\epsilon}_{i-1}}^{-1}}} \; {{v}_{{\epsilon}_{i}}^{-1}} \; {\widetilde{{v}_{{\epsilon}_{i+1}}^{-1}}} \; \ldots \; {\widetilde{{v}_{{\epsilon}_{N}}^{-1}}} \right]^{T} \;\; ; \\ {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon} -{i}}^{-1}}} &= \text{diag} \left({\widetilde{\boldsymbol{v}_{\boldsymbol{\epsilon}-{i}}^{-1}}}\right) \end{aligned} $$
((55))
Again, using the fact that q
1(f) is a multivariate normal distribution, we have:
$${} \begin{aligned} \left\langle \|{\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}-{i}}^{-1}}}^{\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f} \right)\|^{2} \right\rangle_{q_{1}(\boldsymbol{f})} &= \| {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}-{i}}^{-1}}}^{\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H}\,{\widehat{\boldsymbol{f}}}_{\text{PM}}\right) \|^{2} \\&\quad + \text{Tr} \left(\boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}-{i}}^{-1}}} \boldsymbol{H} {\widehat{\boldsymbol{\Sigma}}}\right) \end{aligned} $$
((56))
and considering as constants all terms free of \(\phantom {\dot {i}\!}{v}_{{\epsilon }_{{i}}}\), we have:
$$\begin{array}{@{}rcl@{}} \| {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}-{i}}^{-1}}}^{\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H} \,{\widehat{\boldsymbol{f}}}_{\text{PM}} \right) \|^{2} &=& C + {v}_{{\epsilon}_{i}}^{-1} \left({g}_{i} - \boldsymbol{H}_{i} \,{\widehat{\boldsymbol{f}}}_{\text{PM}} \right)^{2} \;\; ;\notag\\ \;\; \text{Tr} \left(\boldsymbol{H}^{T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}-{i}}^{-1}}} \boldsymbol{H} {\widehat{\boldsymbol{\Sigma}}} \right) &=& C + {v}_{{\epsilon}_{i}}^{-1}\boldsymbol{H}_{i} {\widehat{\boldsymbol{\Sigma}}} \boldsymbol{H}_{i}^{T} \end{array} $$
((57))
where H
i
is the line i of the matrix H, so we can conclude:
$$ \begin{aligned} &\left\langle \| \boldsymbol{V}_{\boldsymbol{\epsilon}}^{-\frac{1}{2}} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f}\right) \|^{2} \right\rangle_{q_{1}(\boldsymbol{f}) \; q_{2-i}\left({v}_{{\epsilon}_{i}}\right)} \\&= C + \left[ \boldsymbol{H}_{i} {\widehat{\boldsymbol{\Sigma}}} \boldsymbol{H}_{i}^{T} + \left({g}_{i} - \boldsymbol{H}_{i} \,{\widehat{\boldsymbol{f}}}_{\text{PM}} \right)^{2} \right] {v}_{{\epsilon}_{i}}^{-1} \end{aligned} $$
((58))
From (52) via (53) and (58), we get:
$$\begin{aligned} &\left\langle \ln p\!\left(\boldsymbol{f},\boldsymbol{z}, \boldsymbol{v}_{\boldsymbol{\epsilon}}, \boldsymbol{v}_{\boldsymbol{f}}|\boldsymbol{g}\right) \right\rangle_{q_{1}(\boldsymbol{f}) \; q_{2-i}\left(v_{{\epsilon}_{i}}\right) \; q_{3}\left(\boldsymbol{v}_{\boldsymbol{f}}\right)} \\& = C - \left(\alpha_{\epsilon 0} + 1 + \frac{1}{2}\right) \ln {v}_{{\epsilon}_{i}} \\ & \quad \left(\beta_{\epsilon 0} + \frac{1}{2} \left[\boldsymbol{H}_{i} {\widehat{\boldsymbol{\Sigma}}} \boldsymbol{H}_{i}^{T} + \left({g}_{i} - \boldsymbol{H}_{i} \,{\widehat{\boldsymbol{f}}}_{\text{PM}} \right)^{2} \right] \right) {v}_{{\epsilon}_{i}}^{-1}\end{aligned} $$
from which we can establish the proportionality corresponding to \(\phantom {\dot {i}\!}q_{2i}({v}_{{\epsilon }_{{i}}})\):
$${} \begin{aligned} q_{2i}\left({v}_{{\epsilon}_{i}}\right) \propto {v}_{{\epsilon}_{i}}^{-\left(\alpha_{\epsilon 0} + 1 + \frac{1}{2}\right)}& \exp \left\{-\left(\beta_{\epsilon 0} + \frac{1}{2} \left[\boldsymbol{H}_{i} {\widehat{\boldsymbol{\Sigma}}} \boldsymbol{H}_{i}^{T} \right.\right.\right.\\ &\qquad +\left.\left.\left. \left({g}_{i} - \boldsymbol{H}_{i} \,{\widehat{\boldsymbol{f}}}_{\text{PM}} \right)^{2} \right] \right) {v}_{{\epsilon}_{i}}^{-1} \right\} \end{aligned} $$
((59))
Equation (59) leads to the following.
Intermediate conclusion 2.
The probability distribution function \(q_{3i}\left ({v}_{{\epsilon }_{i}}\right)\) is an inverse gamma distribution, with the parameters \(\alpha _{\epsilon _{i}}\) and \(\beta _{\epsilon _{i}}\):
We can write:
$${} {\fontsize{7.6pt}{9.6pt}{\begin{aligned} q_{2i}\left(v_{{\epsilon}_{i}}\right) = \mathcal{I}\mathcal{G} \left(v_{{\epsilon}_{i}}|\alpha_{\epsilon_{i}},\beta_{\epsilon_{i}}\right), \left\{ \begin{array}{l} \!\!\alpha_{\epsilon_{i}} = \alpha_{\epsilon 0} + \frac{1}{2} \\ \!\!\beta_{\epsilon_{i}} = \beta_{\epsilon 0} +\frac{1}{2} \left[\boldsymbol{H}_{i} {\widehat{\boldsymbol{\Sigma}}} \boldsymbol{H}_{i}^{T} + \left({g}_{i} - \boldsymbol{H}_{i} \,{\widehat{\boldsymbol{f}}}_{\text{PM}} \right)^{2} \right] \end{array}\right. \end{aligned}}} $$
((60))
∙
Expression of
\(\phantom {\dot {i}\!}\boldsymbol {q_{3j}({v}_{{f}_{j}}):}\)
The proportionality relation concerning \(\phantom {\dot {i}\!}q_{3j}\left (v_{f_{j}}\right)\) established in Eq. (22) refers to \(\phantom {\dot {i}\!}v_{{f}_{j}}\), so in the expression of lnp (f,z,v
ε
,v
f
|g), all the terms free of \(v_{f_{j}}\) can be regarded as constants:
$$ \begin{aligned} &\left\langle \ln p\!\left(\boldsymbol{f},\boldsymbol{v}_{\boldsymbol{\epsilon}},\boldsymbol{v}_{\boldsymbol{f}}|\boldsymbol{g}\right) \right\rangle_{q_{1}(\boldsymbol{f}) \; q_{2}(\boldsymbol{v}_{\boldsymbol{\epsilon}}) \; q_{3-j}\left(v_{f_{j}}\right)}\\ &= -\frac{1}{2} \left\langle \ln \det \left(\boldsymbol{V}_{\boldsymbol{f}}\right) \right\rangle_{q_{3-j}\left({v}_{{f}_{j}}\right)} -\left(\alpha_{f 0} + 1 \right) \ln {v}_{{f}_{j}} \\ & \quad-\frac{1}{2} \left\langle \| \left(\boldsymbol{V}_{\boldsymbol{f}}\right)^{-\frac{1}{2}} \boldsymbol{f} \|^{2} \right\rangle_{q_{1}(\boldsymbol{f}) \; q_{3-j}(v_{f_{j}})} -\beta_{f 0} {v}_{{f}_{j}}^{-1} \end{aligned} $$
((61))
Considering all \(\phantom {\dot {i}\!}{v}_{{f}_{j}}\) free terms as constants, it is easy to verify:
$$ \left\langle \ln \det \left(\boldsymbol{V}_{\boldsymbol{f}}\right) \right\rangle_{q_{3-j}\left({v}_{{f}_{j}}\right)} = C + \ln {v}_{{f}_{j}} $$
((62))
For the second integral:
$$ \left\langle \| \left(\boldsymbol{V}_{\boldsymbol{f}}\right)^{-\frac{1}{2}} \boldsymbol{f} \|^{2} \right\rangle_{q_{1}(\boldsymbol{f}) \; q_{3-j}\left(v_{f_{j}}\right)} = \left\langle \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{f}-{i}}^{-1}}}\right)^{\frac{1}{2}} \boldsymbol{f} \|^{2} \right\rangle_{q_{1}(\boldsymbol{f})} $$
((63))
where we have introduced the notations:
$$ \begin{aligned} {\widetilde{\boldsymbol{v}_{\boldsymbol{f}-{i}}^{-1}}} &= \left[ {\widetilde{{v}_{{f}_{1}}^{-1}}} \; \ldots \; {\widetilde{{v}_{{f}_{i-1}}^{-1}}} \; {{v}_{{f}_{i}}^{-1}} \; {\widetilde{{v}_{{f}_{i+1}}^{-1}}} \; \ldots \; {\widetilde{{v}_{{f}_{N}}^{-1}}} \right]^{T} \;\; ;\\ {\widetilde{\boldsymbol{V}_{\boldsymbol{f}-{i}}^{-1}}} &= \text{diag} \left({\widetilde{\boldsymbol{v}_{\boldsymbol{f}-{i}}^{-1}}}\right) \end{aligned} $$
((64))
Considering the fact that q
1(f) was established as a multivariate normal distribution, we have:
$${} {\fontsize{9.6pt}{9.6pt}{\begin{aligned} \left\langle \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{f}-{i}}^{-1}}}\right)^{\frac{1}{2}} \boldsymbol{f} \|^{2} \right\rangle_{q_{1}(\boldsymbol{f})} &= \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{f}-{i}}^{-1}}}\right)^{\frac{1}{2}} {\widehat{\boldsymbol{f}}}_{\text{PM}} \|^{2} + \text{Tr} \left({\widetilde{\boldsymbol{V}_{\boldsymbol{f}-i}^{-1}}} {\widehat{\boldsymbol{\Sigma}}}\right)\\ &= C + {{v}_{{f}_{i}}^{-1}} \left({\widehat{{f}_{j}}}_{\text{PM}}^{2} + {\widehat{\boldsymbol{\Sigma}}}_{jj}\right) \end{aligned}}} $$
((65))
From (61) via (62) and (65), we get:
$${} {\fontsize{8.4pt}{9.6pt}{\begin{aligned} \left\langle \ln p\!\left(\boldsymbol{f},\boldsymbol{v}_{\boldsymbol{\epsilon}},\boldsymbol{v}_{\boldsymbol{f}}|\boldsymbol{g}\right) \right\rangle_{q_{1}(\boldsymbol{f}) \; q_{2}(\boldsymbol{v}_{\boldsymbol{\epsilon}}) \; q_{3-j}\left(v_{f_{j}}\right)} = &-\left(\alpha_{f 0} + \frac{1}{2} + 1 \right) \ln {v}_{f}\\ &- \left(\!\beta_{f 0} + \frac{1}{2} \left({\widehat{{f}_{j}}}_{\text{PM}}^{2} + {\widehat{\boldsymbol{\Sigma}}}_{jj} \right)\!\! \right)\! {v}_{f}^{-1} \end{aligned}}} $$
((66))
from which we can establish the proportionality corresponding to \(q_{4}\left ({v}_{{f}_{j}}\right)\):
$${} {\fontsize{8.8pt}{9.6pt}{\begin{aligned} q_{3j}\left({v}_{{f}_{j}}\right) \propto {v}_{{f}_{j}}^{-\left(\alpha_{f 0} + \frac{1}{2} + 1 \right)} \exp \left\{-\left[\beta_{f 0} + \frac{1}{2} \left({\widehat{{f}_{j}}}_{\text{PM}}^{2} + {\widehat{\boldsymbol{\Sigma}}}_{jj} \right) \right] {v}_{f}^{-1} \right\} \end{aligned}}} $$
((67))
Equation (67) leads to the following.
Intermediate conclusion 3.
The probability distribution function q
4(v
f
) is an inverse gamma distribution, with the parameters \(\alpha _{f_{j}}\) and \(\beta _{f_{j}}\):
$${} q_{3j}\left({v}_{{f}_{j}}\right) = \mathcal{I}\mathcal{G} \left({v}_{{f}_{j}}|\alpha_{f_{j}},\beta_{f_{j}}\right), \left\{ \begin{array}{l} \alpha_{f_{j}} = \alpha_{f 0} + \frac{1}{2} \\ \beta_{f_{j}} = \beta_{f 0} + \frac{1}{2} \left({\widehat{{f}_{j}}}_{\text{PM}}^{2} + {\widehat{\boldsymbol{\Sigma}}}_{jj} \right) \end{array}\right. $$
((68))
Expressions (51), (60), and (68) resume the distributions families and the corresponding parameters for q
1(f), \(q_{2i}\left (v_{{\epsilon }_{i}}\right)\), i∈{1,2,…,N} and \(q_{3j}\left (v_{f_{j}}\right)\), j∈{1,2,…,M}. However, the parameters corresponding to the multivariate normal distribution are expressed via \({\widetilde {\boldsymbol {V}_{\boldsymbol {\epsilon }}^{-1}}}\) and \({\widetilde {\boldsymbol {V}_{\boldsymbol {f}}^{-1}}}\) (and by extension, all elements forming the three matrices \({\widetilde {v_{{\epsilon }_{i}}^{-1}}}\), i∈{1,2,…,N} and \({\widetilde {v_{f_{j}}^{-1}}}\), j∈{1,2,…,M}).
∙ Computation of
\({\widetilde {\boldsymbol {V}_{\boldsymbol {\epsilon }}^{-1}}}\)
,
\({\widetilde {\boldsymbol {V}_{\boldsymbol {f}}^{-1}}}\)
:For an inverse gamma distribution with parameters α and β, \(\mathcal {I}\mathcal {G}\left (x|\alpha, \beta \right)\), the following relation holds:
$$\left\langle x^{-1} \right\rangle_{\mathcal{I}\mathcal{G}(x|\alpha,\beta)} = \frac{\alpha}{\beta} $$
The prove of the above relation is done by direct computation, using the analytical expression of the inverse gamma distribution:
$${} \begin{aligned} \left\langle x^{-1} \right\rangle_{\mathcal{I}\mathcal{G}(x|\alpha,\beta)} & = \int x^{-1} \frac{{\beta}^{\alpha}}{\Gamma(\alpha)} x^{-\alpha-1} \exp \left\{-\frac{\beta}{x}\right\} \text{d} x\\ &= \frac{{\beta}^{\alpha}}{\Gamma(\alpha)} \frac{\Gamma(\alpha + 1)}{{\beta}^{\alpha+1}} \int \frac{{\beta}^{\alpha+1}}{\Gamma(\alpha + 1)} x^{-(\alpha + 1)-1} \\&\quad \exp \left\{-\frac{\beta}{x}\right\} \text{d} x = \\ & = \frac{\alpha}{\beta} \underbrace{\int \mathcal{I}\mathcal{G}(x|\alpha + 1,\beta)}_{1} \text{d} x = \frac{\alpha}{\beta} \end{aligned} $$
Since \(q_{2i}\left ({v}_{{\epsilon }_{i}}\right)\), i∈{1,2,…,N} and \(q_{3j}\left (v_{f_{j}}\right)\), j∈{1,2,…,M} are inverse gamma distributions, with parameters \(\alpha _{\epsilon _{i}}\) and \(\beta _{\epsilon _{i}}\), i∈{1,2,…,N}, respectively, \(\alpha _{f_{j}}\) and \(\beta _{f_{j}}\), j∈{1,2,…,M}, we can express the expectancies \({\widetilde {v_{{\epsilon }_{i}}^{-1}}}\) and \({\widetilde {v_{f_{j}}^{-1}}}\) via the parameters of the two inverse gamma distributions using the result above:
$$ {\widetilde{{v}_{{\epsilon}_{i}}^{-1}}} = \frac{\alpha_{\epsilon_{i}}}{\beta_{\epsilon_{i}}} \;\;\; ; \;\;\; {\widetilde{{v}_{f}^{-1}}} = \frac{\alpha_{f}}{\beta_{f}} $$
((69))
Using the notation introduced in (37) and (39), we obtain:
$$ \begin{aligned} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} &= \left[ \begin{array}{ccccc} \frac{\alpha_{\epsilon_{1}}}{\beta_{\epsilon_{1}}} & \ldots & 0 & \ldots & 0 \\ \vdots & \ddots & \vdots & \ddots & \vdots \\ 0 & \ldots & \frac{\alpha_{\epsilon_{i}}}{\beta_{\epsilon_{i}}} & \ldots & 0 \\ \vdots & \ddots & \vdots & \ddots & \vdots \\ 0 & \ldots & 0 & \ldots & \frac{\alpha_{\epsilon_{N}}}{\beta_{\epsilon_{N}}} \\ \end{array}\right] = {\widehat{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \;\; ;\\ \;\; {\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}} &= \left[ \begin{array}{ccccc} \frac{\alpha_{f_{1}}}{\beta_{f_{1}}} & \ldots & 0 & \ldots & 0 \\ \vdots & \ddots & \vdots & \ddots & \vdots \\ 0 & \ldots & \frac{\alpha_{f_{j}}}{\beta_{f_{j}}} & \ldots & 0 \\ \vdots & \ddots & \vdots & \ddots & \vdots \\ 0 & \ldots & 0 & \ldots & \frac{\alpha_{f_{M}}}{\beta_{f_{M}}} \\ \end{array}\right] = {\widehat{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}} \end{aligned} $$
((70))
Remark.
In Eq. (70), we have introduced other notations for \({\widetilde {\boldsymbol {V}_{\boldsymbol {f}}^{-1}}}\) and \({\widetilde {\boldsymbol {V}_{\boldsymbol {\epsilon }}^{-1}}}\). All three values were expressed during the model via unknown expectancies, but at this point, we arrive at expressions that do not contain any more integrals to be computed. Therefore, the new notations represent the final expressions for the density functions q that depend only on numerical hyperparameters, set in the prior modeling.
Appendix 3
Computations for PM estimation via VBA, full separability
This section presents the computation for the PM estimation, via VBA, full separability (Subsection 4.3). The expression of the logarithm lnp(f,v
ε
,v
f
|g) was established in the preview section (Eq. (33)).
∙ Expression of
q
1j
(f
j
)
:Using Eq. (41):
$$ \begin{aligned} &\left\langle \ln p\!\left(\boldsymbol{f},\boldsymbol{v}_{\boldsymbol{\epsilon}},\boldsymbol{v}_{\boldsymbol{f}}|\boldsymbol{g}\right) \right\rangle_{q_{1-j}({f}_{j}) \; q_{2}(\boldsymbol{v}_{\boldsymbol{\epsilon}}) \; q_{3}\left(\boldsymbol{v}_{\boldsymbol{f}}\right)}\\ &= C -\frac{1}{2} \left\langle \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\right)^{1/2} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f} \right)\|^{2} \right\rangle_{q_{1-j}({f}_{j})} \\ & -\frac{1}{2} \left\langle \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}}\right)^{\frac{1}{2}} \boldsymbol{f} \|^{2} \right\rangle_{q_{1-j}({f}_{j})} \end{aligned} $$
((71))
For the first norm, considering all the f
j
free terms as constants, we have:
$${} \begin{aligned} \|\left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\right)^{1/2} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f} \right)\|^{2} &= C + \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\right)^{1/2} \boldsymbol{H}^{j}\|^{2}{f}_{j}^{2} \\&\quad- 2 \boldsymbol{H}^{{j} T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \left(\boldsymbol{g} - \boldsymbol{H}^{-{j}}\boldsymbol{f}^{-{j}} \right){f}_{j} \end{aligned} $$
((72))
where H
j represents the column j of the matrix H, H
−j represents the matrix H except the column j, and f
−j represents the vector f except the element f
j
. Introducing the notation
$${} {\widetilde{{f}_{k}}} = \int {f}_{k} q_{1k}({f}_{k})\, \text{d} {f}_{k} \;\; ; \;\;{\widetilde{\boldsymbol{f}^{-{j}}}} = \left[ {\widetilde{{f}_{1}}} \; \ldots \; {\widetilde{{f}_{j-1}}} \; {\widetilde{{f}_{j+1}}} \; \ldots \; {\widetilde{z_{M}}} \right]^{T} $$
((73))
the expectancy of the first norm becomes:
$${} {\fontsize{8.8pt}{9.6pt}{\begin{aligned} \left\langle \|\left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\right)^{1/2} \left(\boldsymbol{g} - \boldsymbol{H}\,\boldsymbol{f}\right)\|^{2} \right\rangle_{q_{1-j}({f}_{j})} &= C + \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\right)^{1/2} \boldsymbol{H}^{j}\|^{2}{f}_{j}^{2}\\ &\quad - 2 \boldsymbol{H}^{{j} T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \left(\boldsymbol{g} - \boldsymbol{H}^{-{j}}{\widetilde{\boldsymbol{f}^{-{j}}}}\right)f_{j} \end{aligned}}} $$
((74))
The expectancy for the second norm, considering all the free f
j
terms as constants:
$$ \left\langle \| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{f}}^{-1}}}\right)^{\frac{1}{2}} \boldsymbol{f} \|^{2} \right\rangle_{q_{1-j}({f}_{j})} = C + {\widetilde{{v}_{{f}_{j}}^{-1}}} {f}_{j}^{2} $$
((75))
From Eqs. (31) and (71) and Eqs. (74) and (75), the proportionality for q
1j
(f
j
) becomes:
$${} \begin{aligned} q_{1j}({f}_{j}) \propto &\exp \left\{\left(\|\left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\right)^{1/2} \boldsymbol{H}^{j}\|^{2} + {\widetilde{v_{f_{j}}^{-1}}} \right){f}_{j}^{2}\right. \\&\qquad \left.- 2 \boldsymbol{H}^{{j} T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \left(\boldsymbol{g} - \boldsymbol{H}^{-{j}}\,{\widetilde{\boldsymbol{f}^{-{j}}}} \right)f_{j} \right\} \end{aligned} $$
((76))
Defining the criterion \(J\left ({f}_{j}\right) = \left (\| \left ({\widetilde {\boldsymbol {V}_{\boldsymbol {\epsilon }}^{-1}}}\right)^{1/2} \boldsymbol {H}^{j}\|^{2} + {\widetilde {v_{f_{j}}^{-1}}} \right) {f}_{j}^{2} - 2 \boldsymbol {H}^{{j} T} {\widetilde {\boldsymbol {V}_{\boldsymbol {\epsilon }}^{-1}}} \left (\boldsymbol {g} - \boldsymbol {H}^{-{j}}\,{\widetilde {\boldsymbol {f}^{-{j}}}} \right)f_{j}\), we arrive to the following.
Intermediate conclusion 4.
The probability distribution function q
1j
(f
j
) is a normal distribution.
In order to compute the mean of the normal distribution, it is sufficient to compute the solution that minimizes the criterion J(f
j
):
$$ \frac{\partial J({f}_{j})}{\partial {f}_{j}} = 0 \Leftrightarrow {\widehat{{f}_{j}}}_{\text{PM}} = \frac{\boldsymbol{H}^{{j} T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \left(\boldsymbol{g} - \boldsymbol{H}^{-{j}}\,{\widetilde{\boldsymbol{f}^{-j}}} \right)}{\|\left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\right)^{1/2} \boldsymbol{H}^{j}\|^{2} + {\widetilde{v_{f}^{-1}}}} $$
((77))
For the variance, we apply the same identification strategy as in the previous case, obtaining:
$$ q_{1}({f}_{j}) = \mathcal{N}\left({f}_{j} | {\widehat{{f}_{j}}}_{\text{PM}}, \text{var}_{j} \right), \left\{ \begin{array}{l} {\widehat{{f}_{j}}}_{\text{PM}} = \frac{\boldsymbol{H}^{{j} T} {\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}} \left(\boldsymbol{g} - \boldsymbol{H}^{-{j}}{\widetilde{\boldsymbol{f}^{-j}}}\right)}{\| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\right)^{1/2} \boldsymbol{H}^{j}\|^{2} + {\widetilde{v_{f_{j}}^{-1}}}} \\ \text{var}_{j} = \frac{1}{\| \left({\widetilde{\boldsymbol{V}_{\boldsymbol{\epsilon}}^{-1}}}\right)^{1/2} \boldsymbol{H}^{j}\|^{2} + {\widetilde{v_{{f}_{j}}^{-1}}}} \end{array}\right. $$
((78))
∙ Expression of
\(\phantom {\dot {i}\!}\boldsymbol {q_{2i}({v}_{{\epsilon }_{i}})}\)The proportionality relation corresponding to \(\phantom {\dot {i}\!}q_{2i}\left (v_{{\epsilon }_{i}}\right)\) established in Eq. (31) refers to \(\phantom {\dot {i}\!}v_{{\epsilon }_{i}}\), so in the expression of lnp (f,v
ε
,v
f
|g), all the terms free of \(\phantom {\dot {i}\!}v_{{\epsilon }_{i}}\) can be regarded as constants:
$$ \begin{aligned} \ln p\!\left(\boldsymbol{f},\boldsymbol{v}_{\boldsymbol{\epsilon}},\boldsymbol{v}_{\boldsymbol{f}}|\boldsymbol{g}\right) = C &-\left(\alpha_{\epsilon 0} + 1 + \frac{1}{2}\right) \ln {v}_{{\epsilon}_{i}} \\&-\left(\beta_{\epsilon 0} + \frac{1}{2} \left({g}_{i} - \boldsymbol{H}_{i}\,\boldsymbol{f}\right)\right) {v}_{{\epsilon}_{i}}^{-1} \end{aligned} $$
((79))
With the notation:
$$ \left\langle \boldsymbol{f} \right\rangle_{q_{1}(\boldsymbol{f})} = \left[ {\widehat{{f}_{1}}}_{\text{PM}} \ldots {\widehat{{f}_{j}}}_{\text{PM}} \ldots {\widehat{{f}_{M}}}_{\text{PM}} \right]^{T} \stackrel{Not}{=} {\widehat{\boldsymbol{f}}}_{\text{PM}} $$
((80))
the expectancy of the logarithm becomes:
$$ \begin{aligned} &\left\langle \ln p\!\left(\boldsymbol{f},\boldsymbol{v}_{\boldsymbol{\epsilon}},\boldsymbol{v}_{\boldsymbol{f}}|\boldsymbol{g}\right) \right\rangle_{q_{1}(\boldsymbol{f}) \; q_{2-i}\left({v}_{{\epsilon}_{i}}\right) \; q_{3}(\boldsymbol{v}_{\boldsymbol{f}})} \\&= C -\left(\alpha_{\epsilon 0} + 1 + \frac{1}{2}\right) \ln {v}_{{\epsilon}_{i}} \\ & \qquad -\left(\beta_{\epsilon 0} + \frac{1}{2} \left[\boldsymbol{H}_{i} {\widehat{\boldsymbol{\Sigma}}} \boldsymbol{H}_{i}^{T} + \left({g}_{i} - \boldsymbol{H}_{i} \,{\widehat{\boldsymbol{f}}}_{\text{PM}} \right)^{2}\right] \right) {v}_{{\epsilon}_{i}}^{-1} \end{aligned} $$
((81))
and the proportionality relation for \(q_{2i}\left ({v}_{{\epsilon }_{i}}\right)\) becomes:
$${} \begin{aligned} q_{2i}\left({v}_{{\epsilon}_{i}}\right) \propto {v}_{{\epsilon}_{i}}^{-\left(\alpha_{\epsilon 0} + 1 + \frac{1}{2}\right)} &\exp \left\{ -\left(\beta_{\epsilon 0} + \frac{1}{2} \left[\boldsymbol{H}_{i} {\widehat{\boldsymbol{\Sigma}}} \boldsymbol{H}_{i}^{T}\right.\right.\right. \\&\qquad + \left.\left.\left. \left({g}_{i} - \boldsymbol{H}_{i} \,{\widehat{\boldsymbol{f}}}_{\text{PM}} \right)^{2}\right] \right) {v}_{{\epsilon}_{i}}^{-1} \right\} \end{aligned} $$
((82))
Equation 82 leads to the following.
Intermediate conclusion 5.
The probability distribution function \(q_{2i}\left ({v}_{{\epsilon }_{i}}\right)\) is an inverse gamma distribution, with the parameters \(\alpha _{\epsilon _{i}}\) and \(\beta _{\epsilon _{i}}\).
[]
$${\kern15pt} q_{2i}\left(v_{{\epsilon}_{i}}\right) = \mathcal{I}\mathcal{G} \left(v_{{\epsilon}_{i}}|\alpha_{\epsilon_{i}},\beta_{\epsilon_{i}}\right), \left\{ \begin{array}{l} \alpha_{\epsilon_{i}} = \alpha_{\epsilon 0} + \frac{1}{2} \\ \beta_{\epsilon_{i}} = \beta_{\epsilon 0} + \frac{1}{2} \left[\boldsymbol{H}_{i} {\widehat{\boldsymbol{\Sigma}}} \boldsymbol{H}_{i}^{T} + \left({g}_{i} - \boldsymbol{H}_{i} \,{\widehat{\boldsymbol{f}}}_{\text{PM}} \right)^{2}\right] \end{array}\right. $$
((83))
∙ Expression of
\(\phantom {\dot {i}\!}\boldsymbol {q_{3j}({v}_{{f}_{j}})}\)The proportionality relation corresponding to \(\phantom {\dot {i}\!}q_{3j}\left (v_{f_{j}}\right)\) established in Eq. (31) refers to \(\phantom {\dot {i}\!}v_{{f}_{j}}\), so in the expression of lnp (f,v
ε
,v
f
|g), all the terms free of \(v_{f_{j}}\) can be regarded as constants:
$${} \begin{aligned} \ln p\!\left(\boldsymbol{f},\boldsymbol{v}_{\boldsymbol{\epsilon}},\boldsymbol{v}_{\boldsymbol{f}}|\boldsymbol{g}\right) = C &-\frac{1}{2} \ln {v}_{{f}_{j}} -\frac{1}{2} \left\langle {f}_{j}^{2} \right\rangle_{q_{1j}({f}_{j})} {v}_{{f}_{j}}^{-1}\\ &-\left(\alpha_{f_{j} 0} + 1 \right) \ln {v}_{{f}_{j}} - \beta_{f_{j} 0} {v}_{{f}_{j}}^{-1} \end{aligned} $$
((84))
The integral of the logarithm:
$${} {\fontsize{8.4pt}{9.6pt}{\begin{aligned} \left\langle \ln p\!\left(\boldsymbol{f},\boldsymbol{v}_{\boldsymbol{\epsilon}},\boldsymbol{v}_{\boldsymbol{f}}|\boldsymbol{g}\right) \right\rangle_{q_{1}(\boldsymbol{f}) \; q_{2}(\boldsymbol{v}_{\boldsymbol{\epsilon}}) \; q_{3-j}\left(v_{f_{j}}\right)} &= C -\left(\alpha_{f 0} + \frac{1}{2} + 1 \right)\ln {v}_{{f}_{j}} \\ &\quad-\left[\!\beta_{f 0} + \frac{1}{2} \left({\widehat{{f}_{j}}}_{\text{PM}}^{2} + \text{var}_{j}\right)\!\right]{v}_{{f}_{j}}^{-1} \end{aligned}}} $$
((85))
Equation 85 leads to the following.
Intermediate conclusion 6.
The probability distribution function \(q_{3j}\left ({v}_{{f}_{j}}\right)\) is an inverse gamma distribution, with the parameters \(\alpha _{f_{j}}\) and \(\beta _{f_{j}}\).
$${} q_{3j}\left({v}_{{f}_{j}}\right) = \mathcal{I}\mathcal{G} \left({v}_{{f}_{j}}|\alpha_{f_{j}}, \beta_{f_{j}}\right), \left\{ \begin{array}{l} \alpha_{f_{j}} = \alpha_{f 0} + \frac{1}{2} \\ \beta_{f_{j}} = \beta_{f 0} + \frac{1}{2} \left({\widehat{{f}_{j}}}_{\text{PM}}^{2} + \text{var}_{j}\right) \end{array}\right. $$
((86))
Appendix 4
List of symbols and abbreviations
List of symbols
During the article, all the terms written in bold represent vectors or matrices.
-
1.
H—the matrix used in the linear model considered during all the article. \(\boldsymbol {H} \in \mathcal {M}_{N\times M}\). The matrix corresponds to the IFT and can be derived from Eq. (2).
-
2.
H
i
represents the i line of the matrix H. \(\boldsymbol {H}_{i} \in \mathcal {M}_{1\times M}\)
-
3.
g
0 represents the “theoretical” signal, i.e., the signal corresponding to the considered model (2) that does not account for the noise, g
0=H
f. During the synthetic simulation section, the comparison between the estimated signal \({\widehat {\boldsymbol {g}_{0}}}\) and the theoretical signal g
0 is particular important, measuring if the propose algorithm selects the solution corresponding to the biological phenomena.
-
4.
f represents the PC vector, \(\boldsymbol {f} \in \mathcal {M}_{1\times M}\). This is the fundamental unknown of our model. All the estimates of the PC vector are denoted \({\widehat {\boldsymbol {f}}}\) and in specific cases the particular estimation used in the model is indicated: \({\widehat {\boldsymbol {f}}_{\textit {JMAP}}}\) or \({\widehat {\boldsymbol {f}}_{\text {PM}}}\). During the article, the subscript used for indicating an element of the PC vector is i: f
i
and the element is not bold, being a scalar.
-
5.
ε represents the errors: \(\boldsymbol {\epsilon } = \left [{\epsilon }_{1}, {\epsilon }_{2}, \ldots, {\epsilon }_{N}\right ]^{T} \in \mathcal {M}_{N \times 1},\) is an N-dimensional vector
List of abbreviations
-
1.
CT—circadian time
-
2.
CTS—circadian timing system
-
3.
FFT—fast Fourier transform
-
4.
IGSM—infinite Gaussian scale mixture
-
5.
IP—inverse problem
-
6.
JMAP—joint maximum a posteriori
-
7.
KL—Kullback-Leibler
-
8.
PC vector—periodic component vector
-
9.
PM—posterior mean
-
10.
RT-BIO—RealTime Biolumicorder
-
11.
TSVD—truncated single value decomposition
-
12.
TRM—Tikhonov regularization methods
-
13.
VBA—variational Bayesian approximation
-
14.
ZT—Zeitgeber time