This paper presents a variational Bayes approach to a Lévy adaptive regression kernel (LARK) model that represents functions with an overcomplete system. In particular, we develop a variational inference method for a LARK model with multiple kernels (LARMuK) which estimates arbitrary functions that could have jump discontinuities. The algorithm is based on a variational Bayes approximation method with simulated annealing. We compare the proposed algorithm to a simulation-based reversible jump Markov chain Monte Carlo (RJMCMC) method using numerical experiments and discuss its potential and limitations.

Authors and Affiliations

1. Samsung SDS, Seoul, South Korea

   Youngseon Lee

2. Department of Statistics, Inha University, Incheon, South Korea

   Seongil Jo

3. Department of Statistics, Seoul National University, Seoul, South Korea

   Jaeyong Lee

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Seongil Jo was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) grant funded by the Korea government(MIST) (No. 2020R1C1C1A01013338). Jaeyong Lee was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (Nos. 2018R1A2A3074973 and 2020R1A4A1018207)

Appendix

1.1A. Evidence Lower Bound

We here provide the details of the evidence lower bound (ELBO) for the proposed variational approximation algorithm. Recall that we regard the number of featuresJ as a variational parameter. The ELBO of the LARMuK model described in Sect. 3.1 is

To derive the ELBO, we first evaluate each term of the right side in (16) as follows:

• For the first term, we have

  $$\begin{aligned} \mathbb {E}_q\log p(\mathbf{Y}|\varvec{\beta }, \varvec{\chi }, \varvec{\lambda }, \mathbf{c},\sigma ^2, J)\propto & {} \mathbb {E}_q \left[ -\frac{n}{2}\log (2\pi \sigma ^2)-\frac{1}{2\sigma ^2} \sum _{i=1}^n (y_i-\mathbf{g_i}^T \varvec{\beta })^2\right] \\\propto & {} \frac{n}{2} \mathbb {E}_q \log \frac{1}{\sigma ^2}-\frac{1}{2} \mathbb {E}_q \frac{1}{\sigma ^2} \sum _{i=1}^n \mathbb {E}_q (y_i-\mathbf{g_i}^T \varvec{\beta })^2\\\propto & {} \frac{n}{2} \left( \Psi \left( \frac{r_0}{2}\right) -\log \frac{r_0 R_0}{2}\right) \\&-\,\frac{1}{2 R_0} \sum _{i=1}^n \mathbb {E}_q (y_i-\mathbf{g_i}^T \varvec{\beta })^2, \end{aligned}$$

  where\(\mathbf{g_i} = (g_{c_1} (x_i, w_1), \ldots , g_{c_J} (x_i, w_J))^T\) and

  $$\begin{aligned} \mathbb {E}_q \left[ \sum _{i=1}^n (y_i-\mathbf{g_i}^T \varvec{\beta })^2\right]= & {} \mathbb {E}_q \left[ \sum _{i=1}^n y_i^2-2\sum _{i=1}^n y_i (\mathbf{g_i}^T \varvec{\beta }) +\sum _{i=1}^n(\mathbf{g_i}^T \varvec{\beta })^2\right] \\= & {} \sum _{i=1}^ny_i^2 -2\sum _{i=1}^n y_i \mathbb {E}_q [\mathbf{g_i}^T \varvec{\beta }]+\sum _{i=1}^n\mathbb {E}_q [(\mathbf{g_i}^T \varvec{\beta })^2]\\= & {} \sum _{i=1}^ny_i^2 -2 \sum _{i=1}^n y_i \left[ \sum _{j=0}^{J_0}\mu _{0j} g_{0ij}\right] \\&+\,\sum _{i=1}^n\left[ \sum _{j=1}^{J_0}[\sigma _{0j}^2 g_{0ij}^2 +\mu _{0j}^2 g_{0ij}^2]+\sum _{l,j,l\ne j} [\mu _{0j}\mu _{0l} g_{0ij} g_{0il}]\right] \\= & {} \sum _{i=1}^n[y_i^2-2y_i (\mathbf{g_{0i}}^T {\varvec{\mu }}_0) +(\mathbf{g_{0i}}^T {\varvec{\mu }}_0)^2 + \sum _{j=1}^{J_0} \sigma _{0j}^2 g_{0ij}^2 ]\\= & {} \sum _{i=1}^n (y_i-\mathbf{g_{0i}}^T {\varvec{\mu }}_0)^2 + \sum _{j=1}^{J_0}\sum _{i=1}^n \sigma _{0j}^2 g_{0ij}^2. \end{aligned}$$

• For the second term, since the variational parameters\(\varvec{\beta }, \varvec{\chi }, \varvec{\lambda }, \mathbf{c}\), and\(\sigma ^2\) are determined independently under givenJ, we have

  $$\begin{aligned}&\mathbb {E}_q\log p(\varvec{\beta },\varvec{\chi }, \varvec{\lambda }, \mathbf{c}, \sigma ^2, J)\\= & {} \mathbb {E}_q\log p(\varvec{\beta }|J)+\mathbb {E}_q\log p(\varvec{\chi }|J)+\mathbb {E}_q\log p(\varvec{\lambda }|J)+\mathbb {E}_q\log p(\mathbf{c}|J)\\&+ \mathbb {E}_q\log p(\sigma ^2) + \mathbb {E}_q\log p(J), \end{aligned}$$

  and the each term is evaluated as

  $$\begin{aligned} \mathbb {E}_q \log p(\varvec{\beta }|J)= & {} \mathbb {E}_q \left[ -\frac{J_0}{2}\log (2\pi \sigma _\beta ^2)-\frac{1}{2\sigma _\beta ^2} \sum _{j=1}^{J_0} (\beta _j-0)^2\right] \\\propto & {} \frac{J_0}{2}\log \frac{1}{\sigma _\beta ^2}-\frac{1}{2\sigma _\beta ^2} \sum _{j=1}^{J_0} (\sigma _{0j}^2 +\mu _{0j}^2), \\ \mathbb {E}_q \log p(\varvec{\chi }|J)= & {} \sum _{j=1}^{J_0} \mathbb {E}_q \log p(\chi _j)\\= & {} \sum _{j=1}^{J_0} \mathbb {E}_q [\log I(0\le \chi _j \le 1)]\\= & {} \sum _{j=1}^{J_0} \log I(0\le \chi _{0j}\le 1) = 0, \\ \mathbb {E}_q \log p(\varvec{\lambda }|J)= & {} \sum _{j=1}^{J_0} \mathbb {E}_q \log p(\lambda _j)\\= & {} \sum _{j=1}^{J_0}\mathbb {E}_q [\log b_\lambda ^{a_\lambda }-\log \Gamma (a_\lambda )+(a_\lambda -1)\log \lambda _j - b_\lambda \lambda _j]\\\propto & {} \sum _{j=1}^{J_0} \left[ (a_\lambda -1) \mathbb {E}_q\log \lambda _j -b_\lambda \mathbb {E}_q\lambda _j\right] \\= & {} \sum _{j=1}^{J_0} \left[ (a_\lambda -1)\log \lambda _{0j}-b_\lambda \lambda _{0j}\right] \\= & {} (a_\lambda -1) \sum _{j=1}^{J_0}\log \lambda _{0j} -b_\lambda \sum _{j=1}^{J_0} \lambda _{0j},\\ \mathbb {E}_q \log p(\mathbf{c}|J)= & {} \sum _{j=1}^{J_0} \mathbb {E}_q \log p(c_j)\\\propto & {} \sum _{j=1}^{J_0} \mathbb {E}_q[I(c_j=0)\cdot \log p_0+ I(c_j=1)\cdot \log p_1 + I(c_j=2)\cdot \log p_2]\\\propto & {} \sum _{j=1}^{J_0}[\nu _{0j}\log p_0+\nu _{1j}\log p_1 +\nu _{2j}\log p_2]\\= & {} \log p_0 \sum _{j=1}^{J_0} \nu _{0j}+\log p_1 \sum _{j=1}^{J_0} \nu _{1j} +\log p_2 \sum _{j=1}^{J_0} \nu _{2j}, \\ \mathbb {E}_q \log p(\sigma ^2)= & {} \mathbb {E}_q \left[ \log \frac{r R}{2}^{r/2} -\log \Gamma \left( \frac{r}{2}\right) +\left( \frac{r}{2}-1\right) \log \frac{1}{\sigma ^2} - \frac{r R}{2\sigma ^2}\right] \\\propto & {} \left( \frac{r}{2}-1\right) \mathbb {E}_q \log \frac{1}{\sigma ^2}-\frac{r R}{2} \mathbb {E}_q \frac{1}{\sigma ^2}\\\propto & {} \left( \frac{r}{2}-1\right) \left( \Psi \left( \frac{r_0}{2}\right) -\log \frac{r_0 R_0}{2}\right) -\frac{r R}{2 R_0}, \\ \mathbb {E}_q\log p(J)= & {} \mathbb {E}_q [J\log M -M -\log J!]\\\propto & {} J_0\log M -\log J_0!. \end{aligned}$$

• Note that the joint entropy of random variables is calculated with the marginal and conditional entropies, i.e.,\(H_q(X|Y)=H_q(X,Y)-H_q(Y)\). Therefore, we need to compute the entropy of each variational parameter as

  $$\begin{aligned} H_q(\varvec{\beta }|J)= & {} -\mathbb {E}_q\log q(\varvec{\beta }|J) =-\mathbb {E}_{q(J)}\mathbb {E}_{q(\varvec{\beta }|J)}\log q(\varvec{\beta }|J) \\= & {} -\mathbb {E}_{q(J)} \sum _{j=1}^J \left[ -\frac{1}{2}\log (2\pi \sigma _{0j}^2)-\frac{1}{2\sigma _{0j}^2} \mathbb {E}_q(\beta _j-\mu _{0j})^2\right] \\= & {} \sum _{j=1}^{J_0} \left[ \frac{1}{2}\log (2\pi \sigma _{0j}^2)+\frac{1}{2\sigma _{0j}^2}\mathbb {E}_q(\beta _j-\mu _{0j})^2\right] \\= & {} \sum _{j=1}^{J_0} \left[ \frac{1}{2} \log (2\pi \sigma _{0j}^2)+\frac{1}{2}\right] \\\propto & {} \frac{1}{2} \sum _{j=1}^{J_0} \log \sigma _{0j}^2, \\ H_q(\varvec{\chi }|J)= & {} -\mathbb {E}_{q(J)}\mathbb {E}_{q(\varvec{\chi }|J)}(\log q(\varvec{\chi }|J))=-\sum _{j=1}^{J_0}\log I(\chi _j=\chi _{0j})=0,\\ H_q(\varvec{\lambda }|J)= & {} -\mathbb {E}_{q(J)}\mathbb {E}_{q(\varvec{\lambda }|J)}(\log q(\varvec{\lambda }|J))=-\sum _{j=1}^{J_0}\log I(\lambda _j=\lambda _{0j})=0, \\ H_q(\mathbf{c}|J)= & {} -\mathbb {E}_q\log q(\mathbf{c}|J) =-\mathbb {E}_{q(J)}\mathbb {E}_{q(\mathbf{c}|J)}\log q(\mathbf{c}|J)\\= & {} -\sum _{j=1}^{J_0} \mathbb {E}_q (\log q(c_j))= -[\nu _{j0} \log \nu _{j0}+\nu _{j1} \log \nu _{j1}+\nu _{j2} \log \nu _{j2}]\\= & {} -\sum _{j=1}^{J_0} [\nu _{j0} \log \nu _{j0}+\nu _{j1} \log \nu _{j1}+\nu _{j2} \log \nu _{j2}],\\ H_q(\sigma ^2)= & {} -\mathbb {E}_q \left[ \log \left( \frac{r_0 R_0}{2}\right) ^{\frac{r_0}{2}} -\log \Gamma \left( \frac{r_0}{2}\right) +\left( \frac{r_0}{2}-1\right) \log \frac{1}{\sigma ^2} - \frac{r_0 R_0}{2\sigma ^2}\right] \\= & {} \log \Gamma \left( \frac{r_0}{2}\right) -\frac{r_0}{2} \log \frac{r_0 R_0}{2}-\left( \frac{r_0}{2}-1\right) \mathbb {E}_q \log \frac{1}{\sigma ^2} +\frac{r_0 R_0}{2}\mathbb {E}_q \frac{1}{\sigma ^2}\\= & {} \log \Gamma \left( \frac{r_0}{2}\right) -\frac{r_0}{2} \log \frac{r_0 R_0}{2}-\left( \frac{r_0}{2}-1\right) \left( \Psi \left( \frac{r_0}{2}\right) -\log \frac{r_0 R_0}{2}\right) +\frac{r_0}{2}\\= & {} \log \Gamma \left( \frac{r_0}{2}\right) -\log \frac{r_0 R_0}{2}-\left( \frac{r_0}{2}-1\right) \Psi \left( \frac{r_0}{2}\right) +\frac{r_0}{2},\\ H_q(J)= & {} -\mathbb {E}_q \log q(J)=-\log I(J=J_0)=0. \end{aligned}$$

In summary, the ELBO is given as

Reprints and permissions