A new model-free and data-adaptive feature screening procedure referred to as the concordance filter is developed for ultrahigh-dimensional data. The proposed method is based on the concordance filter which measures concordance between random vectors and can work adaptively with several types of predictors and response variables. We apply the concordance filter to deal with feature screening problems emerging from a wide range of real applications, such as nonparametric regression and survival analysis, among others. It is shown that the concordance filter enjoys the sure screening and rank consistency properties under weak regularity conditions. In particular, the concordance filter can still be powerful in the presence of censoring and heavy tails. We further demonstrate the superior performance of the concordance filter over existing screening methods by numerical examples and medical applications.

We are grateful for resources from the High Performance Computing Center of Central South University. We have developed theMFSIS R package about the proposed CSIS method and this package is available in R-CRAN viahttps://cran.r-project.org/package=MFSIS. We acknowledge the support in part by National Statistical Scientific Research Project of China (No. 2022LZ28 & 2021LY042), Postgraduate Scientific Research Innovation Project of Hunan Province (No. CX20200148), Changsha Municipal Natural Science Foundation (No. kq2202080) and the Open Research Fund from the Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen(No. B10120210117-OF04)

1. Xuewei Cheng: Conceptualization, Methodology, Software, Writing original draft. Gang Li: Writing–review & editing, Validation. Hong Wang: Supervision, Writing–review & editing.

Authors and Affiliations

1. MOE-LCSM, School of Mathematics and Statistics, Hunan Normal University, Changsha, China

   Xuewei Cheng

2. Key Laboratory of Applied Statistics and Data Science, College of Hunan Province, Hunan Normal University, Changsha, China

   Xuewei Cheng

3. School of Mathematics and Statistics, Central South University, Changsha, China

   Xuewei Cheng & Hong Wang

4. School of Public Health, University of California Los Angeles, Los Angeles, USA

   Gang Li

Corresponding author

Correspondence toHong Wang.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Xuewei Cheng, Gang Li and Hong Wang have contributed equally to this work.

Appendix A Proof of the sure screening property

In this Appendix, we first prove the property of rank consistency of the proposed method by concentration inequalities. Then, we provide the proof of property of sure screening by Hoeffing’s inequality for U-statistic. All of the techniques of proof of theorems are followed from Li et al. (2012a) and Song et al. (2014).

Proof of Theorem 1

We show that\(E|\omega _{k}|\ge c_{0}n^{-\kappa }\) for some\(c_{0}>0\), if\(k\in {\mathcal {M}}_{*}\), we have

where\(F_{\Delta \epsilon _{k}\vert \Delta m_{k}}\) is the conditional cumulative distribution function of\(\Delta \epsilon _{k}=H(Y_{1})-H(Y_{2})-\rho _{k}^{*}\{m_{k}(X_{1k})-m_{k}(X_{2k})\}\) given\(\Delta m_{k}=m_{k}(X_{1k})-m_{k}(X_{2k})\). The second equality holds since the status\(\delta\) is independent of the responseY. The third equality holds since\(E\{I(Y_{1}<Y_{2})\}=1/2\).

Because\(m_{k}(\cdot )\) is monotone,\(m_{k}(X_{1k})-m_{k}(X_{2k})\) is either greater than or less than zero for all\(X_{1k}<X_{2k}\). This implies that\(1-F_{\Delta \epsilon _{k}\vert \Delta m_{k}}(-\rho _{k}^{*}\Delta m_{k})\) is either greater or less than 1/2 due to Condition (C3).\(E(\omega _{k})\) is either greater or less than zero for\(k\in {\mathcal {M}}_{*}\). From the above analysis and taking into account the symmetry of\(F_{\Delta \epsilon _{k}\vert \Delta m_{k}}(t)\), we have

In the following, we further establish the lower bound of\(\vert E(\omega _{k})\vert\).

Without loss of generality, assuming\(m_{k}(\cdot )\) is monotone increasing and\(\rho _{k}^{*}>c^{*}n^{-\kappa }\). Note that\(E(\omega _{k})\) can be equivalently written as

By the Gaussian inequality for the symmetric unimodal distribution (Pukelsheim1994; Sellke and Sellke1997),

whereX is a unimodal random variable with a mode at the origin zero and variance\(\sigma ^{2}\). Using the Gaussian inequality for the symmetric unimodal distribution obtained above and condition (C4), we have

By the condition (C2) and the inequality\(E|X-Y|>E|X |\), where bothX andY are i.i.d random variables with\(E(X)=E(Y)=0\). We have

for some constant\(M>0\). By Condition (C3-C4), we have

By Chebyshev’s inequality and Condition (C4), the second term\(I_{2}\) can be further bounded below as

where the second inequality holds due to Condition (C3) and Inequality (A6). The above two inequalities for\(I_{1}\) and\(I_{2}\) yield that

Let\(M=8\sigma _{1}^{2}/c_{{\mathcal {M}}_{*}}\). Then,\(M\ge \sqrt{3}\sigma _{2}\) whenn is large. We have\(E(\omega _{k})\ge c_{0}n^{-\kappa }\), where\(c_{0}=\frac{c^{*} c^{2}_{{\mathcal {M}}_{*}}}{32\sigma _{1}^2}\). Similarly, if\(m_{k}(\cdot )\) is monotone decreasing and\(\rho _{k}^{*}<-c^{*}n^{-\kappa }\), we can show that\(E(\omega _{k})\le -c_{0}n^{-\kappa }\). Therefore,\(E(\vert \omega _{k}\vert )\ge c_{0}n^{-\kappa }\) when\(\vert \rho _{k}^{*}\vert >c^{*}n^{-\kappa }\) for any\(k\in {\mathcal {M}}_{*}\). Theorem 1 is then proved.\(\square\)

Lemma 1

Hoeffing’s inequality for U-statistics (Hoeffding1963) Let\(h=(x_{1},x_{2},...,x_{m})\) be a symmetric kernel of the U-statistics\(U_{n}\), with\(a\le (x_{1},x_{2},...,x_{m})\le b\). Then, for\(t>0\) and\(m<n\), we have

Proof of Theorem 2

By the definition of\(\omega _{k}\) (3), we have, for each\(k=1,2,...,p\),

where\(K=\sum _{j \ne i}^{n}I(Y_{i}<Y_{j},\delta _{i}=1)\) is the number of all comparable pairs (in particular,\(K=\left( {\begin{array}{c}n\\ 2\end{array}}\right)\) when data is fully observed) and\(h[(X_{ik},Y_{i}),(X_{jk},Y_{j})]=[\delta _{i}I(X_{ik}<X_{jk})I(Y_{i}<Y_{j}) +\delta _{j}I(X_{ik}>X_{jk})I(Y_{i}>Y_{j})]-\frac{1}{2}\).

This means that\(\omega _{k}\) is a U-statistics with symmetric kernel\(h[(X_{ik},Y_{i}),(X_{jk},Y_{j})]\). As the indicator function is bounded by 1, we have that

Also,\(-\frac{1}{2}\le E(\omega _{k})\le \frac{1}{2}\). Taking application of Hoeffding’s inequality of Lemma 1, for\(0<\kappa <\frac{1}{2}\) and any\(c_{1}>0\), there exists a positive constant\(c_{2}\) such that

where\(c_{2}=c_{1}^{2}\). Thus,

Denote the set\(S_{n}=\left\{ \mathop {\text {max}}\limits _{k \in {\mathcal {M}}_{*}}\mid \omega _{k}-E(\omega _{k})\mid \ge c_{0}n^{-\kappa }/2\right\}\). On this set\(S_{n}\) and the result of Theorem 1, we have

From inequality (A13), it is easy to check that there exists a positive constant\(c_{2}\) such that

By such a choice of\(\gamma _{n}=c_{3}n^{-\kappa }\) with\(c_{3}\le c_{0}/2\), we have

Appendix B Example 4: other complicated nonlinear models

In this subsection, we illustrate that the CSIS can be applied to more complicated scenarios. Following DC-SIS (Li et al.2012b) and PC-Screen (Liu et al.2022), we consider the following four sophisticated scenarios with\(\varepsilon \sim N(0,1)\). These models are frequently applied to economic time series analysis (Lovell1963) and income inequality study (Klein et al.2015).

Model 4.a (Dummy variables):\(Y=3X_{1}+3\text {sgn}(X_{2})+3I(X_{3}>0.5) +3I(X_{4}<0.5)+3I(-0.5<X_{5}<0.5)+\varepsilon\)

Model 4.b (Additive structure):\(Y=5X_{1}+2\text {sin}(\pi X_{2}/2)+2X_{3}I(X_{3}>0) +2\text {exp}(5X_{4})+3(1/X_{5})+\varepsilon\)

Model 4.c (A sophisticated nonlinear model & strong signal):\(Y=5\text {arcsin}[X_{1}I(-1\le X_{1}\le 1)] +2\text {arccos}[X_{2}I(-1\le X_{2}\le 1)] +2\text {arctan}(X_{3}) +2\text {tan}[X_{4}I(-\frac{\pi }{2}\le X_{4}\le \frac{\pi }{2})] +2\text {sin}[X_{5}I(-\frac{\pi }{2}\le X_{5}\le \frac{\pi }{2})] +\varepsilon\)

Model 4.d (A sophisticated nonlinear model & weak signal):\(Y=\text {arcsin}[X_{1}I(-1\le X_{1}\le 1)] -0.8\text {arccos}[X_{2}I(-1\le X_{2}\le 1)]+0.6\text {arctan}(X_{3}) -0.4\text {tan}[X_{4}I(-\frac{\pi }{2}\le X_{4}\le \frac{\pi }{2})] +0.2\text {sin}[X_{5}I(-\frac{\pi }{2}\le X_{5}\le \frac{\pi }{2})] +\varepsilon\)

The average values of the proportion\({\mathcal {S}}\) and the minimum model size\({\mathcal {R}}\) that includes all five active features are presented in Table7 under 100 replications. In the nonlinear examples, all five competitors perform well. From Model 4.a to Model 4.d, DC-SIS performs comparably to CSIS in each scenario and outperforms reasonably than other three methods, especially in Model 4.c and Model 4.d. As a result, CSIS enjoys the sure screening property requiring a relatively smaller model size to recover the truly important variables in this example.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions