355Accesses
4Citations
Abstract
Recent advances in data processing have stimulated the demand for learning graphs of very large scales. Graph neural networks (GNNs), being an emerging and powerful approach in solving graph learning tasks, are known to be difficult to scale up. Most scalable models apply node-based techniques in simplifying the expensive graph message-passing propagation procedure of GNNs. However, we find such acceleration insufficient when applied to million- or even billion-scale graphs. In this work, we proposeSCARA, a scalable GNN with feature-oriented optimization for graph computation.SCARA efficiently computes graph embedding from the dimension of node features, and further selects and reuses feature computation results to reduce overhead. Theoretical analysis indicates that our model achieves sub-linear time complexity with a guaranteed precision in propagation process as well as GNN training and inference. We conduct extensive experiments on various datasets to evaluate the efficacy and efficiency ofSCARA. Performance comparison with baselines shows thatSCARA can reach up to\(800\times \) graph propagation acceleration than current state-of-the-art methods with fast convergence and comparable accuracy. Most notably, it is efficient to process precomputation on the largest available billion-scale GNN dataset Papers100M (111 M nodes, 1.6 B edges) in 13 s.
This is a preview of subscription content,log in via an institution to check access.
Access this article
Subscribe and save
- Get 10 units per month
- Download Article/Chapter or eBook
- 1 Unit = 1 Article or 1 Chapter
- Cancel anytime
Buy Now
Price includes VAT (Japan)
Instant access to the full article PDF.








Similar content being viewed by others
Notes
The source code and data used in the paper have been made available at:https://github.com/gdmnl/SCARA-PPR
References
Al-Rfou, R., Perozzi, B., Zelle, D.: Ddgk: learning graph representations for deep divergence graph kernels. In: The World Wide Web Conference, pp. 37–48 (2019)
Andersen, R., Borgs, C., Chayes, J., Hopcraft, J., Mirrokni, V.S., Teng, S.H.: Local computation of pagerank contributions. In: Algorithms and models for the web-graph, vol. 4863, pp. 150–165. Springer Berlin Heidelberg, Berlin, Heidelberg (2007)
Andersen, R., Chung, F., Lang, K.: Local graph partitioning using PageRank vectors. In: 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS’06), pp. 475–486. IEEE (2006)
Atwood, J., Towsley, D.: Diffusion-convolutional neural networks. In: 29th Advances in Neural Information Processing Systems pp. 2001–2009 (2016)
Bojchevski, A., Klicpera, J., Perozzi, B., Kapoor, A., Blais, M., Rózemberczki, B., Lukasik, M., Günnemann, S.: Scaling graph neural networks with approximate pagerank. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 2464–2473 (2020)
Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM58(3), 1–37 (2011)
Chandrasekaran, V., Sanghavi, S., Parrilo, P.A., Willsky, A.S.: Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim.21(2), 572–596 (2011)
Chen, Z., Bruna, J., Li, L.: Supervised community detection with line graph neural networks. In: 7th International Conference on Learning Representations (2019)
Chen, J., Ma, T., Xiao, C.: Fastgcn: fast learning with graph convolutional networks via importance sampling. In: International Conference on Learning Representations (2018)
Chen, M., Wei, Z., Ding, B., Li, Y., Yuan, Y., Du, X., Wen, J.R.: Scalable graph neural networks via bidirectional propagation. In: 33rd Advances in Neural Information Processing Systems (2020)
Chen, J., Zhu, J., Song, L.: Stochastic training of graph convolutional networks with variance reduction. Int. C. Mach. Learn.3, 1503–1532 (2018)
Chiang, W.L., Liu, X., Si, S., Li, Y., Bengio, S., Hsieh, C.J.: Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 257–266 (2019)
Chien, E., Peng, J., Li, P., Milenkovic, O.: Adaptive universal generalized pagerank graph neural network. In: 9th International Conference on Learning Representations (2021)
Coudron, M., Lerman, G.: On the sample complexity of robust PCA. In: 25th Advances in Neural Information Processing Systems (2012)
Ding, M., Kong, K., Li, J., Zhu, C., Dickerson, J.P., Huang, F., Goldstein, T.: VQ-GNN: a universal framework to scale up graph neural networks using vector quantization. In: 34th Advances in Neural Information Processing Systems (2021)
Fey, M., Lenssen, J.E., Weichert, F., Leskovec, J.: GNNAutoScale: scalable and expressive graph neural networks via historical embeddings. In: 38th International Conference on Machine Learning. PMLR 139 (2021)
Fogaras, D., Rácz, B., Csalogány, K., Sarlós, T.: Towards scaling fully personalized pagerank: algorithms, lower bounds, and experiments. Inter. Math.2(3), 333–358 (2005)
Hamilton, W.L., Ying, R., Leskovec, J.: Inductive representation learning on large graphs. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1025–1035 (2017)
Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., Leskovec, J., Barzilay, R., Battaglia, P., Bengio, Y., Bronstein, M., Günnemann, S., Hamilton, W., Jaakkola, T., Jegelka, S., Nickel, M., Re, C., Song, L., Tang, J., Welling, M., Zemel, R.: Open graph benchmark : datasets for machine learning on graphs. In: 33rd Advances in Neural Information Processing Systems (2020)
Huang, Z., Zhang, S., Xi, C., Liu, T., Zhou, M.: Scaling up graph neural networks via graph coarsening. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, vol. 1, pp. 675–684 (2021)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations (2017)
Klicpera, J., Bojchevski, A., Günnemann, S.: Predict then propagate: graph neural networks meet personalized pagerank. In: 7th International Conference on Learning Representations, pp. 1–15 (2019)
Li, X., Zhu, R., Cheng, Y., Shan, C., Luo, S., Li, D., Qian, W.: Finding global homophily in graph neural networks when meeting heterophily. In: 39th International Conference on Machine Learning (2022)
Liao, N., Luo, S., Li, X., Shi, J.: LD2: Scalable heterophilous graph neural network with decoupled embedding. In: Advances in Neural Information Processing Systems, vol. 36 (2023)
Liao, N., Mo, D., Luo, S., Li, X., Yin, P.: SCARA: scalable graph neural networks with feature-oriented optimization. Proc. VLDB Endowm.15(11), 3240–3248 (2022)
Lin, Z., Liu, R., Su, Z.: Linearized alternating direction method with adaptive penalty for low-rank representation. In: 24th Advances in Neural Information Processing Systems (2011)
Lin, D., Wong, R.C.W., Xie, M., Wei, V.J.: Index-free approach with theoretical guarantee for efficient random walk with restart query. In: 2020 IEEE 36th International Conference on Data Engineering (ICDE), pp. 913–924 (2020)
Liu, H., Liao, N., Luo, S.: Simga: A simple and effective heterophilous graph neural network with efficient global aggregation. arXiv e-prints (2023)
Mo, D., Luo, S.: Agenda: robust personalized pageranks in evolving graphs. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 1315–1324. ACM, Virtual Event Queensland Australia (2021)
Page, L., Brin, S., Motwani, R., Winograd, T.: The pagerank citation ranking: bringing order to the web. Tech. rep. (1999)
Sinha, A., Shen, Z., Song, Y., Ma, H., Eide, D., Hsu, B.J.P., Wang, K.: An overview of microsoft academic service (mas) and applications. In: Proceedings of the 24th International Conference on World Wide Web, pp. 243–246 (2015)
Sun, L., Dou, Y., Yang, C., Wang, J., Yu, P.S., He, L., Li, B.: Adversarial attack and defense on graph data: a survey. arXiv e-prints (2018)
Thekumparampil, K.K., Wang, C., Oh, S., Li, L.J.: Attention-based graph neural network for semi-supervised learning. arXiv e-prints (2018)
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. In: 8th International Conference on Learning Representations (2017)
Wang, H., He, M., Wei, Z., Wang, S., Yuan, Y., Du, X., Wen, J.R.: Approximate graph propagation. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, vol. 1, pp. 1686–1696. Association for Computing Machinery (2021)
Wang, K., Luo, S., Lin, D.: River of no return: graph percolation embeddings for efficient knowledge graph reasoning. arXiv e-prints (2023)
Wang, C., Pan, S., Long, G., Zhu, X., Jiang, J.: Mgae: marginalized graph autoencoder for graph clustering. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM ’17, p. 889–898. Association for Computing Machinery, New York, NY, USA (2017)
Wang, X., Zhang, M.: How powerful are spectral graph neural networks. In: 39th International Conference on Machine Learning (2022)
Wang, S., Yang, R., Xiao, X., Wei, Z., Yang, Y.: Fora: simple and effective approximate single-source personalized pagerank. Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data MiningPart F1296, 505–514 (2017)
Wang, S., Yang, R., Wang, R., Xiao, X., Wei, Z., Lin, W., Yang, Y., Tang, N.: Efficient algorithms for approximate single-source personalized PageRank queries. ACM Trans. Datab. Syst.44(4), 1–37 (2019)
Wu, H., Gan, J., Wei, Z., Zhang, R.: Unifying the global and local approaches: an efficient power iteration with forward push. In: Proceedings of the 2021 International Conference on Management of Data, vol. 1, pp. 1996–2008 (2021)
Wu, F., Souza, A., Zhang, T., Fifty, C., Yu, T., Weinberger, K.: Simplifying graph convolutional networks. In: K. Chaudhuri, R. Salakhutdinov (eds.) Proceedings of the 36th International Conference on Machine Learning, vol. 97, pp. 6861–6871 (2019)
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., Yu, P.S.: A comprehensive survey on graph neural networks. IEEE Trans. Neur. Netw. Learn. Syst.32(1), 4–24 (2021)
Yang, R., Shi, J., Xiao, X., Yang, Y., Liu, J., Bhowmick, S.S.: Scaling attributed network embedding to massive graphs. Proc. VLDB Endow.14(1), 37–49 (2021)
Ying, R., He, R., Chen, K., Eksombatchai, P., Hamilton, W.L., Leskovec, J.: Graph convolutional neural networks for web-scale recommender systems. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 974–983 (2018)
Zeng, H., Zhou, H., Srivastava, A., Kannan, R., Prasanna, V.: Graphsaint: Graph sampling based learning method. In: International Conference on Learning Representations (2019)
Zhang, J., Zhang, H., Xia, C., Sun, L.: Graph-bert: only attention is needed for learning graph representations. arXiv e-prints (2020)
Zhang, M., Chen, Y.: Link prediction based on graph neural networks. Adv. Neural Inf. Process. Syst.31, 5165–5175 (2018)
Zhang, Z., Cui, P., Zhu, W.: Deep learning on graphs: a survey. IEEE Tran. Knowl. Data Eng.14(8), 1 (2020)
Zhu, Z., Peng, J., Li, J., Chen, L., Yu, Q., Luo, S.: Spiking graph convolutional networks. In: 31th International Joint Conference on Artificial Intelligence (2022)
Zügner, D., Akbarnejad, A., Günnemann, S.: Adversarial attacks on neural networks for graph data. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2847–2856 (2018)
Acknowledgements
This research is supported by Singapore MOE funding (MOE-T2EP20122-0003, RS05/21), NTU-NAP Startup Grant (022029-00001), and the Joint NTU-WeBank Research Centre on FinTech. Xiang Li is supported by Shanghai Science and Technology Committee General Program No. 22ZR1419900 and National Natural Science Foundation of China No. 62202172.
Author information
Authors and Affiliations
Nanyang Technological University, Singapore, Singapore
Ningyi Liao, Dingheng Mo & Siqiang Luo
East China Normal University, Shanghai, China
Xiang Li
Google Research, San Francisco, USA
Pengcheng Yin
- Ningyi Liao
You can also search for this author inPubMed Google Scholar
- Dingheng Mo
You can also search for this author inPubMed Google Scholar
- Siqiang Luo
You can also search for this author inPubMed Google Scholar
- Xiang Li
You can also search for this author inPubMed Google Scholar
- Pengcheng Yin
You can also search for this author inPubMed Google Scholar
Corresponding author
Correspondence toSiqiang Luo.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Liao, N., Mo, D., Luo, S.et al. Scalable decoupling graph neural network with feature-oriented optimization.The VLDB Journal33, 667–683 (2024). https://doi.org/10.1007/s00778-023-00829-6
Received:
Revised:
Accepted:
Published:
Issue Date:
Share this article
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative