Movatterモバイル変換

[0]ホーム
URL:

Skip to main content

Advertisement

Springer Nature Link
Log in

FastDRC: Fast and Scalable Genome Compression Based on Distributed and Parallel Processing

  • Conference paper
  • First Online:

Abstract

With the advent of next-generation sequencing technology, sequencing costs have fallen sharply compared to the previous sequencing technologies. Genomic big data has become the significant big data application. In the face of growing genomic data, its storage and migration face enormous challenges. Therefore, researchers have proposed a variety of genome compression algorithms, but these algorithms cannot meet the processing requirements for large amount of biological data and high processing speed. This manuscript proposes a parallel and distributed referential genome compression algorithm-Fast Distributed Referential Compression (FastDRC). This algorithm compresses a large number of genomic sequences in parallel under the Apache Hadoop distributed computing framework. Experiments show that the compression efficiency of the FastDRC is greatly improved when it compresses large quantities of genomic data. Moreover, FastDRC leads to the only distributed computing method known to us in the field of genome compression. The source code for FastDRC can be obtained from this link:https://github.com/GhostCCCatHenry/FastDRC.

This is a preview of subscription content,log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 5719
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 7149
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide -see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Similar content being viewed by others

References

  1. Kahn, S.D.: On the future of genomic data. Science331(6018), 728–729 (2011)

    Article  Google Scholar 

  2. Pearson, W.R.: Rapid and sensitive sequence comparison with FASTP and FASTA. Methods Enzymol.183(1), 63–98 (1990)

    Article  Google Scholar 

  3. Xie, X., Zhou, S., Guan, J.: CoGI: towards compressing genomes as an image. IEEE/ACM Trans. Comput. Biol. Bioinform.12(6), 1275–1285 (2015)

    Article  Google Scholar 

  4. Deorowicz, S., Grabowski, S., Ochoa, I., et al.: ERGC: an efficient referential genome compression algorithm. Bioinformatics31(21), 3468–3475 (2015)

    Article  Google Scholar 

  5. Wandelt, S., Leser, U.: FRESCO: referential compression of highly similar sequences. IEEE/ACM Trans. Comput. Biol. Bioinform.10(5), 1275–1288 (2014)

    Article  Google Scholar 

  6. Wu, X.-D., Ji, S.-W.: Comparative study on MapReduce and spark for big data analytics. J. Softw.29(6), 1770–1791 (2018)

    MathSciNet  Google Scholar 

  7. Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The hadoop distributed file system. In: Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), MSST 2010, pp. 1–10. IEEE Computer Society, Washington, DC (2010)

    Google Scholar 

  8. Abecasis, G.: The 1000 genomes project consortium. An integrated map of genetic variation from 1,092 human genomes. Nature491, 56–65 (2012)

    Google Scholar 

  9. Vavilapalli, V.K,, Murthy, A.C., Douglas, C., Agarwal, S., Konar, M., Evans, R., et al.: Apache hadoop YARN: yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing, p. 5. ACM, New York (2013)

    Google Scholar 

  10. Liu, Y.S., et al.: High-speed and high-ratio referential genome compression. Bioinformatics33(21), 3364–3372 (2017)

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank all reviewers for their valuable comments and suggestions to improve the quality of our manuscript.

Funding

This work was supported by the National Key R&D Program of China [2017YFB1401302, 2017YFB0202200], the National Natural Science Foundation of P. R. China [No. 61572260, 61872196], Outstanding Youth of Jiangsu Natural Science Foundation [BK20170100], Key R&D Program of Jiangsu [BE2017166], Postgraduate Research & Practice Innovation Program of Jiangsu Province [KYCX19_0906, KYCX19_0921], The Natural Science Foundation of the Jiangsu Higher Education Institutions of China [19KJD520006] and Modern Educational Technology Research Program of Jiangsu Province in 2019 [2019-R-67748].

Author information

Authors and Affiliations

  1. School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing, 210023, China

    Yimu Ji, Houzhi Fang, Haichang Yao, Shuai Chen, Kui Li & Shangdong Liu

  2. School of Computer and Software, Nanjing Institute of Industry Technology, Nanjing, 210023, China

    Haichang Yao

  3. School of Software and Electrical Engineering, Swinburne University of Technology, Melbourne, 3122, Australia

    Jing He

  4. Institute of High Performance Computing and Big Data, Nanjing University of Posts and Telecommunications, Nanjing, 210003, China

    Yimu Ji

  5. Nanjing Center of HPC China, Nanjing, 210003, China

    Yimu Ji

  6. Jiangsu HPC and Intelligent Processing Engineer Research Center, Nanjing, 210003, China

    Yimu Ji

Authors
  1. Yimu Ji

    You can also search for this author inPubMed Google Scholar

  2. Houzhi Fang

    You can also search for this author inPubMed Google Scholar

  3. Haichang Yao

    You can also search for this author inPubMed Google Scholar

  4. Jing He

    You can also search for this author inPubMed Google Scholar

  5. Shuai Chen

    You can also search for this author inPubMed Google Scholar

  6. Kui Li

    You can also search for this author inPubMed Google Scholar

  7. Shangdong Liu

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence toHaichang Yao.

Editor information

Editors and Affiliations

  1. Department of Computer Science and Software Engineering, Swinburne University of Technology, Hawthorn, Melbourne, VIC, Australia

    Sheng Wen

  2. School of Computer Science, The University of Sydney, Camperdown, NSW, Australia

    Albert Zomaya

  3. Department of Computer Science, St. Francis Xavier University, Antigonish, NS, Canada

    Laurence T. Yang

Rights and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ji, Y.et al. (2020). FastDRC: Fast and Scalable Genome Compression Based on Distributed and Parallel Processing. In: Wen, S., Zomaya, A., Yang, L.T. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2019. Lecture Notes in Computer Science(), vol 11945. Springer, Cham. https://doi.org/10.1007/978-3-030-38961-1_27

Download citation

Publish with us

Access this chapter

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 5719
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 7149
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide -see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

[8]ページ先頭
©2009-2025 Movatter.jp