Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up

Go library for downloading content from internet, in parallel.

License

NotificationsYou must be signed in to change notification settings

c4milo/gofetch

Repository files navigation

Build StatusGoDoc

Go library to download files from the internerds using Go 1.7 or greater.

Features

  • Resumes downloads if interrupted.
  • Allows parallel downloading of a single file by requesting multiple data chunks at once over HTTP.
  • Reports download progress through a Go channel if indicated to do so.
  • Supports file integrity verification if a checksum is provided.
  • Supports ETags, skipping downloading a file if it hasn't changed on the server.
  • Can be combined withhttps://github.com/cenkalti/backoff to support retrying with exponential back-off

Gotchas

When downloading file chunks concurrently, you may encounter some issues:

  • Servers may limit the number of concurrent connections you have open against them
  • Servers may accept the connections but will not send anything, in this case the default HTTP client will timeout for you.If you provide your own client, make sure it has proper timeouts or your connection will block for several seconds, dependingon your operating system network timeouts.

Example

package mainimport ("fmt""io""os""github.com/c4milo/gofetch")funcmain() {gf:=gofetch.New(gofetch.WithDestDir(os.TempDir()),gofetch.WithConcurrency(10),gofetch.WithETag(),)progressCh:=make(chan gofetch.ProgressReport)varmyFile*os.Filegofunc() {varerrerrormyFile,err=gf.Fetch("http://releases.ubuntu.com/15.10/ubuntu-15.10-server-amd64.iso",progressCh)iferr!=nil {panic(err)}}()// pogressCh is close by gofetch once a download finishesvartotalWrittenint64forp:=rangeprogressCh {// p.WrittenBytes does not accumulate, it represents the chunk size written// in the current operation.totalWritten+=p.WrittenBytesfmt.Printf("%d of %d\n",totalWritten,p.Total)}destFile,err:=os.Create("/tmp/ubuntu-15.10-server-amd64.iso")iferr!=nil {panic(err)}deferfunc() {iferr:=destFile.Close();err!=nil {panic(err)}}()if_,err:=io.Copy(destFile,myFile);err!=nil {panic(err)}}

About

Go library for downloading content from internet, in parallel.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages


[8]ページ先頭

©2009-2025 Movatter.jp