dsqueue
packagemodule
This package is not in the latest version of its module.
Details
Validgo.mod file
The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
- Redistributable license
Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
- Tagged version
Modules with tagged versions give importers more predictable builds.
Stable versionWhen a project reaches major version v1 it is considered stable.
- Learn more about best practices
Repository
Links
Open Source Insights
README¶
go-dsqueue
Buffered FIFO interface to the datastore
The dsqueue package provides a buffered FIFO queue backed by aBatching Datastore. Queued items are persisted in the datastore when the input buffer is full, after some amount of idle time, and when the queue is shuddown.
Documentation
https://pkg.go.dev/github.com/ipfs/go-dsqueue
Example
ds = getBatchingDatastore() dsq := dsqueue.New(ds, "ExampleQueue")defer dsq.Close()c, err := cid.Decode("QmPNHBy5h7f19yJDt7ip9TvmMRbqmYsa6aetkrsc1ghjLB")if err != nil { panic(err)}dsq.Put(c.Bytes())out := <-dsq.Out()c2, err := cid.Parse(out)if err != nil { panic(err)}if c2 != c { fmt.Fprintln(os.Stderr, "cids are not quual")}Lead Maintainer
Contributing
Contributions are welcome! This repository is part of the IPFS project and therefore governed by ourcontributing guidelines.
License
Documentation¶
Overview¶
Package dsqueue provides a buffered FIFO interface to the datastore forstoring and retrieving items. Queued items are persisted across restarts.
Index¶
Constants¶
const (DefaultBufferSize = 16 * 1024DefaultIdleWriteTime =time.MinuteDefaultCloseTimeout = 10 *time.Second)
Variables¶
This section is empty.
Functions¶
This section is empty.
Types¶
typeDSQueue¶
type DSQueue struct {// contains filtered or unexported fields}DSQueue provides a FIFO interface to the datastore for storing items.
Items in the process of being provided when a crash or shutdown occurs maybe in the queue when the node is brought back online depending on whetherthey were fully written to the underlying datastore.
Input to the queue is buffered in memory. The contents of the buffer arewritten to the datastore when the input buffer is full (seeWithBufferSize), or when the queue has been idle for some time (seeWithIdleWriteTime) since the previous batch write or dequeue. Items todequeue are read, in order, from the input buffer if there are none in thedatastore. Otherwise they are read from the datastore.
If queued items are read from the input buffer before it reaches its limit,then queued items can remain in memory. When the queue is closed, anyremaining items in memory are written to the datastore.
func (*DSQueue)Clear¶
Clear clears all queued records from memory and the datastore. Returns thenumber of items removed from the queue.
func (*DSQueue)GetN¶added inv0.0.4
GetN retrieves up to n items that are currently available in the queue. Ifthere are no items currently available, then none are returned and GetN doesnot wait for any.
GetN is used to poll the DSQueue for items and return batches of thoseitems. This is the most efficient way of fetching currently available items.
GetN and Out can both be used to read items from the DSQueue, but theyshould not be used concurrently as items will be returned by one or theother indeterminately.
typeOption¶
type Option func(*config)
Option is a function that sets a value in a config.
funcWithBufferSize¶
WithBufferSize sets the limit on number of items kept in input buffermemory, at which they are all written to the datastore. A value of 0 meansthe buffer size is unlimited, and items are only written to the datastorewhen the queue has been idle more then the idle write time or when the queueis closed.
funcWithCloseTimeout¶
WithCloseTimeout sets the duration that Close waits to finish writing itemsto the datastore. A value of 0 means wait until finished with no timeout.
funcWithDedupCacheSize¶
WithDedupCacheSize sets the size of the LRU cache used to deduplicate itemsin the queue.
By default, the deduplication cache is disabled (size = 0).
funcWithIdleWriteTime¶
WithIdleWriteTime sets the amout of time that the queue must be idle (noinput or output) before all buffered input items are written to thedatastore. A value of zero means that buffered input items are notautomatically flushed to the datastore. A non-zero value must be greaterthan one second.
Source Files