mem
package
This package is not in the latest version of its module.
Details
Validgo.mod file
The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
- Redistributable license
Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
- Tagged version
Modules with tagged versions give importers more predictable builds.
- Stable version
When a project reaches major version v1 it is considered stable.
- Learn more about best practices
Repository
Links
Open Source Insights
Documentation¶
Overview¶
Package mem provides utilities that facilitate memory reuse in byte slicesthat are used as buffers.
Experimental¶
Notice: All APIs in this package are EXPERIMENTAL and may be changed orremoved in a later release.
Index¶
- func IsBelowBufferPoolingThreshold(size int) bool
- func NewWriter(buffers *BufferSlice, pool BufferPool) io.Writer
- type Buffer
- type BufferPool
- type BufferSlice
- type NopBufferPool
- type Reader
- func (r *Reader) Close() error
- func (r *Reader) Discard(n int) (discarded int, err error)
- func (r *Reader) Peek(n int, res [][]byte) ([][]byte, error)
- func (r *Reader) Read(buf []byte) (n int, _ error)
- func (r *Reader) ReadByte() (byte, error)
- func (r *Reader) Remaining() int
- func (r *Reader) Reset(s BufferSlice)
- type SliceBuffer
Examples¶
Constants¶
This section is empty.
Variables¶
This section is empty.
Functions¶
funcIsBelowBufferPoolingThreshold¶
IsBelowBufferPoolingThreshold returns true if the given size is less than orequal to the threshold for buffer pooling. This is used to determine whetherto pool buffers or allocate them directly.
funcNewWriter¶
func NewWriter(buffers *BufferSlice, poolBufferPool)io.Writer
NewWriter wraps the given BufferSlice and BufferPool to implement theio.Writer interface. Every call to Write copies the contents of the givenbuffer into a new Buffer pulled from the given pool and the Buffer isadded to the given BufferSlice.
Example¶
/* * * Copyright 2024 gRPC authors. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * */package mainimport ("bytes""crypto/rand""errors""fmt""io""testing""google.golang.org/grpc/mem")const (minReadSize = 1// Should match the constant in buffer_slice.go (another package)readAllBufSize = 32 * 1024 // 32 KiB)func newBuffer(data []byte, pool mem.BufferPool) mem.Buffer {return mem.NewBuffer(&data, pool)}func (s) TestBufferSlice_Len(t *testing.T) {tests := []struct {name stringin mem.BufferSlicewant int}{{name: "empty",in: nil,want: 0,},{name: "single",in: mem.BufferSlice{newBuffer([]byte("abcd"), nil)},want: 4,},{name: "multiple",in: mem.BufferSlice{newBuffer([]byte("abcd"), nil),newBuffer([]byte("abcd"), nil),newBuffer([]byte("abcd"), nil),},want: 12,},}for _, tt := range tests {t.Run(tt.name, func(t *testing.T) {if got := tt.in.Len(); got != tt.want {t.Errorf("BufferSlice.Len() = %v, want %v", got, tt.want)}})}}func (s) TestBufferSlice_Ref(t *testing.T) {// Create a new buffer slice and a reference to it.bs := mem.BufferSlice{newBuffer([]byte("abcd"), nil),newBuffer([]byte("abcd"), nil),}bs.Ref()// Free the original buffer slice and verify that the reference can still// read data from it.bs.Free()got := bs.Materialize()want := []byte("abcdabcd")if !bytes.Equal(got, want) {t.Errorf("BufferSlice.Materialize() = %s, want %s", string(got), string(want))}}func (s) TestBufferSlice_MaterializeToBuffer(t *testing.T) {tests := []struct {name stringin mem.BufferSlicepool mem.BufferPoolwantData []byte}{{name: "single",in: mem.BufferSlice{newBuffer([]byte("abcd"), nil)},pool: nil, // MaterializeToBuffer should not use the pool in this case.wantData: []byte("abcd"),},{name: "multiple",in: mem.BufferSlice{newBuffer([]byte("abcd"), nil),newBuffer([]byte("abcd"), nil),newBuffer([]byte("abcd"), nil),},pool: mem.DefaultBufferPool(),wantData: []byte("abcdabcdabcd"),},}for _, tt := range tests {t.Run(tt.name, func(t *testing.T) {defer tt.in.Free()got := tt.in.MaterializeToBuffer(tt.pool)defer got.Free()if !bytes.Equal(got.ReadOnlyData(), tt.wantData) {t.Errorf("BufferSlice.MaterializeToBuffer() = %s, want %s", string(got.ReadOnlyData()), string(tt.wantData))}})}}func (s) TestBufferSlice_Reader(t *testing.T) {bs := mem.BufferSlice{newBuffer([]byte("abcd"), nil),newBuffer([]byte("abcd"), nil),newBuffer([]byte("abcd"), nil),}wantData := []byte("abcdabcdabcd")reader := bs.Reader()var gotData []byte// Read into a buffer of size 1 until EOF, and verify that the data matches.for {buf := make([]byte, 1)n, err := reader.Read(buf)if n > 0 {gotData = append(gotData, buf[:n]...)}if err == io.EOF {break}if err != nil {t.Fatalf("BufferSlice.Reader() failed unexpectedly: %v", err)}}if !bytes.Equal(gotData, wantData) {t.Errorf("BufferSlice.Reader() returned data %v, want %v", string(gotData), string(wantData))}// Reader should have released its references to the underlying buffers, but// bs still holds its reference and it should be able to read data from it.gotData = bs.Materialize()if !bytes.Equal(gotData, wantData) {t.Errorf("BufferSlice.Materialize() = %s, want %s", string(gotData), string(wantData))}}// TestBufferSlice_ReadAll_Reads exercises ReadAll by allowing it to read// various combinations of data, empty data, EOF.func (s) TestBufferSlice_ReadAll_Reads(t *testing.T) {testcases := []struct {name stringreads []readStepwantErr stringwantBufs int}{{name: "EOF",reads: []readStep{{err: io.EOF,},},},{name: "data,EOF",reads: []readStep{{n: minReadSize,},{err: io.EOF,},},wantBufs: 1,},{name: "data+EOF",reads: []readStep{{n: minReadSize,err: io.EOF,},},wantBufs: 1,},{name: "0,data+EOF",reads: []readStep{{},{n: minReadSize,err: io.EOF,},},wantBufs: 1,},{name: "0,data,EOF",reads: []readStep{{},{n: minReadSize,},{err: io.EOF,},},wantBufs: 1,},{name: "data,data+EOF",reads: []readStep{{n: minReadSize,},{n: minReadSize,err: io.EOF,},},wantBufs: 1,},{name: "error",reads: []readStep{{err: errors.New("boom"),},},wantErr: "boom",},{name: "data+error",reads: []readStep{{n: minReadSize,err: errors.New("boom"),},},wantErr: "boom",wantBufs: 1,},{name: "data,data+error",reads: []readStep{{n: minReadSize,},{n: minReadSize,err: errors.New("boom"),},},wantErr: "boom",wantBufs: 1,},{name: "data,data+EOF - whole buf",reads: []readStep{{n: minReadSize,},{n: readAllBufSize - minReadSize,err: io.EOF,},},wantBufs: 1,},{name: "data,data,EOF - whole buf",reads: []readStep{{n: minReadSize,},{n: readAllBufSize - minReadSize,},{err: io.EOF,},},wantBufs: 1,},{name: "data,data,EOF - 2 bufs",reads: []readStep{{n: readAllBufSize,},{n: minReadSize,},{n: readAllBufSize - minReadSize,},{n: minReadSize,},{err: io.EOF,},},wantBufs: 3,},}for _, tc := range testcases {t.Run(tc.name, func(t *testing.T) {pool := &testPool{allocated: make(map[*[]byte]struct{}),}r := &stepReader{reads: tc.reads,}data, err := mem.ReadAll(r, pool)if tc.wantErr != "" {if err == nil || err.Error() != tc.wantErr {t.Fatalf("ReadAll() returned err %v, wanted %q", err, tc.wantErr)}} else {if err != nil {t.Fatal(err)}}gotData := data.Materialize()if !bytes.Equal(r.read, gotData) {t.Fatalf("ReadAll() returned data %q, wanted %q", gotData, r.read)}if len(data) != tc.wantBufs {t.Fatalf("ReadAll() returned %d bufs, wanted %d bufs", len(data), tc.wantBufs)}// all but last should be full buffersfor i := 0; i < len(data)-1; i++ {if data[i].Len() != readAllBufSize {t.Fatalf("ReadAll() returned data length %d, wanted %d", data[i].Len(), readAllBufSize)}}data.Free()if len(pool.allocated) > 0 {t.Fatalf("got %d allocated buffers, wanted none", len(pool.allocated))}})}}func (s) TestBufferSlice_ReadAll_WriteTo(t *testing.T) {testcases := []struct {name stringsize int}{{name: "small",size: minReadSize,},{name: "exact size",size: readAllBufSize,},{name: "big",size: readAllBufSize * 3,},}for _, tc := range testcases {t.Run(tc.name, func(t *testing.T) {pool := &testPool{allocated: make(map[*[]byte]struct{}),}buf := make([]byte, tc.size)_, err := rand.Read(buf)if err != nil {t.Fatal(err)}r := bytes.NewBuffer(buf)data, err := mem.ReadAll(r, pool)if err != nil {t.Fatal(err)}gotData := data.Materialize()if !bytes.Equal(buf, gotData) {t.Fatalf("ReadAll() = %q, wanted %q", gotData, buf)}data.Free()if len(pool.allocated) > 0 {t.Fatalf("wanted no allocated buffers, got %d", len(pool.allocated))}})}}func main() {var bs mem.BufferSlicepool := mem.DefaultBufferPool()writer := mem.NewWriter(&bs, pool)for _, data := range [][]byte{[]byte("abcd"),[]byte("abcd"),[]byte("abcd"),} {n, err := writer.Write(data)fmt.Printf("Wrote %d bytes, err: %v\n", n, err)}fmt.Println(string(bs.Materialize()))}var (_ io.Reader = (*stepReader)(nil)_ mem.BufferPool = (*testPool)(nil))// readStep describes what a single stepReader.Read should do - how much data// to return and what error to return.type readStep struct {n interr error}// stepReader implements io.Reader that reads specified amount of data and/or// returns the specified error in specified steps.// The read data is accumulated in the read field.type stepReader struct {reads []readStepread []byte}func (s *stepReader) Read(buf []byte) (int, error) {if len(s.reads) == 0 {panic("unexpected Read() call")}read := s.reads[0]s.reads = s.reads[1:]_, err := rand.Read(buf[:read.n])if err != nil {panic(err)}s.read = append(s.read, buf[:read.n]...)return read.n, read.err}// testPool is an implementation of BufferPool that allows to ensure that:// - there are matching Put calls for all Get calls.// - there are no unexpected Put calls.type testPool struct {allocated map[*[]byte]struct{}}func (t *testPool) Get(length int) *[]byte {buf := make([]byte, length)t.allocated[&buf] = struct{}{}return &buf}func (t *testPool) Put(buf *[]byte) {if _, ok := t.allocated[buf]; !ok {panic("unexpected put")}delete(t.allocated, buf)}func (s) TestBufferSlice_Iteration(t *testing.T) {tests := []struct {name stringbuffers [][]byteoperations func(t *testing.T, c *mem.Reader)}{{name: "empty",operations: func(t *testing.T, r *mem.Reader) {if r.Remaining() != 0 {t.Fatalf("Remaining() = %v, want 0", r.Remaining())}_, err := r.Peek(1, nil)if err == nil {t.Fatalf("Peek(1) returned error <nil>, want non-nil")}discarded, err := r.Discard(1)if got, want := discarded, 0; got != want {t.Fatalf("Discard(1) = %d, want %d", got, want)}if err == nil {t.Fatalf("Discard(1) returned error <nil>, want non-nil")}if r.Remaining() != 0 {t.Fatalf("Remaining() after Discard = %v, want 0", r.Remaining())}},},{name: "single_buffer",buffers: [][]byte{[]byte("0123456789")},operations: func(t *testing.T, r *mem.Reader) {if r.Remaining() != 10 {t.Fatalf("Remaining() = %v, want 10", r.Remaining())}res := make([][]byte, 0, 10)res, err := r.Peek(5, res)if err != nil {t.Fatalf("Peek(5) return error %v, want <nil>", err)}if len(res) != 1 || !bytes.Equal(res[0], []byte("01234")) {t.Fatalf("Peek(5) = %v, want [[01234]]", res)}if cap(res) != 10 {t.Fatalf("Peek(5) did not use the provided slice.")}discarded, err := r.Discard(5)if got, want := discarded, 5; got != want {t.Fatalf("Discard(5) = %d, want %d", got, want)}if err != nil {t.Fatalf("Discard(5) return error %v, want <nil>", err)}if r.Remaining() != 5 {t.Fatalf("Remaining() after Discard(5) = %v, want 5", r.Remaining())}res, err = r.Peek(5, res[:0])if err != nil {t.Fatalf("Peek(5) return error %v, want <nil>", err)}if len(res) != 1 || !bytes.Equal(res[0], []byte("56789")) {t.Fatalf("Peek(5) after Discard(5) = %v, want [[56789]]", res)}discarded, err = r.Discard(100)if got, want := discarded, 5; got != want {t.Fatalf("Discard(100) = %d, want %d", got, want)}if err == nil {t.Fatalf("Discard(100) returned error <nil>, want non-nil")}if r.Remaining() != 0 {t.Fatalf("Remaining() after Discard(100) = %v, want 0", r.Remaining())}},},{name: "multiple_buffers",buffers: [][]byte{[]byte("012"), []byte("345"), []byte("6789")},operations: func(t *testing.T, r *mem.Reader) {if r.Remaining() != 10 {t.Fatalf("Remaining() = %v, want 10", r.Remaining())}res, err := r.Peek(5, nil)if err != nil {t.Fatalf("Peek(5) return error %v, want <nil>", err)}if len(res) != 2 || !bytes.Equal(res[0], []byte("012")) || !bytes.Equal(res[1], []byte("34")) {t.Fatalf("Peek(5) = %v, want [[012] [34]]", res)}discarded, err := r.Discard(5)if got, want := discarded, 5; got != want {t.Fatalf("Discard(5) = %d, want %d", got, want)}if err != nil {t.Fatalf("Discard(5) return error %v, want <nil>", err)}if r.Remaining() != 5 {t.Fatalf("Remaining() after Discard(5) = %v, want 5", r.Remaining())}res, err = r.Peek(5, res[:0])if err != nil {t.Fatalf("Peek(5) return error %v, want <nil>", err)}if len(res) != 2 || !bytes.Equal(res[0], []byte("5")) || !bytes.Equal(res[1], []byte("6789")) {t.Fatalf("Peek(5) after advance = %v, want [[5] [6789]]", res)}},},{name: "close",buffers: [][]byte{[]byte("0123456789")},operations: func(t *testing.T, r *mem.Reader) {r.Close()if r.Remaining() != 0 {t.Fatalf("Remaining() after Close = %v, want 0", r.Remaining())}},},{name: "reset",buffers: [][]byte{[]byte("0123")},operations: func(t *testing.T, r *mem.Reader) {newSlice := mem.BufferSlice{mem.SliceBuffer([]byte("56789"))}r.Reset(newSlice)if r.Remaining() != 5 {t.Fatalf("Remaining() after Reset = %v, want 5", r.Remaining())}res, err := r.Peek(5, nil)if err != nil {t.Fatalf("Peek(5) return error %v, want <nil>", err)}if len(res) != 1 || !bytes.Equal(res[0], []byte("56789")) {t.Fatalf("Peek(5) after Reset = %v, want [[56789]]", res)}},},{name: "zero_ops",buffers: [][]byte{[]byte("01234")},operations: func(t *testing.T, c *mem.Reader) {if c.Remaining() != 5 {t.Fatalf("Remaining() = %v, want 5", c.Remaining())}res, err := c.Peek(0, nil)if err != nil {t.Fatalf("Peek(0) return error %v, want <nil>", err)}if len(res) != 0 {t.Fatalf("Peek(0) got slices: %v, want empty", res)}discarded, err := c.Discard(0)if err != nil {t.Fatalf("Discard(0) return error %v, want <nil>", err)}if got, want := discarded, 0; got != want {t.Fatalf("Discard(0) = %d, want %d", got, want)}if c.Remaining() != 5 {t.Fatalf("Remaining() after Discard(0) = %v, want 5", c.Remaining())}res, err = c.Peek(2, res[:0])if err != nil {t.Fatalf("Peek(2) return error %v, want <nil>", err)}if len(res) != 1 || !bytes.Equal(res[0], []byte("01")) {t.Fatalf("Peek(2) after zero ops = %v, want [[01]]", res)}},},}for _, tt := range tests {t.Run(tt.name, func(t *testing.T) {var slice mem.BufferSlicefor _, b := range tt.buffers {slice = append(slice, mem.SliceBuffer(b))}c := slice.Reader()slice.Free()defer c.Close()tt.operations(t, c)})}}Output:Wrote 4 bytes, err: <nil>Wrote 4 bytes, err: <nil>Wrote 4 bytes, err: <nil>abcdabcdabcd
Types¶
typeBuffer¶
type Buffer interface {// ReadOnlyData returns the underlying byte slice. Note that it is undefined// behavior to modify the contents of this slice in any way.ReadOnlyData() []byte// Ref increases the reference counter for this Buffer.Ref()// Free decrements this Buffer's reference counter and frees the underlying// byte slice if the counter reaches 0 as a result of this call.Free()// Len returns the Buffer's size.Len()int// contains filtered or unexported methods}A Buffer represents a reference counted piece of data (in bytes) that can beacquired by a call to NewBuffer() or Copy(). A reference to a Buffer may bereleased by calling Free(), which invokes the free function given at creationonly after all references are released.
Note that a Buffer is not safe for concurrent access and instead eachgoroutine should use its own reference to the data, which can be acquired viaa call to Ref().
Attempts to access the underlying data after releasing the reference to theBuffer will panic.
funcCopy¶
func Copy(data []byte, poolBufferPool)Buffer
Copy creates a new Buffer from the given data, initializing the referencecounter to 1.
It acquires a []byte from the given pool and copies over the backing arrayof the given data. The []byte acquired from the pool is returned to thepool when all references to the returned Buffer are released.
funcNewBuffer¶
func NewBuffer(data *[]byte, poolBufferPool)Buffer
NewBuffer creates a new Buffer from the given data, initializing the referencecounter to 1. The data will then be returned to the given pool when allreferences to the returned Buffer are released. As a special case to avoidadditional allocations, if the given buffer pool is nil, the returned bufferwill be a "no-op" Buffer where invoking Buffer.Free() does nothing and theunderlying data is never freed.
Note that the backing array of the given data is not copied.
funcReadUnsafe¶
ReadUnsafe reads bytes from the given Buffer into the provided slice.It does not perform safety checks.
funcSplitUnsafe¶
SplitUnsafe modifies the receiver to point to the first n bytes while itreturns a new reference to the remaining bytes. The returned Bufferfunctions just like a normal reference acquired using Ref().
typeBufferPool¶
type BufferPool interface {// Get returns a buffer with specified length from the pool.Get(lengthint) *[]byte// Put returns a buffer to the pool.//// The provided pointer must hold a prefix of the buffer obtained via// BufferPool.Get to ensure the buffer's entire capacity can be re-used.Put(*[]byte)}BufferPool is a pool of buffers that can be shared and reused, resulting indecreased memory allocation.
funcDefaultBufferPool¶
func DefaultBufferPool()BufferPool
DefaultBufferPool returns the current default buffer pool. It is a BufferPoolcreated with NewBufferPool that uses a set of default sizes optimized forexpected workflows.
funcNewTieredBufferPool¶
func NewTieredBufferPool(poolSizes ...int)BufferPool
NewTieredBufferPool returns a BufferPool implementation that uses multipleunderlying pools of the given pool sizes.
typeBufferSlice¶
type BufferSlice []Buffer
BufferSlice offers a means to represent data that spans one or more Bufferinstances. A BufferSlice is meant to be immutable after creation, and methodslike Ref create and return copies of the slice. This is why all methods havevalue receivers rather than pointer receivers.
Note that any of the methods that read the underlying buffers such as Ref,Len or CopyTo etc., will panic if any underlying buffers have already beenfreed. It is recommended to not directly interact with any of the underlyingbuffers directly, rather such interactions should be mediated through thevarious methods on this type.
By convention, any APIs that return (mem.BufferSlice, error) should reducethe burden on the caller by never returning a mem.BufferSlice that needs tobe freed if the error is non-nil, unless explicitly stated.
funcReadAll¶added inv1.69.0
func ReadAll(rio.Reader, poolBufferPool) (BufferSlice,error)
ReadAll reads from r until an error or EOF and returns the data it read.A successful call returns err == nil, not err == EOF. Because ReadAll isdefined to read from src until EOF, it does not treat an EOF from Readas an error to be reported.
Important: A failed call returns a non-nil error and may also returnpartially read buffers. It is the responsibility of the caller to free theBufferSlice returned, or its memory will not be reused.
func (BufferSlice)CopyTo¶
func (sBufferSlice) CopyTo(dst []byte)int
CopyTo copies each of the underlying Buffer's data into the given buffer,returning the number of bytes copied. Has the same semantics as the copybuiltin in that it will copy as many bytes as it can, stopping when either dstis full or s runs out of data, returning the minimum of s.Len() and len(dst).
func (BufferSlice)Free¶
func (sBufferSlice) Free()
Free invokes Buffer.Free() on each Buffer in the slice.
func (BufferSlice)Len¶
func (sBufferSlice) Len()int
Len returns the sum of the length of all the Buffers in this slice.
Warning¶
Invoking the built-in len on a BufferSlice will return the number of buffersin the slice, and *not* the value returned by this function.
func (BufferSlice)Materialize¶
func (sBufferSlice) Materialize() []byte
Materialize concatenates all the underlying Buffer's data into a singlecontiguous buffer using CopyTo.
func (BufferSlice)MaterializeToBuffer¶
func (sBufferSlice) MaterializeToBuffer(poolBufferPool)Buffer
MaterializeToBuffer functions like Materialize except that it writes the datato a single Buffer pulled from the given BufferPool.
As a special case, if the input BufferSlice only actually has one Buffer, thisfunction simply increases the refcount before returning said Buffer. Freeing thisbuffer won't release it until the BufferSlice is itself released.
func (BufferSlice)Reader¶
func (sBufferSlice) Reader() *Reader
Reader returns a new Reader for the input slice after taking references toeach underlying buffer.
typeNopBufferPool¶
type NopBufferPool struct{}NopBufferPool is a buffer pool that returns new buffers without pooling.
func (NopBufferPool)Get¶
func (NopBufferPool) Get(lengthint) *[]byte
Get returns a buffer with specified length from the pool.
typeReader¶
type Reader struct {// contains filtered or unexported fields}Reader exposes a BufferSlice's data as an io.Reader, allowing it to interfacewith other systems.
Buffers will be freed as they are read.
A Reader can be constructed from a BufferSlice; alternatively the zero valueof a Reader may be used after calling Reset on it.
func (*Reader)Close¶
Close frees the underlying BufferSlice and never returns an error. Subsequentcalls to Read will return (0, io.EOF).
func (*Reader)Discard¶added inv1.77.0
Discard skips the next n bytes, returning the number of bytes discarded.
It frees buffers as they are fully consumed.
If Discard skips fewer than n bytes, it also returns an error.
func (*Reader)Peek¶added inv1.77.0
Peek returns the next n bytes without advancing the reader.
Peek appends results to the provided res slice and returns the updated slice.This pattern allows re-using the storage of res if it has sufficientcapacity.
The returned subslices are views into the underlying buffers and are onlyvalid until the reader is advanced past the corresponding buffer.
If Peek returns fewer than n bytes, it also returns an error.
func (*Reader)Reset¶added inv1.74.0
func (r *Reader) Reset(sBufferSlice)
Reset frees the currently held buffer slice and starts reading from theprovided slice. This allows reusing the reader object.
typeSliceBuffer¶
type SliceBuffer []byte
SliceBuffer is a Buffer implementation that wraps a byte slice. It providesmethods for reading, splitting, and managing the byte slice.
func (SliceBuffer)ReadOnlyData¶
func (sSliceBuffer) ReadOnlyData() []byte
ReadOnlyData returns the byte slice.
Source Files