optimization:`ReverseChunks::next`,`LinesChunkBuffer::fill` and`BytesChunkBuffer::fill` in`tail`#9667

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open

fereidani wants to merge3 commits intouutils:main

base:main

Choose a base branch

Open

fereidani wants to merge3 commits intouutils:mainfromfereidani:main

Conversation

Copy link

Hi,
First of all, I'm really happy that this project came together!
Here are a few improvements I identified and implemented:

ForReverseChunks::next, the original implementation allocated a new buffer on every call tonext(), which was quite expensive. I changed it to use a preallocatedVec<u8> that is zero-filled on each iteration whenself.block_idx > 0.
a same optimization was applied toBytesChunkBuffer::fill andLinesChunkBuffer::fill. In the previous version, the linechunk = self.chunks.pop_front().unwrap(); dropped a perfectly reusableBox on every loop iteration. I removed the unnecessarychunk.clone(), transferred ownership of the existing value back into the chunks container, and switched to usingOption<Box<Chunk>>. This allows us to reuse the allocation when the first chunk is removed, eliminates the need for new allocations in most cases.

Let me know if you'd like to discuss any of these in more detail!

fereidani added3 commits

OptimizeReverseChunks::next by reusing a buffer

d80a542

reduces heap allocations by reusing memory inLinesChunkBuffer::fill

dccaece

reduces heap allocations by reusing memory inBytesChunkBuffer::fill

e860f66

Copy link

GNU testsuite comparison:

Skipping an intermittent issue tests/tail/overlay-headers (passes in this run but fails in the 'main' branch)

Copy link

Contributor