Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Reading variable sized data (a small entry followed by a large entry) leads to IndexOutOfBoundsException #183

Closed
@buko

Description

@buko

This is a bit of a doozy.

We're in the process of moving an application to lmdbjava. We observe that when running the tests the following occurs:

  1. If the tests are all run in the same JVM process then we consistently get very strange errors related to ByteBuf validation constraints:

For example:

Caused by: java.lang.IndexOutOfBoundsException: readerIndex: 1297, writerIndex: 1095 (expected: 0 <= readerIndex <= writerIndex <= capacity(1095))        at io.netty.buffer.AbstractByteBuf.checkIndexBounds(AbstractByteBuf.java:112)        at io.netty.buffer.AbstractByteBuf.writerIndex(AbstractByteBuf.java:135)        at org.lmdbjava.ByteBufProxy.out(ByteBufProxy.java:166)        at org.lmdbjava.ByteBufProxy.out(ByteBufProxy.java:41)        at org.lmdbjava.KeyVal.valOut(KeyVal.java:134)        at org.lmdbjava.Cursor.seek(Cursor.java:377)        at org.lmdbjava.Cursor.first(Cursor.java:126)        at org.lmdbjava.CursorIterable.executeCursorOp(CursorIterable.java:125)        at org.lmdbjava.CursorIterable.update(CursorIterable.java:172)        at org.lmdbjava.CursorIterable.access$100(CursorIterable.java:52)        at org.lmdbjava.CursorIterable$1.hasNext(CursorIterable.java:99)

This occurs using the very standard code to iterate over all the keys in a table:

try (finalCursorIterable<ByteBuf>cursor =table.iterate(txn)) {for (finalKeyVal<ByteBuf>kv :cursor) {finalByteBufkeyBuffer =kv.key();finalByteBufvalueBuffer =kv.val();
  1. If the maven build is run using -DforkCount=1C -DreuseForks=false then all the tests run perfectly fine.These system properties tell maven to run each test class in its own JVM and it eliminates all errors in the build at the expense of a much slower build.

Additionally if we run a single test using the -Dtest=$TestName syntax the test passes fine.

Clearly there is some erorr related to static state that is leading to inter-test dependencies. If we eliminate static state by running each Test in its own JVM all the tests pass. If we run the tests run the tests in the same JVM fails.

I've tried to dig into this and get some idea of what's happening and I doi believe the problem is related to the ByteBufProxy class. There are some red flags in this class:

  1. This class maintains significant static state in a threadlocal BUFFERS cache. There seems to be no way to reset or clear this BUFFERS. As far I can tell the cache persists even if you call Env.close() and it might explain why we're getting ByteBufs whose readIndex is larger than their writeIndex.

  2. The ByteBufProxy class is also hardcoded to use PooledByteBufAllocator.DEFAULT. We've had problems in the past with multiple components (netty, netty extensions) all mucking around with PooledByteBufAllocator.DEFAULT. We tend toavoid using PooledByteBufAllocator.DEFAULT and instead having each component use its own explicit PooledByteBufAllocator. Unfortunately there's no way to configure ByteBufProxy to use a different ByteBufAllocator.

  3. The ByteBufProxy class makes several questionable assumptions. ByteBufs are allocated and not released, and no attempt is made to clear() ByteBufs returned from the pool.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions


      [8]ページ先頭

      ©2009-2025 Movatter.jp