Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Duplicates conditions added to a status list#3026

teabot started this conversation inGeneral
Discussion options

JOSDK v4.9.7

I have a resource whose status includes a list of conditions. This is serialised as an array. In my reconcile method I update the conditions list like so:

shouldPatchStatus = falsefor each condition type {  var previous = get previous condition from list  var desired = compute new condition state  if (previous != desired) {    if (previous == null) {      add desired to list    } else if (desired == null) {      remove previous from list    } else {      apply state of desired to previous (update in place)    }    shouldPatchStatus = true  } }return shouldPatchStatus ? UpdateControl.patchStatus : UpdateControl.noUpdate

In practice I find that I get duplicate entries appearing in the conditions list, usually differing only bylastTransitionTime with a delta of a few ms.

When I look at logs, I see the two instances of the condition apparently added by two reconciliation cycles, ms apart. One follows the other (no overlap), but they can take place on different threads. They both seem to operate on the same resource version. The logs look something like:

1  Reconcile resource 422. Add condition3. PatchStatus4. Reconcile resource 425. Add condition6. PatchStatus

As I undertand it:

  1. JOSDK does not perform multiple concurrent reconciles on the same resource, so if this is true, I do not see how there could be concurrent updates to the same list. The logs appear sequential also - suggesting that the reconciliations are not concurrent.
  2. With SSA, array updates are atomic, so in the event of a race, I might either see a conflict, or "last writer wins", but inexplicably I see two elements for a given type persisted in the status.

Can you suggest why this may be happening and how I might guard against it?

You must be logged in to vote

Replies: 1 comment 9 replies

Comment options

Hi@teabot , yeah, it's probably not a concurrency issue, I don't recall any tbh. I would suggest to log the requests and responses, and see if something buils up from that, there are somethimes funky issues with SSA.

You must be logged in to vote
9 replies
@teabot
Comment options

I think for the primary, SSA is not used. The update of the CR after reconcile seems to follow this path:

i.j.o.p.e.ReconciliationDispatcher#handleExecution  i.j.o.p.e.ReconciliationDispatcher#handleDispatch    i.j.o.p.e.ReconciliationDispatcher#handleReconcile      i.j.o.p.e.ReconciliationDispatcher#reconcileExecution        i.j.o.p.e.ReconciliationDispatcher#updateStatusGenerationAware         i.j.o.p.e.ReconciliationDispatcher.CustomResourceFacade#patchStatus           i.f.k.c.d.i.HasMetadataOperation#editStatus             i.f.k.c.d.i.BaseOperation#getItemOrRequireFromServer             i.f.k.c.d.i.HasMetadataOperation#patch(io.fabric8.kubernetes.client.dsl.base.PatchContext, T, T)               i.f.k.c.d.i.BaseOperation#handlePatch                 i.f.k.c.d.i.OperationSupport#handlePatch(io.fabric8.kubernetes.client.dsl.base.PatchContext, T, T, java.lang.Class<T>)                   i.f.k.c.d.i.PatchUtils#jsonDiff                   i.f.k.c.d.i.OperationSupport#handlePatch(io.fabric8.kubernetes.client.dsl.base.PatchContext, T, java.lang.String, java.lang.Class<T>)

I am supposing thatPatchUtils#jsonDiff does not take the same atomic approach to updating lists as SSA? Is it possible to use SSA for primaries in 4.9.7? If not, I guess I could switch to a status update for list adds/removes instead? WDYT?

@csviri
Comment options

Hi could you pls add stack trace with line numbers?

So you are using v4.9.7 ? You don't have to use UpdateControl , you can just do SSA using the client.

@teabot
Comment options

Sure, here is the trace:

handlePatch:397, OperationSupport (io.fabric8.kubernetes.client.dsl.internal)handlePatch:764, BaseOperation (io.fabric8.kubernetes.client.dsl.internal)lambda$patch$2:231, HasMetadataOperation (io.fabric8.kubernetes.client.dsl.internal)apply:-1, HasMetadataOperation$$Lambda/0x0000007002016da0 (io.fabric8.kubernetes.client.dsl.internal)patch:236, HasMetadataOperation (io.fabric8.kubernetes.client.dsl.internal)editStatus:76, HasMetadataOperation (io.fabric8.kubernetes.client.dsl.internal)editStatus:44, HasMetadataOperation (io.fabric8.kubernetes.client.dsl.internal)patchStatus:419, ReconciliationDispatcher$CustomResourceFacade (io.javaoperatorsdk.operator.processing.event)updateStatusGenerationAware:226, ReconciliationDispatcher (io.javaoperatorsdk.operator.processing.event)reconcileExecution:150, ReconciliationDispatcher (io.javaoperatorsdk.operator.processing.event)handleReconcile:117, ReconciliationDispatcher (io.javaoperatorsdk.operator.processing.event)handleDispatch:91, ReconciliationDispatcher (io.javaoperatorsdk.operator.processing.event)handleExecution:64, ReconciliationDispatcher (io.javaoperatorsdk.operator.processing.event)run:452, EventProcessor$ReconcilerExecutor (io.javaoperatorsdk.operator.processing.event)runWorker:1144, ThreadPoolExecutor (java.util.concurrent)run:642, ThreadPoolExecutor$Worker (java.util.concurrent)runWith:1596, Thread (java.lang)run:1583, Thread (java.lang)
@teabot
Comment options

I think my options are one of:

  1. Update to a version of JOSDK that supports SSA on primaries
  2. Perform primary SSA manually with client
  3. UseupdateStatus instead ofpatchStatus when adding/removing list elements
  4. Use map instead of list (breaking schema change)
@csviri
Comment options

yes, in you case option 2. would be trivial.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Category
General
Labels
None yet
2 participants
@teabot@csviri
[8]ページ先頭
©2009-2025 Movatter.jp