@kitbellew

Could you let me know when this this draft PR stabilizes and gets re-based? Thanks.

No rush, I have at least two reported SN defects which will a significant amount of time.

Sorting outos-lib andkyo issues has the benefit of exerciser recent SN 0.5.10-SNAPSHOT changes.

I gave this PR an initial read or three. I'd like to be able to study a 'near final' version and not have
that version go from Draft to committed before discussion.

Knowing that such a pause will exist will allow me to focus on exploring some reports that would show that
work accomplished to date is solid or to flush out remaining defects.

@LeeTibbert as requested, this one is ready to review. cc@WojciechMazur

Please review just the last commit, as the PR contains#4622 and#4623 and is to be merged after them.

@kitbellew

Thank you for the heads up and pause.

I am beginning to review this PR for concept. I'll focus on theWatcher changes.
I am running/have_run out of time for this session. Will pick up when I am fresh,
probably tomorrow morning UTC-4.

Delay may work in my/our favor. The required prerequsite PR may have been
merged by then, clearing away some shrubbery and making it easier to focus
on the intended changes.

I suspect that this is a major architectural change and deserves close review
and probably theoretical analysis.

No disrespect.

I have spent at least two long, rested sessions examining this PR and have reached
as far as I can with static analysis. To go further, I would have to build it and insert
some probes: Who is actually doing what to whom?

I believe that I am looking a the most current code. Hard to tell for sure.

Let me speak plainly with the intent of helping the dialectic, not offending, but pick my
words carefully.

At the line-of-code level I saw an attention to detail that made it believably correct.
I also saw that areas of complexity in earlier drafts, possibly in other PRs had been
addressed. Very well crafted.

I believe that is is necessary to rise above the line-by-line and ask two questions
of grand tactics.

• Is this code necessary to facilitate work on further PRs, say ones to address
  remaining disadvantages inWatcher code?

  If yes and asked to put my marker on red or black, I would demure and suggest
  an alternate reviewer.

  One concrete observation: I believe I understand the drive to centralize os process waiting & reaping.
  I'll report that out of respect for the author, I spent a significant amount of time in UnixProcessGen2
  trying to figure out what was a Watcher in the traditional sense and what were the methods
  with Watcher related names actually doing in that class. As usual, I hate to confess my limitations
  in front of my colleagues & betters.

  Another reviewer may not have a problem tracing that code.

• Is this code any better than the code on, say November 1?

  I believe it makes more obvious solutions to a correct, lower cost Watcher more difficult.

  I can elaborate on that if someone can suggest the proper place. Condensing &
  foreshadowing a much longer discussion. With a tiny number of pids, who cares how the Watcher is implemented?
  Moving to small to huge, the fewer process the Watcher actually has to watch the better the
  latency of each process watched. One can do make rough estimates or measurements of expected code paths.
  With 100% 'Java9' 'never waitPid` processes, the Watcher becomes equivalent to "watch all processes".
  With 0% an 'easy' implementation may create the Watcher, but has no processes to watch.

  This is Open Source and a developer can always present a solution which rolls back these changes when
  an actual working PR is presented.

  At the very least those alternatives should be discussed and possibly some design studies done.
  I suspect that one such is in progress studying an event driven Watcher for macOS.

I do agree that the pid waiting & reaping code as of Nov 1 would benefit from improvement,
where for improvement is measured by both correctness and time for a competent person to
trace.

renamed as ExitChecker, to remove confusion with GenericProcessWatcher, as the watcher has an active thread while this new checker doesn't do anything until asked, so it provides ability to check for exit.

It is nice to see the sketch of a development plan.

I think this dialogue is moving forward. To try to have some kind of order, let me
split my replies into Present and Future.

Present.

If you say all of this is going to change and soon, perhaps you might think of leaving this
as Draft while you implement those changes. That would leave the code in mainline in a
less complex state in case you get hit by a bus. It would also give time for some complexity
to go away or collapse into a form which can be more easily explained.

Yes, there is an overhead to holding lots of 'pending' files in current Work-In-Progress
files. People following the PR grumble about too many changes. In this case, from
what you describe the 'pending' files are the ones which would be changed in the WIP
in any case. That is, very little shrubbery.

i must be in the minority who thinks that the proposed state is a bit clearer, with a few design patterns deployed to add clear division of labor, as well as remove several if-else and throw platform-dependent statements.

| or instance, it kept insisting that some code in MacOS's implementation of kevent makes NOTE_EXITSTATUS completely
|unreliable, and I had to try it myself before running into problems and agreeing

I've encountered this statement before. It sounds like a macOS 10.n, where n is something like 5, issue.
It has been reported as fixed in 10.n+mumble. macOS is now at 16.n and SN CI now runs at macOS 15.

Agreed we need to be on the look out and probably exercise these.

On a closely related topic.

More and more in my reading aboutkevent,poll(), andppoll() I have been coming across
poorly described issues when closing a, say pidfd in one process which is being used in appoll() in another.
The latter may not get waken up. I have not tested this out, but it is rising in my "be sensitive to not introducing this bug'
design list.

If you are going where I think you are going, a single os waiter, this may not be a concern.

I think eventfd was specifically designed to wake up all waiters, so the close there may not
be an issue. Agreed, executing eventfd on macos might be difficult.

I mention the the 'pidfd' possible behavior because I had described allocating one and
using in both theProcess and theWatcher.

Somewhere down my interrupt stack and getting further down by the day, another
contributor and I are dealing with 'asynchronous close' not waking up processes
which are waiting in 'accept()' or 'connect()' when the relevant fd is closed in
a second process. A hack is in place to buy time. The situations sound very
familiar. The long term solution for Linux is probably setting eventfd on close.
A macOs solution is harder, probably similar to 'self-pipe trick' but avoiding signals.

os-lib also has a problem with hanging processes after doingdestroy()
in one process and then awaitFor(). Yes 0.5.9 problem, not this PR
but a faint suggestion that we may want to be careful withasynchronous close.

Future

Code simplification

Unifying Gen1 and Gen2 to the greatest extent sensible is certainly a good goal.

• I have been increasingly having the intrusive thought that perhaps all of this (good) collapse
  open the way to a 'clean sheet of paper' design. At least conceptually, the series of
  Evolutions may get us somewhat similar but may also trap us in a box canyon.

Process waiting architecture.

possibly better than ppolling/keventing one process at a time

I suggest this is the crux. I also think that the JVM specification does us no
favors here.

Let me describe two contrasting models and apply some math. Yes, I am skipping over
a discussion that should be here about 'on what dimensions/factors are we trying to optimize'.
And yes, this is more than a bit geeky.
I think that prior SN devo experiments in this area show that we need some form of a child process exit
Watcher to implement the JDK 9 'onExit' requirements.

For analysis lets identify two models or design architectures for how process reaping happens
between Watcher and Process threads. I use process reaping because the os process id slot
is the underlying shared resource and driver of events.

One model can be described as "Watcher does most of the work", say "Load the Watcher".
It does all waiting for notification of the exit of any of the Process it is watching. If it knew that the Process whose
child is being watched was waiting in a javalib 'waitFor()' it could set a condition variable or
use, on Linux, eventfd. Unfortunately, in the simple case, it can not know this, so it must
reap the now exited child process itself.

Another model can be described as "Processes do the majority of the waiting and reaping
while the Watcher handles 'no pending or eventual waitFor' cases". Say "Load the Processes".

I'll assume an event driven Watcher. The removal sequential scan of Processes being watched is
under active investigation in order to avoid a linear scan of potentially large lists and also sleep()s.

I think this is the key area needing both design studies an analysis.
In the simplest case, that leaves us with one Watcher thread which lives until that Runtime exits, one
Process thread, and one child thread. Now a watched child exits.

If there the parent is not in a waitFor, the Watcher reaps the process.

If the process is in some for of waitFor, there is a race to reap the child and
one must be careful about how one closes a, say, pidfd which might be being
used by the other.

I think advantage in reduced code complexity goes to the "Watcher does most of work" model
here, although the runtime costs are slightly lower when the Process wins the race. There
is no trip through the thread scheduler.

I will also note that probably 95+ of current and near term uses will be of
this simple case. Most cases in SN Process test prior to Fall 2025
fall into this description.

One Watcher, One Process, two waitFor

Now expand into the case where there is one Watcher, one Process, its child, and
two threads each calling waitFor on that process.

Now there are two chances that the slightly more runtime efficient process reaping will happen.

And one chance that the Watcher will have to spend time reaping the child. There are
no other Processes being monitored, so no one is delayed.

One Watcher, Two Process each with a waitFor.

For the sake of analysis, consider the case where the child process for each of the Processes
exits at the 'same' kernel time as seen by ppoll() etc.

Now there are two chances that the slightly more runtime efficient process reaping will happen.

And one chance that the Watcher will have to spend time reaping the children. Time spent
reaping the first, increases the latency in reporting the exit of the second child.
Granted that we are talking milliseconds here, the line of reasoning will progress.

If each Process itself has several threads in waitFor, the chances for a low latency
detection in the Process itself goes up.

Theoretical advantage to 'Load the processes" model.

One Watcher, Large N Process, each with possibly more than pending 'waitFor'

Consider an extreme or upper bound Pandemonium case. Will SN ever actually encounter this?
Probably but it is relatively easy to avoid it by initial design choice.

Now there are large N+ chances that the slightly more runtime efficient process reaping will happen.

And one chance that the Watcher will have to spend time reaping the children. If the Watcher wins
the race, it has to spend time reaping each child. For a quick analysis, one can do a guestimate
of the number of events arriving in a 'ppoll()' interval. It will probably be a Poisson distribution
weighted towards one but with a long right tail. What is the cumulative area under that tail, for some
cutoff point? It is reasonable, to me, to believe that with large enough N, some number, 5? 10?
processes will be delayed for more that the time it would take a standalone process to reap().

By far and away, advantage goes to the "load the Processes model"

Alternatives

A trial design study implementation may be able to remove the "Watcher reaps" and attendant
latency at large N by having the Watcher not reap but notify the relevant Process that a reaping
is needed. That means that something in the Process needs to be waiting for that notification
independent of any caller, or not, of a 'waitFor' on that instance. This probably means another
thread or Future and expanded resource use.

Measuring theory

Theory allows one to make informed guesses, but how do we measure the slow points of
whatever model we implement. Can we get numbers to help choose between rough implementations
of the models.

Summation

In the "single Watcher, single Process, single waitFor() or not" case doing
the "Load the Watcher" model with an os wait in one placemay simplify the code at the
cost of a guaranteed trip through the thread scheduler.

Amortized over current and foreseeable work loads, the "Load the Processes" is highly
likely to result in lower latency in responding to child exit events (greater responsiveness).
The cost is waiting in two code places and, in the limit, many runtime places.

The test of skill in either model, and probably in any related model, is if the irreducible
complexity of the subject domain can be tamed so that a competent devo can trace and
maintain it.

Granted that having a black belt is the price of entry, does one need a first degree black
belt or a tenth degree?

| must be in the minority

As far as I can tell, there are only two voices so far, so equality, not minority.
My intent is honest discussion, possibly leading to iterative improvement.

I do not have either the desire or the standing to be Horatius Cocles, who
barred the bridge.

I do not doubt for a moment l that there is point goodness here which should be captured here and would
certainly like to discuss them over beverages someday.

It is off-top from this PR but does pertain to the 'future work' sketch presented earlier to
this discussion.

I propose to do a private design study using the current (Nov 5) mainline code and
adapting it so that UnixProcessGen2 registers with the current "watch every Process"
Watcher only whenProcessHandle.onExit() has been called.

If that study fails CI or does not show a reduction of number of Processes
being watched, then I will know.

If shows what I think it will, I could submit a Draft PR for discussion.
My intent is that suchwould_not become a candidate for merge.

I want to stay out of your way whilst you are on such a productive roll.

Reducing the load on a Watcher using a sequential scan is just shy of essential.
It may buy us enough time to work on and qualify a proper event driven Watcher.

I believe that reducing the load on any eventual event driven Watcher will
also pay off; to a lesser but still worthwhile degree.

If the proposed change does work, it would be relatively easy to implement, I think
even in the "Load the Watcher' model.

I think the above may offer the greatest benefit for time to market.

An alternative would be for me to do an independent design study using current mainline
code of implementing first either of linux or macos, then the other one. The idea would be
to see what two independent developers experienced, especially in terms of os specific
features and quirks. Not to spend scarce developer time on duplicate effort all the way to merge PR, but to get
two readings of the land.

My intent would be that such a study never itself become a candidate for merge. There is too much
change likely to happen over the next few weeks. I also, as said before, want to stay out of your way
whilst you are on such a productive roll. Anything I discover or avoid through design might save you some thrashing.

Of course, I will be reading along as you post your efforts.