Fixes

• Piped output with simultaneous logging to files fixed
• Logging channels finalization unified for distinct types of output (stdout/err, devnull, files, pipes)
• Pipe output encoding fixed for Python3
• Exceptions handled on Job completion

Features

• Piped output from job stdout and stderr refined considering multiprocess environment
• Optional automatic logging of the piped output to the specified files added
• Job interface extended with the memory constraints
• Job interface extended with theonfinish() callback

Fixes

• Some asserts fixed
• Typos fixed

Features

• Load balancing refined to discourage killing of long-running jobs on low memory condition if not significantly smaller but significantly shorter jobs are executed (and can be postponed)
• Redundant rescheduling of the postponed jobs eliminated
• WebUi a bit densified (default font updated)
• Memory consumption estimation for the executing jobs refined

Bug Fixes

• Available memory estimation on rescheduling fixed
• Table lines selection on mouse hovering fixed

Features

• JSON serialization implemented
• Fully self-contained pip distribution (without the need to copy./views any more):https://pypi.org/project/pyexpool/
• Query formatlimit parameter applied to both tasks and jobs
• Query filters with optional values

Fixes

• [Relative] import under Python3
• Listing of the hierarchical tasks (repetitive separate listing of subtasks omitted)
• Processing of hierarchical subtasks:
  • tasks finalization
  • numterm evaluation
• Zombie apps cleaned even without the user signal
• Subtasks counting fixed

Breaking Changes

Query formatjlim parameter refactored tolimit and its functionality extended

Features

• DHMS format for the duration queries:20d3m1.25 means 20 days 3 minute 1.25 sec
• Default values for the range bounds added+/-inf:1.. is the same as1..inf

Fixes

• Duration displayed for the running tasks
• Tasks processing with sub-sub+ jobs
• Queries parsing for various cases
• WebUI errors reporting refined

Fixes

• Fixed listing of the jobs in the hierarchical tasks
• Worker processes tasks added to the listing in the Tasks tab
• Super-tasks of the tasks are included into the listing
• Rest API / queries manual page rendering on github is fixed
• Description of the WebUI deployment and usage is drafted

Features

• WebUI is implemented for the interactive tracing and profiling of Jobs and Tasks
• REST API is implemented:
  • Customization of the displaying properties of the items (Jobs/Tasks)
  • Items filtering by the values of their properties with support of the range queries and optional values
  • Dedicated endpoints for the failed items and scheduled jobs and tasks
• ExecPool,Task andJob APIs extended
• onfinishcalllbacks are added, which are always executed unlike theondone callbacks

And lots of less significant changes

Fixes

• Python 3.7 compatibility (async argument in theExecPool method renamed toconcur)
• ExecPool termination refined to be faster
• Locks handling refined
• Linting and syntax related fixed

Breaking Changes

• Job argumentontimeout renamed torsrtonto
• timeout placeholder (deleted in v2.2) is recovered n theExecPool since it is going to be implemented in future
• Extended APIs affected the order of the positional parameters

Features

• Hierarchical subtasks, so as each task may include another tasks or jobs (cycles are not supported and the corresponding validation is not performed)
• onfinish callback added to the Task to be called on both graceful completion and termination of all scheduled subtaks (tasks & jobs)
• Task related tests added

Fixes

• Task callback calls fixed
• Linting related fixes
• Exceptions catching fixed
• Typos and other minor issues

Breaking Changes

Task interface extended andtimeout removed from it since the timeout has not been implemented anyway.
Note:timeout placeholder will be recovered in the following release to not break the interface and since it is going to be implemented in future.

Features & Optimizations

• Multiprocess Affinity Mask for multithreaded workers with possibility to mask multiple cores / hardware threads or reserve them masking only the first logical CPU (physical hardware thread) to maximize CPU cache L1/2/3
• ExecPool initialization interface updated (afnstep replaced with AffinityMask)
• Available RSS/RAM in the OS is considered in memory constrains besides the nominal RAM
• Scheduling refined to reduce the number of rescheduling and execute huge jobs with minimal swapping

Fixes

• Fixed and optimized affinity masking to maximize the dedicate cache (logical cores enumeration fixed)
• Termination refined to not leave spawning non-registered workers on termination

Features and Optimizations

• Jobs queue reordering optimized
• Tracing refined and redundant warnings removed

Fixes

• ontimeout auto restart is respected and does not cause termination of the related jobs
• _LIMIT_WORKERS_RAM and_CHAINED_CONSTRAINTS related scheduling bugs are fixed

Known Bugs

• Single non-registered worker process might remain after termination of the [heavily loaded] execution pool on rescheduling, fixed sincev2.1-MultiprocAfn
• Issues in the logical CPUs enumeration prevent cache maximization, fixed sincev2.1-MultiprocAfn