Repository files navigation

Theconcurrent-log-handler package provides robust logging handlers for Python's standardlogging package (PEP 282).It enables multiple processes (and threads) to safely write to a single log file, with built-in support for size-basedand time-based log rotation and optional log compression.

This package is meant for applications that run in multiple processes, potentially across different hosts sharing anetwork drive, and require a centralized logging solution without the complexity of external logging services.

NOTE: If you're reading this and the links to files likeCHANGELOG.md don't work,or links within the document either, try viewingthis document on GitHub instead.

SeeCHANGELOG.md for details.

• Version 0.9.28: (June 10th, 2025)

  • Fixes errors when apps, esp. asyncio based, try to log during interpreter shutdown.Issue#80
  • Fix missing rollovers when a worker was restarted before the next logging event,but after the last scheduled rollover.Issue #81

• Version 0.9.27: (June 6th, 2025)

  • Fixes Issue#73Fix timed rotation handler's file cleanup logic.
  • Fixes Issue#79Harden timed handler's rollover mechanism against timestamp errors or other sync corruption.

• Important Notice (June 2025): Background Logging Utility Deprecated

  • Theconcurrent_log_handler.queue module is nowdeprecated (will be removed in v1.0.0).
  • It has compatibility issues with complex logging setups and other robustness concerns.
  • Recommendation:
    • Simply use the standard CLH handlers (ConcurrentRotatingFileHandler orConcurrentTimedRotatingFileHandler) directly in your application.Synchronous logging calls are simpler, more reliable, and performant enough for most use cases.
    • If you need non-blocking logging calls, use the standard library patterns shown inPerformance Patterns.
  • The core CLH handlers remain fully supported and are not affected by this deprecation.

• Version 0.9.26: (May 2025)

  • Improved performance, especially on POSIX systems.
  • Added testing for Python 3.13 and improved project configuration and documentation.

• Concurrent Logging: Multiple processes and threads safely write to the same log file.
• File Rotation: Size-based and time-based rotation with optional compression.
• Cross-Platform: Windows and POSIX support with reliable file locking.
• Customizable: Control naming, permissions, line endings, and lock file placement.
• Performance Optimized: Keeps files open between writes for better performance.
• Python 3.6 through current versions: Modern Python support.
• Focused Design: Reliably handles file operations. For non-blocking behavior, see our recommendedApplication-Level Performance Patterns, including patterns forgraceful degradation to synchronous logging,or using sync logging only forhigher prioritylevels.

CLH is primarily designed for scenarios where:

• Multiple processes of a Python application need to log to a shared file.
• These processes might be on the same machine or on different machines accessing a shared network drive.
• Log files need to be automatically rotated based on size or time.

Note that this package is not primarily intended for intensive high-throughput logging scenarios,but rather for general-purpose logging in multi-process applications.

While CLH offers a robust file-based solution, consider these alternatives for different needs:

• Cloud Logging Services: Azure Monitor, AWS CloudWatch Logs, Google Cloud Logging, Logstash, etc. These areexcellent for distributed systems and offer advanced analysis features.
• Custom Logging Server: Implement a centralized logging server as demonstrated inthePython Logging Cookbook.CLH'sQueueHandler andQueueListener can be adapted for this pattern.

Install the package usingpip:

pip install concurrent-log-handler

This will also installportalocker. On Windows,portalocker has a dependency onpywin32.

To install from source:

python setup.py install

Here's a simple example usingConcurrentRotatingFileHandler for size-based rotation:

importloggingfromconcurrent_log_handlerimportConcurrentRotatingFileHandlerimportoslogger=logging.getLogger(__name__)logger.setLevel(logging.INFO)# Use an absolute path for the log filelogfile=os.path.abspath("mylogfile.log")# Configure the handler: rotate after 512KB, keep 5 backups# Mode "a" for appendrotate_handler=ConcurrentRotatingFileHandler(logfile,"a",maxBytes=512*1024,backupCount=5)logger.addHandler(rotate_handler)logger.info("This is an exciting log message!")logger.info("Multiple processes can write here concurrently.")

For a few more basic code examples, seesrc/example.py.For more advanced usage including non-blocking patterns, seethePerformance Patterns guide.

To ensure correct and reliable operation when using CLH in a multi-process environment, please keep the followingin mind:

1. Handler Instantiation per Process:

   • Each processmust create its own instance of the CLH handler (ConcurrentRotatingFileHandler orConcurrentTimedRotatingFileHandler).
   • Youcannot serialize a handler instance and reuse it in another process.
     • For example, you cannot pass it from a parent to a child process viamultiprocessing in spawn mode.
   • This limitation is because the file lock objects and other internal states within the handler cannot be safelyserialized and shared across process boundaries.
   • This requirementdoes not apply to threads within the same process; threads can share a single CLH instance.
   • This requirement alsomay not apply to child processes created viafork() (e.g., with Gunicorn--preload),where file descriptors might be inherited. However, explicit instantiation in each process is the safest approach.

2. Multiprocessing and Spawn mode:

   Just to reemphasize the point above:

   • If you usemultiprocessing with the defaultspawn start method (the default on Windows and macOS),each child process must create its own CLH handler instance.
   • In your child process startup code, instantiate the handler as shown in the example above.
   • Usually this means you can call your standard logging setup function in the child.
   • Don't just initialize your logging code in the parent process and allow child processes to inherit loggers.

3. Consistent Configuration:

   • All processes writing to thesame log filemust use identical settings for the CLH handler (e.g.,maxBytes,backupCount,use_gzip, rotation interval, etc.).
   • Do not mix CLH handlers with other logging handlers (likeRotatingFileHandler from the standard library) writingto the same file. This can lead to unpredictable behavior and data loss.

4. Networked/Cloud Storage:

   • When logging to files on network shares (NFS, SMB/CIFS) or cloud-synced folders (Dropbox, Google Drive, OneDrive),ensure that the advisory file locking provided byportalocker works correctly in your specific environment.
   • Thelock_file_directory option allows you to place the lock file in a different location (e.g., a local fastfilesystem) than the log file itself. This can resolve issues with locking on certain network shares. However, ifmultiple hosts write to the same shared log, theymust all have access to this common lock file location.
   • Alternatively, configure your cloud sync software to ignore CLH lock files (typically.<logfilename>.lock orfiles in thelock_file_directory).
   • If you run into problems, try thekeep_file_open=False option to close the log file after each write. Thismay help with certain networked filesystems but can impact performance.

5. One Handler Instance per Log File:

   • If your application writes to multiple distinct log files, each log filerequires its own dedicated CLH handler instance within each process.

   • It is possible to have multiple CLH handlers that point to the same log file,for example, if you want to log different log levels or formats to the samefile. However, all other rotation settings must be identical across thesehandlers.

CLH provides two main handler classes:

This handler rotates logs when the file size exceedsmaxBytes. Note that the actual file sizes may exceedmaxBytes. How much larger depends on the size of the log message being written when the rollover occurs.

fromconcurrent_log_handlerimportConcurrentRotatingFileHandler# Example: Rotate at 10MB, keep 3 backups, compress rotated logshandler=ConcurrentRotatingFileHandler("app.log",maxBytes=10*1024*1024,backupCount=3,use_gzip=True)

This handler rotates logs at specified time intervals (e.g., hourly, daily, weekly). It canalso rotate based on sizeifmaxBytes is set to a non-zero value.

By default, it rotates hourly and does not perform size-based rotation (maxBytes=0).

fromconcurrent_log_handlerimportConcurrentTimedRotatingFileHandler# Example: Rotate daily, keep 7 backups, also rotate if file exceeds 20MBhandler=ConcurrentTimedRotatingFileHandler(filename="app_timed.log",when="D",# 'D' for daily, 'H' for hourly, 'M' for minute, 'W0'-'W6' for weeklyinterval=1,backupCount=7,maxBytes=20*1024*1024,# Optional size-based rotationuse_gzip=True)

It's recommended to use keyword arguments when configuring this class due to the number of parameters and theirordering. For more details on time-based rotation options (when,interval,utc), refer to the Python standardlibrary documentation forTimedRotatingFileHandler.

Both handlers share several configuration options (passed as keyword arguments):

• logfile /filename: Path to the log file.
• mode: File open mode (default:'a' for append).
• backupCount: Number of rotated log files to keep.
• encoding: Log file encoding (e.g.,'utf-8').
• delay: Defer file opening until the first log message is emitted (boolean).
• use_gzip: (Default:False) IfTrue, compresses rotated log files using gzip.
• owner: Tuple(uid, gid) or['username', 'groupname'] to set file ownership on Unix.
• chmod: Integer file mode (e.g.,0o640) to set permissions on Unix.
• lock_file_directory: Path to a directory where lock files should be stored, instead of next to the log file.
• newline: (Default: platform-specific) Specify newline characters. E.g.,'' (letterminator handle it).
• terminator: (Default: platform-specific, e.g.,\n on POSIX,\r\n on Windows). Specify record terminator.
  • To force Windows-style CRLF on Unix:kwargs={'newline': '', 'terminator': '\r\n'}
  • To force Unix-style LF on Windows:kwargs={'newline': '', 'terminator': '\n'}
• namer: A callable function to customize the naming of rotated files. SeeBaseRotatingHandler.namer in Python docs.
• keep_file_open: Defaults toTrue for enhanced performance by keeping the log file (and lock file) open betweenwrites. This is recommended for most use cases.
  • Set toFalse if you need to ensure the log file is closed after each write, for example, for compatibility withcertain networked filesystems.
  • On Windows, the log file will always be closed after writes to allow for rotation, but this option still affectswhether the lock file is kept open.

CLH handlers aresynchronous by design for maximum reliability andpredictability. However, some applications need non-blocking logging to preventthread blocking during logging bursts or high-frequency logging scenarios.

With CLH's recent performance improvements (v0.9.26+), many applications won'tneed non-blocking patterns. Test your specific use case first - synchronouslogging is simpler and more reliable.

• Multiple threads logging heavily during error conditions
• Web application request threads that shouldn't block on I/O
• High-frequency logging in performance-critical applications
• Applications with strict latency requirements

For most non-blocking needs, use Python's standard libraryQueueHandler:

importloggingimportlogging.handlersimportqueueimportatexitfromconcurrent_log_handlerimportConcurrentRotatingFileHandler# Create queue and handlerslog_queue=queue.Queue(maxsize=10000)file_handler=ConcurrentRotatingFileHandler("app.log",maxBytes=10*1024*1024)queue_handler=logging.handlers.QueueHandler(log_queue)# Set up background processinglistener=logging.handlers.QueueListener(log_queue,file_handler)listener.start()atexit.register(listener.stop)# Configure loggerlogging.getLogger().addHandler(queue_handler)logging.info("This won't block your thread!")

For comprehensive patterns including:

• Web framework integration (Django, Flask, FastAPI)
• Graceful degradation when queues fill up
• Production monitoring and error handling
• Critical vs background logging strategies
• Performance comparisons and best practices

See the complete guide:Performance Patterns

This approach uses Python's standard library tools, giving you full controlwhile keeping CLH focused on reliable file operations.

You can configure CLH using Python'slogging.config.fileConfig:

[loggers]keys = root[handlers]keys = concurrentRotatingFile, concurrentTimedRotatingFile[formatters]keys = simpleFormatter[logger_root]level = INFOhandlers = concurrentRotatingFile, concurrentTimedRotatingFile[handler_concurrentRotatingFile]class = concurrent_log_handler.ConcurrentRotatingFileHandlerlevel = INFOformatter = simpleFormatterargs = ("rotating_size.log","a")kwargs = {'maxBytes': 10485760,'backupCount': 5,'use_gzip': True}# For Python < 3.7, kwargs are not supported. You might need to subclass or use code config.[handler_concurrentTimedRotatingFile]class = concurrent_log_handler.ConcurrentTimedRotatingFileHandlerlevel = INFOformatter = simpleFormatterargs = ("rotating_timed.log",)# filenamekwargs = {'when':'H','interval': 1,'backupCount': 24,'use_gzip': True}[formatter_simpleFormatter]format = %(asctime)s - %(name)s - %(levelname)s - %(message)s

Note: Ensure youimport concurrent_log_handler in your Python codebefore callingfileConfig(). Python 3.7+ isrecommended forkwargs support in config files.

⚠️ Removal Timeline: Thequeue module will be removed in v1.0.0 (planned for Summer 2025)

If you're currently usingconcurrent_log_handler.queue.setup_logging_queues():

This utility is deprecated and will be removed in v1.0.0 due to compatibilityissues with complex logging setups. It was provided for convenience but is notan integral part of CLH's functionality.

Important: consider whether you need non-blocking logging at all. Typicalapplications can get reasonable performance with the normal (synchronous) CLH handlers.

1. Remove calls tosetup_logging_queues()

2. Replace with the standard library patterns shown above or inPerformance Patterns

   or: simply use the CLH handlers directly in your application without thequeue utility.

3. Test your application with the new approach

The old utility:

• ❌ Modified all handlers globally (monkey patching)
• ❌ Had compatibility issues with advanced logging libraries
• ❌ Used unbounded queues (memory risk)
• ❌ Lacked proper error handling

The new approach:

• ✅ Uses standard library patterns
• ✅ Gives applications explicit control
• ✅ Works with any logging setup
• ✅ Provides better error visibility

For detailed migration examples, seePerformance Patterns.

• maxBytes is a Guideline: The actual log file size might slightly exceedmaxBytesbecause the check is performedbefore writing a new log message. The file cangrow by the size of that last message. This behavior prioritizes preserving logrecords. StandardRotatingFileHandler is stricter but may truncate.

• backupCount Performance: Avoid excessively highbackupCount values (e.g., > 20-50).Renaming many files during rotation can be slow, and this occurs while the logfile is locked. Consider increasingmaxBytes instead if you need to retainmore history in fewer files.

• Gzip Compression: Enablinguse_gzip adds CPU overhead.

• Restart on Configuration Change: If you change logging settings (e.g.,rotation parameters), it's often best to restart all application processes toensure all writers use the new, consistent configuration.

If you plan to modify or contribute to CLH:

1. Clone the repository:

   git clone https://github.com/Preston-Landers/concurrent-log-handler.gitcd concurrent-log-handler

2. Create a virtual environment and activate it:

   python -m venv venvsource venv/bin/activate# On Windows: venv\Scripts\activate

3. Install in editable mode with development dependencies:

   pip install -e .[dev]

   This installshatch,black and other development tools.

4. Run tests:

   # Run tests on the current (venv) Python versionpytest# or run:hatchtest# Generate coverage reportpytest --cov --cov-report=html --cov-report=xml --cov-report=lcov --cov-report=term-missing# Tests across all supported Python versions in the GitHub Actions matrix# However, Python 3.6 and 3.7 are not supported by GitHub Actions due to their EOL status.

5. Build for distribution:

   hatch build --clean

   The distributable files will be in thedist/ folder. To upload (maintainers only):

   pip install twinetwine upload dist/*

This package is a fork of Lowell Alleman'sConcurrentLogHandler 0.9.1. The fork was created to address ahanging/deadlockingissue (Launchpad Bug #1265150) and has sinceincorporated numerous other fixes, features, and modernizations.

• GitHub Repository:https://github.com/Preston-Landers/concurrent-log-handler
• PyPI Package:https://pypi.org/project/concurrent-log-handler/

SeeCHANGELOG.md for a detailed history of changes.

The original version was by Lowell Alleman. Subsequent contributions are listed inCONTRIBUTORS.md.

This project is licensed under the terms of theLICENSE file (Apache 2.0 terms).