Description

Currently, the airflow project uses PEP-3143 style daemons to launch tasks (as implemented inhttps://pypi.org/project/python-daemon/), however this is targeted towards unix daemons. As a result, running airflow on windows requires multiple levels of abstraction each with their own problems. Would it be possible to use something like daemoniker (https://daemoniker.readthedocs.io/en/latest/) to launch tasks? What are the challenges and issues?

In machine learning workflows, with large datasets, it is a huge time-saver if the pipeline tasks can be run on the GPU. WSL 1 does not support GPU passthrough, docker through WSL 2 supports GPU passthrough only with the Insiders build, additionally it has issues with networking when connected to VPN (microsoft/WSL#5068).

Use case / motivation

Natively running airflow without WSL 1/2 or docker on Windows. This is helpful in cases where the company ecosystem is windows-based.

Possible implementation

The daemon module is only used to daemonize the scheduler and webserver. Here's a sample code that runs the scheduler (airflow origin/v1-10-stable) using daemoniker, comments are welcome:

# airflow/bin/cli.pyfromdaemonikerimportDaemonizer...ifargs.daemon:withDaemonizer()as (is_setup,daemonizer):ifis_setup:pid,stdout,stderr,log_file=setup_locations("scheduler",args.pid,args.stdout,args.stderr,args.log_file)_is_parent=daemonizer(pid,stdout_goto=stdout,stderr_goto=stderr        )job.run()