This pull request makes significant enhancements to the server implementation in whisper.cpp, especially around concurrency and request handling. Here’s a summary of the key changes:

Major Changes

1. Multi-threaded Task Queue for Inference

• Introduces a newWhisperTaskQueue class, which manages a queue of inference tasks and processes them using multiple worker threads (the number of workers is now configurable via a new--workers server argument).
• All incoming POST requests for transcription are now handled asynchronously via this queue, allowing multiple requests to be processed in parallel.
• Each task encapsulates all request information and handles its own completion signaling and abort reasons (e.g., client disconnect, server shutdown).

2. Graceful Shutdown and Robust Handling

• The server now handles shutdowns more gracefully: ongoing and queued tasks are notified and can be aborted with meaningful error responses (503 for server shutdown,499 for client disconnect).
• The task queue is properly destroyed and re-created when a new model is loaded or on shutdown, ensuring clean resource management.

3. Refactoring of Main Server Logic

• The request handler for audio transcription (POST /inference_path) is refactored to create a fresh copy of the parameter set for each request, improving thread safety.
• The actual inference is now performed inside the worker threads, so the server remains responsive to new requests.
• The code for response formatting (plain text, SRT, VTT, verbose JSON) is encapsulated within the task result logic.

4. Command-Line Interface and Usability Improvements

• Adds a--workers N argument to control the number of concurrent worker threads for handling requests.
• Usage help text and parameter parsing are updated accordingly.

5. Minor Fixes and Improvements

• Swaps the use ofstd::atomic_flag for a more idiomaticstd::atomic<bool> for the termination signal.
• Reduces use of global and shared state, improving modularity and reliability.

Example: New Option

./server --workers 4

This will launch the server with 4 worker threads for handling requests concurrently.