Repository files navigation

This is a script that I worked on to help empower physicians to alleviate the burden of documentation by utilizing a medical scribe to create SOAP notes. Expensive solutions could potentially share personal health information with their cloud-based operations. It utilizesKoboldcpp andWhisper on a local server that is concurrently running theServer.py script. TheClient.py script can then be used by physicians on their device to record patient-physician conversations after a signed consent is obtained and process the result into a SOAP note.

Regards,

Braedon Hendy

Youtube Demo

• 2024-03-17 - updatedclient.py to allow forOpenAI token access whenGPT button is selected. A prompt will show to allow for scrubbing of any personal health information.
• 2024-03-28 - updatedclient.py to allow forWhisper to run locally when set toTrue in the settings.
• 2024-03-29 - addedScrubadub to be used to remove personal information prior toOpenAI token access.
• 2024-04-26 - added alternative server file to useFaster-Whisper
• 2024-05-03 - added alternative server file to useWhisperX
• 2024-05-06 - added real-timeWhisper processing
• 2024-05-13 - addedSSL and OHIP scrubbing
• 2024-05-14 -GPT model selection
• 2024-06-01 - template options and further fine-tuning for local and remote real-timeWhisper

Example instructions for running on a single machine:

I will preface that this will run slowly if you are not using a GPU but will demonstrate the capability.

InstallPython3.10.9HERE. (if the hyperlink doesn't workhttps://www.python.org/downloads/release/python-3109/). Make sure you click the checkbox to select "Add Python to Path".

Next, you need to install software to convert the audio file to be processed. PressWindows key +R, you can run the command line by typingpowershell. Copy/type the following:

Set-ExecutionPolicy-ExecutionPolicy RemoteSigned-Scope CurrentUserInvoke-RestMethod-Uri https://get.scoop.sh|Invoke-Expressionscoop install ffmpeg

If this was successful, you need to download the files that I wroteHERE. Unzip the files (if the hyperlink doesn't workhttps://github.com/1984Doc/AI-Scribe).

Run theclient.py (it may prompt for installation of various dependencies viapip)

I would recommend using theGPT option using anAPI key. The cost for running each model may determine the overall choice and can be selected in theSettings menu of the program.

Example instructions for running on a server with a GPU:

InstallPython3.10.9HERE. (if the hyperlink doesn't workhttps://www.python.org/downloads/release/python-3109/). Make sure you click the checkbox to select "Add Python to Path".

PressWindows key +R, you can run the command line by typingcmd. Copy/type the following, running each line by pressingEnter:

pip install openai-whisper

Now you need to download the AI model (it is large). I recommend theMistral 7B v0.2 orMeta Llama 3 models. These can be found onHuggingFace.

You now need to launch the AI model with the following software that you can downloadHERE. It will download automatically and you will need to open it (if hyperlink doesn't workhttps://github.com/LostRuins/koboldcpp/releases). If you have anNVidia RTX-based card, the below instructions can be modified usingKoboldcpp.exe rather thankoboldcpp_nocuda.exe.

Once theKoboldcpp.exe is opened, click theBrowse button and select the model downloaded. Now click theLaunch button.

You should see a window open and can ask it questions to test!

If this was successful, you need to download the files that I wroteHERE. Unzip the files (if the hyperlink doesn't workhttps://github.com/1984Doc/AI-Scribe).

Run theserver.py file. This will download the files to help organize the text after converting from audio.

Run theclient.py file and edit the IP addresses in theSettings menu.

1. Download and install janAI and configure with your LLM of choice.
2. Start the JanAI server.
3. Open the python client applications and set the Model Endpoint to your settings in the JanAI (Typicallyhttp://localhost:1337/v1/ by default)
4. Set your model to the one of choice (Gemma 2 2b recommended Model ID: "gemma-2-2b-it")
5. Save the settings
6. Click the KoboldCPP button to enable custom endpoint.