Repository files navigation

A general purposevoice user interface for the IntelliJ Platform, inspired byTavis Rudd.Possible use cases: visually impaired andRSI users. Originally developed as part of aJetBrains hackathon, it is now a community-supported project. For background information, check outthis presentation.

• idiolect-azure

To get started, press the button in the toolbar, then speak a command, e.g. "Hi, IDEA!" Idiolect supports a simplegrammar. For a complete list of commands, please refer tothe wiki. Click the button once more to deactivate.

For a full list of all actions that can be activated by voice ask Idiolect:

What can I say?

or

What can I say about "activate"?

There area lot of actions, and some of them are not easy to say or remember. To make it easier you cancustomise the phrases.

Edit custom phrases

Some of the more useful commands are:

• activate project tool window (commit, database, debug, find, gradle, run, terminal...)
• back/forward, down/up, left/right, page up/down, scroll up/down
• next/previous word
• method down/up
• find, find in project
• go to ... (class, bookmark, declaration, line, implementation...)
• close all editors
• next method
• find usages
• call hierarchy

• create (editor config file, grpc request action, liquibase changelog, vue single file comp)
• new class/dockerfile/element/file...
• code cleanup
• code completion
• collapse block
• extract class/function/method/etc
• generate getter/setter/test method...
• new class (or "create new class "something")
• rename (or rename to "something")
• cut / copy / paste / delete
• cut line end/backward
• delete to (line/word) (start/end)
• toggle column mode
• reformat code
• fix it
• whoops

• debug
• context debug
• context run
• coverage

• git add/pull/merge/push/stash...
• checkin files
• annotate

For Linux or macOS users:

git clone https://github.com/OpenASR/idiolect && cd idiolect && ./gradlew runIde

For Windows users:

git clone https://github.com/OpenASR/idiolect & cd idiolect & gradlew.bat runIde

Recognition works with most popular microphones (preferably 16kHz, 16-bit). For best results, minimize background noise.

Contributors who have IntelliJ IDEA installed can simply open the project. Otherwise, run the following command from the project's root directory:

./gradlew runIde -PluginDev

Idiolect is implemented using theIntelliJ Platform SDK. For more information about the plugin architecture, please refer tothe wiki page.

plugin.xml defines a number of<extensionPoint>s which would allow other plugins to integrate with or extend/customise the capabilities of Idiolect.

An example of this is provided inidiolect-azure which implementsAsrProviderand adds its own settings underTools/Idiolect.

Listens for audio input, recognises speech to text and returns anNlpRequest with possible utterances.Doesnot resolve the intent.

Possible alternative implementations could:

• integrate with Windows SAPI 5 Speech API
• integrate with Dragon/Nuance API

Processes anNlpRequest.The default implementation invokesIdeService.invokeAction(ExecuteVoiceCommandAction, nlpRequest)and the action is handled byExecuteVoiceCommandAction andActionRecognizerManager.handleNlpRequest()

Processes audio input, recognises speech to text and executes actions.The default implementationAsrControlLoop uses theAsrProvider andNlpProvider.

Some APIs such as AWS Lex implement the functionality ofAsrProvider andNlpProvider in a single call.

Processes anNlpRequest (utterance/alternatives) and resolves anNlpResponse withintentName andslots.ActionRecognizerManager.handleNlpRequest() iterates through theIntentResolvers until it finds a match.

The Idiolect implementations use either exact-match or regular expressions on the recognized text.Alternative implementations may use AI to resolve the intent.

Many of the auto-generated trigger phrases are not suitable for voice activation. You can add your own easier tosay and remember phrases in~/.idea/phrases.properties

Fulfills anNlpResponse (intent + slots), performing desired actions.ActionRecognizerManager.handleNlpRequest() iterates through theIntentHandlers until the intent is actioned.

Handles two flavours of intent prefix:

• Template.id.${template.id} eg:Template.id.maven-dependency
• Template.${template.groupName}.${template.key} eg:Template.Maven.dep

template.id is often null.template.key is the "Abbreviation" that you would normally type before pressingTAB.

The default trigger phrases are generated from the template description or key and are often not suitable for voice activation.You can add your own trigger phrase -> live template mapping in~/.idea/phrases.properties and it will be resolved byCustomPhraseRecognizer.

Reads audio prompts/feedback to the user

Any interfaces which are registered to the topic in plugin.xml under<applicationListeners> will be notified when

• listening state changes
• recognition is returned by theAsrProvider
• request is fulfilled by anIntentHandler
• there is a failure
• a prompt/message is provided for the user

plugin.xml defines<action>s:

This action is invoked when the user clicks on the button in the toolbar.This simply tellsAsrService to activate or standby.When theAsrService is active, theAsrSystem,

by defaultASRControlLoop(see below).

A debugging aid to use one of theActionRecognizerextension classes configured inplugin.xml to generate anActionCallInfowhich is thenrunInEditor().

Similar toExecuteActionFromPredefinedText but uses theIdiolect.VoiceCommand.Text data attached to the invokingAnActionEvent.

There are many Actions (classes which extendAnAction) provided by IDEA:

• ExternalSystemActions
• LangActions
• PlatformActions
• VcsActions

WhenAsrControlLoop detects an utterance, it invokesPatternBasedNlpProvider.processUtterance()which typically callsinvokeAction() and/or one or more of the methods ofIdeService

• Interactive IDE Voice Control
• Using Python to Code by Voice
• How a Blind Developer uses Visual Studio

• Breandan Considine
• Nicholas Albion