Looks like the same as#1190

Looks like the same as#1190

I will have a look

@xenova a few thoughts on what i did:

I added abort_signal to PretrainedOptions. Most places just pass the options around, but many places rebuild them, so you will see more changes than expected because of this.

I also added it to the ModelTokenizerProcessorConstructorArgs constructor and there are edge cases like _call_text_to_spectrogram. Not sure I like this though. :/

This PR so far only handles downloading models, etc.

Do you know where I should look as far as ONNX/ORT to cancel generation?

Looks like the same as#1190

I will have a look

I can see reasons to have a custom fetch() but I think those are orthogonal to the ability to abort.

But perhaps we could have both.

I want to be able to abort generation as well as downloads, in which case fetch is not enough.

For reference:microsoft/onnxruntime#23703

Also, I madeabort_signal required in various places... that is to help me find the places I have missed. I will make it optional when done.

Looks like the same as#1190

I will have a look

I can see reasons to have a custom fetch() but I think those are orthogonal to the ability to abort.

But perhaps we could have both.

I want to be able to abort generation as well as downloads, in which case fetch is not enough.

I’ve been trying to figure out a way to stop text generation and so far I’ve come up with the following:

Something like thisabortOnSignal helper function to wrap async operations to abort:

/** * A helper function to wrap asynchronous operations that can be stopped/aborted by an AbortSignal. *@param {Promise<unknown>} promise Any promise/async operation to be aborted. *@param {AbortSignal | null | undefined} signal The abort signal from an AbortController that can cancel the wrapped promise/async operation. *@returns {Promise<unknown>} Either the param promise or a rejected promise with AbortError if aborted. */exportfunctionabortOnSignal(promise,signal){if(!signal){returnpromise;}returnnewPromise((resolve,reject)=>{constabortHandler=()=>{reject(newDOMException('Aborted','AbortError'));};signal.addEventListener('abort',abortHandler,{once:true});promise.then(resolve,reject).finally(()=>{signal.removeEventListener('abort',abortHandler);});});}

Then add an abortsignal param to thesessionRun function inmodels.js and wrap thesession.run invocation with theabortOnSignal function with thesignal param passed into it:

/** * Executes an InferenceSession using the specified inputs. * NOTE: `inputs` must contain at least the input names of the model. *  - If additional inputs are passed, they will be ignored. *  - If inputs are missing, an error will be thrown. * *@param {Object} session The InferenceSession object to run. *@param {Object} inputs An object that maps input names to input tensors. *@param {AbortSignal | null | undefined} signal Optional abort signal from an AbortController to cancel the InferenceSession. *@returns {Promise<Object>} A Promise that resolves to an object that maps output names to output tensors. *@private */asyncfunctionsessionRun(session,inputs,signal=null){constcheckedInputs=validateInputs(session,inputs);try{// pass the original ort tensorconstortFeed=Object.fromEntries(Object.entries(checkedInputs).map(([k,v])=>[k,v.ort_tensor]));letoutput=awaitabortOnSignal(session.run(ortFeed),signal);output=replaceTensors(output);returnoutput;}catch(e){if(e.name==='AbortError'){console.log("Generation aborted by user.");}// Error messages can be long (nested) and uninformative. For this reason,// we apply minor formatting to show the most important informationconstformatted=Object.fromEntries(Object.entries(checkedInputs).map(([k,{ type, dims, data}])=>[k,{// Extract these properties from the underlying ORT tensor                type, dims, data,}]));// This usually occurs when the inputs are of the wrong type.console.error(`An error occurred during model execution: "${e}".`);console.error('Inputs given to model:',formatted);throwe;}}

To use it you pass the signal into the generator as an argument:

constprompt=“Howmanyletter‘r’sareintheword‘strawberry’?”constcontroller=newAbortController();constgenerator=awaitpipeline('text-generation',MODEL,{dtype:'q4f16',device:'webgpu'});constgenerationArgs={max_new_tokens:maxTokens,streamer:streamer,signal:controller.signal};awaitgenerator(prompt,generationArgs);

Here’s a little demo of it working (Ignore the text output nonsense. I think my laptop processor is messed up somehow):

It works at stopping the generation but the way I have it now feels kinda messy passing thesignal around so much.

TakeTextGenerationPipeline for example. The signal gets passed in through its_call args intothis.model.generate then intothis.forward then intothis._forward which isdecoderForward then intodecoderForwards invocation ofsessionRun then finally into theabortOnSignal wrapped aroundsession.run.

I'm sure there's a cleaner way to do it. Would love to get your guys take on this approach.

I haven't tried it myself yet, but it is definitely undocumented. I don't see signal or anything in the docs.

https://onnxruntime.ai/docs/api/js/interfaces/InferenceSession.SessionOptions.html#freeDimensionOverrides

Oh, I see now... you are simulating an abort. The text generation keeps going though.

If you are using progress callbacks, you can just ignore them (after the user hits stop) to the same effect.

Oh, I see now... you are simulating an abort. The text generation keeps going though.

If you are using progress callbacks, you can just ignore them (after the user hits stop) to the same effect.

Oh, the text generation actually does stop completely when the abort signal is triggered. I did some testing with Chrome dev tools and the CPU usage drops to 0% when aborted, showing the generation process fully terminates.

The way it works is:

1. When the abort signal triggers, theabortOnSignal wrapper rejects the ONNX session.run() Promise with an AbortError
2. This error bubbles up through the generation loop ingenerate(), stopping the entire process

But yeah, I'm not super happy with having to pass the signal through so many layers. I was trying to follow the pattern used by the Fetch API where you need to get the abort signal all the way down to the actual async operation being aborted. Would be cool if in the future theInferenceSession could support the abort signal internally.

But maybe there's a cleaner way to structure this. Like maybe we could store the signal at the pipeline level instead of threading it through all those args. Let me know if you have any other ideas for improving it.

It seems like you're looking forhttps://github.com/huggingface/transformers.js/blob/main/src/models.js#L1573,
you can refer tohttps://github.com/huggingface/transformers.js-examples/blob/main/deepseek-r1-webgpu/src/worker.js#L114.

It seems like you're looking forhttps://github.com/huggingface/transformers.js/blob/main/src/models.js#L1573, you can refer tohttps://github.com/huggingface/transformers.js-examples/blob/main/deepseek-r1-webgpu/src/worker.js#L114.

I've been using theEosTokenCriteria and found it works great at stopping text generation at known specific token outputs. However, when I encountered the funny-looking text output issue in the video I posted above and then saw these abort signal PR's, it got me thinking: it would be cool if we could use an abort signal to stop any long-running async task in the entire pipeline, whether the task is downloading with fetch, token generation, or any other time-consuming asynchronous task. What do you guys think?

I updated the API to make sure abort_signal is never a naked parameter. Instead it is always part of an options object so others can be added later. Also set default value to null instead of undefined just so it doesn't stick out in the codebase as different.

TODO: at least forpipeline(), I will create anInterruptableStoppingCriteria that will activate when the signal activates. I'll add it if no other stopping criteria, and convert any other to a list version and add this one to it if there is an abort_signal passed. Thanks for the headsup@emojiiii and@joecal

[edit: that should probably be a separate PR]

I have a version of transformers.js with abort signal handling at @sroussey/transformers

[edit, both 3.3.3 and 3.4.0 versions]