The main features for this release are:

• Support for Inference endpoints! credits to@vvmnnnkv
• Custom requests viarequest andstreamingRequest
• Possibility to import the methods directly without the need to instantiate anHfInference class: great for tree-shaking
• New NLP task:featureExtraction (the existingfeatureExtraction task was renamed tosentenceSimilarity, oops!), credits@radames

The other changes for recent versions are detailed at the end (includingtextGenerationStream for streaming text generation, ...)

Support for Inference Endpoints

Inference endpoints are the next step for using Inference API for a specific model in production.

The different tiers for inference are:

• Inference API (no token): restrictive rate limits
• Inference API - free account: usable rate limits
• Inference API - PRO account: better rate limits
• Inference Endpoints: Unlimited API calls, possibility to deploy on the cloud provider / VPC / infra of your choice, scaling

Here's how you can call an inference endpoint:

constinference=newHfInference("hf_...");constgpt2=inference.endpoint('https://xyz.eu-west-1.aws.endpoints.huggingface.cloud/gpt2');const{ generated_text}=awaitgpt2.textGeneration({inputs:'The answer to the universe is'});

You can even use the free inference API backend with this syntax:

constendpoint=inference.endpoint("https://api-inference.huggingface.co/models/google/flan-t5-xxl");const{ generated_text}=awaitendpoint.textGeneration({inputs:"one plus two equals",});

It's easy to switch between Inference API & Inference Endpoints. So easy, that you can even do this:

awaitinference.textGeneration({model:'https://xyz.eu-west-1.aws.endpoints.huggingface.cloud/gpt2',inputs:'The answer to the universe is'});

Custom requests

@huggingface/inference supports tasks fromhttps://huggingface.co/tasks, and is typed accordingly. But what if your model has additional inputs, or even custom inputs or outputs?

You can now use.request and.streamingRequest!

constoutput=awaitinference.request({inputs:"blablabla",parameters:{custom_parameter_1: ...,    ...}});

For streaming responses, usestreamingRequest rather thanrequest.

All existing tasks can userequest orstreamingRequest instead 🤯

const{generated_text}=awaitinference.textGeneration({model:"gpt2",inputs:"The answer to the universe is"});// small output change for .textGeneration to .request: the raw response is actually an arrayconst[{generated_text}]=awaitinference.request({model:"gpt2",inputs:"The answer to the universe is"});forawait(constoutputofinference.textGenerationStream({model:"google/flan-t5-xxl",inputs:"Repeat 'one two three four'"})){}// is equivalent toforawait(constoutputofinference.streamingRequest({model:"google/flan-t5-xxl",inputs:"Repeat 'one two three four'"})){}

Of course,request andstreamingRequest can also be used with Inference Endpoints! Actually, if you make your own custom models and inputs / outputs for your business use case, it'll probably be what you use.

Individual imports & tree-shakability

You don't like the current API, you don't like classes, and want the strict minimum in your bundle? No need to say more, I know which frontend framework (or should I say library ;)) you use.

Don't worry, you can import individual functions - this release of@hugginface/inference is all about choice and flexibility:

import{textGeneration}from"@huggingface/inference";awaittextGeneration({accessToken:"hf_...",// new parammodel:"gpt2",// or your own inference endpointinputs:"The best, most efficient and purest frontend framework is: "});

Breaking changes

• questionAnswer andtableQuestionAnswer have been renamed toquestionAnswering andtableQuestionAnswering
• The existingfeatureExtraction has been renamed tosentenceSimilarity and a newfeatureExtraction was created 🙇

Other changes from recent releases:

• textGenerationStream to generate streaming content by returning anAsyncIterable. Yay forfor await! Credits to@vvmnnnkv.Demo
• imageToText to caption images among other things. Credits to@vvmnnnkv.Demo
• Validation of outputs: Userequest orstreamingRequest to skip this validation. Credits to@mishig25

Contributors