Cool to have saving function 👍

Maybe useful to mention that the audio is expected to have shape(nb_channels, audio_length)? (unless self.processor expects something else...)

And are we sure it isn't a tensor? Otherwise can also add something likethis?

I guesssynthesiser andtts_pipeline try to ensure this dimension order, but maybe useful in case people gets issues saving.

cfthis comment

How about something more generic likepreset orstyle? At the moment, TTS models mainly make use of this feature (Bark calls itvoice_preset) but maybe music generation models may use this feature at one point?

wdyt@eustlb ?

agree!preset is good IMO

and "sequences"

do we need a condition if self.audio_channels > 1 but missing batch dimension?

Strange this was here, that's an easy unbloat! 😅

yes yes yes! Thanks for adding it

Nice catch!

I don't get why you're adding this for CSM and Dia. The fact that the models are mono is more of a design choice than a parameter, and such things should not change.

Nevertheless, if we add asave_audio inaudio_utils and rely on this function for saving audio in the processor, it will make sense to have it there, WDYT?

Agree with centralizing/standardizing asave_audio function inaudio_utils to avoid code duplication in different processors.

Another idea@eustlb, in several audio libraries I've seen the use of anAudio objects to group together useful properties and methods. WDYT? Here's an example below (quickly drafted with ChatGPT):

importsoundfileassfimportnumpyasnpimportosclassAudio:def__init__(self,waveform:np.ndarray,sample_rate:int,path:str=None):"""        Args:            waveform: NumPy array of shape (channels, samples).            sample_rate: Sampling rate in Hz.            path: Optional original file path.        """ifwaveform.ndim==1:waveform=waveform[np.newaxis, :]# Convert to (1, samples)elifwaveform.shape[0]>waveform.shape[1]:# Assume it's (samples, channels) — transpose to (channels, samples)waveform=waveform.Tself.waveform=waveformself.sample_rate=sample_rateself.path=path@classmethoddefload(cls,path:str):"""        Load audio from file (converted to (channels, samples)).        """ifnotos.path.isfile(path):raiseFileNotFoundError(f"Audio file not found:{path}")data,sample_rate=sf.read(path,always_2d=True)# (samples, channels)waveform=data.T# Transpose to (channels, samples)returncls(waveform,sample_rate,path)defsave(self,path:str):"""        Save audio to file (converted to (samples, channels)).        """data=self.waveform.T# Convert to (samples, channels)sf.write(path,data,self.sample_rate)self.path=pathdefduration_seconds(self)->float:returnself.waveform.shape[1]/self.sample_rate# can have others things for spectrogram, MFCC, augmentations, etc which can call functional versions

It can make working audio with much more convenient (not having to pass sample rate each time and channel dimension consistency). And can be used alongside functional approach

I don't think we should be able to override it. Like the number of channels, that's more of a fixed parameter for a certain model than something that users would be able to play with.

Moreover, I don't see a model where it would make sense to have it in generation config (bark has it but it's silent 🤔).

Sampling rate should live in two places:

1. processor → to prepare the inputs in the correct sampling rate
2. config → to indicate on which sampling rate the model has been trained

priority order should therefore be model config > processor (+ special case of the vocoder)

Agree with priority being in model config

Agreed with not allowing overwrites! (the option to overwrite it was present before, so I left it -> I will remove it then 👍 )

detail but can be safely removed

Trusting you on that one!

That's ok I think, stereo is exceptional. It's almost always not specified, with convention of being mono in this situation.

hum that's a bit misleading for CSM since the voice id does not refer to a predefined voice but rather a voice given a conventional context (cfthis doc example - the model can be fined-tuned to do predefined voices though! cfhere).

Here, setting a voice ID without providing a conversational context will just pick a random voice.

In the future I see multiple different ways of setting this voice, mainly either via a custom way of doing voice cloning, either by using a predefined voice identifier (number/ name/ etc). In both situations that's something that will end up being done in the processor.

What we can do is move fromvoice topreset, which will directly be passed to the processor and actualise current processors__call__ so that they can handle thispreset kwarg.

Can we move this toaudio_utils.py?
The best would be to redefineAudioInput type as a dict with keys audio (typed as former AudioInput) andsampling_rate (float) so that we finally move from having audio in the lib as an array with sampling rate. This should not be breaking as it's only type hinting. This might require removingAudioInput in favor of the current type hinting to places where it's currently been used.

See#39796 (comment) for idea on moving away from audio as independent array

Let's use the official ckpt imo