What is a melodyplot?

A proprietary software named Melodyne, made by Celemony GmbH, has a very intuitive interface for displaying melodies. It sits right in between a wave plot, a pitch contour plot, and a piano-roll representation. Here's a video demonstrating the tool:

https://www.youtube.com/watch?v=2v52AeU59N0

Why is a melodyplot useful for MIR?

Shortly put, the benefit of this visualization is that it superimposes three different time scales in the evolution of the melody:
(1) the time scale of amplitude modulation is shown on the waveform plot
(2) the time scale of frequency modulation is shown on the pitch contour plot
(3) the time scale of musical events is shown as blocks in the piano-roll domain

One person who uses this tool for musicological research is@ethanhein at NYU.

How to use a melodyplot?

I am proposing that we add a visualization like this in librosa, mixing (1) and (3) — the pitch contour (2) can always be added on top by the user, as a simple line plot. The functional prototype would be

melodyplot(y, melody, sr=22050, hop_length=512, max_points=50000.0, x_axis='time', offset=0.0, max_sr=1000, ax=None, **kwargs)

Note that this prototype is the same aswaveplot, with the exception ofmelody andhop_length.

The positional argumentmelody would be a one-dimensionalnp.ndarray encoding the quantized pitch curve in Hertz. The responsibility of quantizing this pitch curve according to a given temperament would rely on the user.

Null values ornp.nan values would encode unvoiced portions of the melody. This is the standard inmir_eval (https://github.com/craffel/mir_eval) and in the MedleyDB dataset.

The keyword argumenthop_length would be an integer, just as inpiptrack orstft. It would specify the sample rate of themelody signal assr/hop_length.

Advanced use cases

The waveform inputy could potentially be stereophonic. In this case, we would use the upper and lower part of the waveform to show the left and right channel—just as in awaveplot.

Regarding waveform colors, i think that we should follow the default color cycle of Matplotlib 2.0, i.e. the Vega category 10 palette: first "#1f77b4" (blue), second "#ff7f0e" (orange), third "#2ca02c" (green), fourth "#d62728" (red), fifth "#9467bd" (purple), and so on. This would allow to display several voices on the same figure. Of course, this is always something that could be changed manually, by way of acolor kwarg that would be passed tofill_between.

Long-term vision: YIN-based melodyplot

Long term (i.e. after#527 is merged), we could potentially havemelody="yin", so that the pitch curve is estimated automatically :) Then, displaying the melody from the first ten seconds of a file would be as terse as:

melodyplot(load(filename, duration=10)[0])

[EDIT: actually, the discussion below has shown that we should separate estimation (via YIN or otherwise) from display. But the possibility still holds. It would be three lines of code (load+YIN+melodyplot), not two.]

Calibrating the thickness of the waveform

From an algorithmic standpoint, one potential difficulty ofmelodyplot is that the amplitude ofy needs to be rescaled to that its apparent vertical thickness is comparable to the pitch range of the melody itself. In the Melody software, that thickness is set to one semitone in twelve-tone equal temperament. This could be our default, but it would be nice to be able to cover other use cases: non-Western temperament (ref#641 !) as well as non-tempered melodies, such as bioacoustic sounds. There is potentially room for a keyword argument here. The name of this keyword argument, as well as its physical unit (cough#925cough), would need some bikeshedding.

Conclusion

librosa.display.melodyplot is a display function which shows a waveformy in the piano-roll domain, in the style of the Melodyne proprietary software, according to some pre-computed quantized pitch curvemelody. I can only assume that this feature would be a huge improvement for doing computational musicology in Python.

Thank you for having read me up until the end. Please let me know what you think!
Vincent.

Post-scriptum

LIBREP means "Librosa Enhancement Proposal". Just an acronym i came up with.