Repository files navigation

This repository provides 1) a list of YouTube videos with Japanese subtitles (JTubeSpeech), 2) scripts for making new lists of new languages, and 3) tiny lists for other languages.

data/{lang}/{YYYYMM}.csv lists as follows. See step4 for download.

• lang: Language ID (ja [Japanese], en [English], ...)
• YYYYMM: Year and month when we collect data
• videoid: YouTube video ID. Its YouTube page ishttps://www.youtube.com/watch?v={videoid}.
• auto: The video has an automatic subtitle or not.
• sub: The video has a manual (i.e., human-generated) subtitle or not.
• channelid: YouTube Channel ID. Its YouTube page ishttps://www.youtube.com/channel/{channelid}.

• Shinnosuke Takamichi (The University of Tokyo, Japan) [main contributor]
• Ludwig Kürzinger (Technical University of Munich, Germany)
• Takaaki Saeki (The University of Tokyo, Japan)
• Sayaka Shiota (Tokyo Metropolitan University, Japan)
• Shinji Watanabe (Carnegie Mellon University, USA)

scripts/*.py are scripts for data collection from YouTube. Since processes of the scripts are language independent, users can collect data of their favorite languages.youtube-dl and ffmpeg are required.

The scriptscripts/make_search_word.py downloads the wikipedia dump file and finds words for searching videos.{lang} is the language code, e.g.,ja (Japanese) anden (English).

$ python scripts/make_search_word.py {lang}

The scriptscripts/obtain_video_id.py obtains YouTube video IDs by searching by words.{filename_word_list} is a word list file made in step1. After this step, the process will take a long time. It is recommended to split the files (e.g.,{filename_word_list}) and run them in parallel.

$ python scripts/obtain_video_id.py {lang} {filename_word_list}

The scriptscripts/retrieve_subtitle_exists.py retrieves whether the video has subtitles or not.{filename_videoid_list} is a videoID list file made in step2. This process will make a CSV file.

$ python scripts/retrieve_subtitle_exists.py {lang} {filename_videoid_list}

The scriptscripts/download_video.py downloads audio and manual subtitles. Note that, this process requires a very large amount of storage.{filename_subtitle_list} is a subtitle list file made in step3. The audio and subtitles will be saved invideo/{lang}/wav16k andvideo/{lang}/txt, respectively.

$ python scripts/download_video.py {lang} {filename_subtitle_list}

Subtitles are not always correctly aligned with the audio and in some cases, subtitles not fit to the audio.The scriptscripts/align.py aligns subtitles and audio with CTC segmentation using an ESPnet 2 ASR model:

$ python scripts/align.py {asr_train_config} {asr_model_file} {wavdir} {txtdir} {output_dir}

The result is written into a segments filesegments.txt and a log filesegments.log in the output directory.Using the segments file, bad utterances or audio files can be sorted-out:

min_confidence_score=-0.3awk -v ms=${min_confidence_score} '{ if ($5 > ms) {print} }' ${output_dir}/segments.txt

There are three types of videos: text-to-speech (a.k.a., TTS) video, single-speaker (i.e., monologue) video, and multi-speaker (e.g., dialogue) video. The scriptscripts/xxx.py obtains scores of speaker variation within a video to classify videos into three types.

$ python scripts/xxx.py

• coming soon

• youtube-dl
• Other corpora by main contributor

• Aug. 2021: first update ({lang}/*_tiny.csv)
• Jan. 2022: add mid-size data ({lang}/*_middile.csv)