wav2vec2-base-en-de-100k

This is a bilingual Wav2Vec 2.0 base model pre-trained from 100 thousand hours of speech (50 thousand hours of English and 50 thousand hours of German). It has been released along with a paper A Comparative Analysis of Bilingual and Trilingual Wav2Vec Models for Automatic Speech Recognition in Multilingual Oral History Archives accepted to INTERSPEECH2024 conference.

Paper

https://www.isca-archive.org/interspeech_2024/lehecka24_interspeech.pdf

Pre-print: http://arxiv.org/abs/2407.17160.

All pre-trained models released along with the paper

• fav-kky/wav2vec2-base-cs-50k (monolingual Czech)
• fav-kky/wav2vec2-base-de-50k (monolingual German)
• fav-kky/wav2vec2-base-cs-en-100k (bilingual Czech+English)
• fav-kky/wav2vec2-base-cs-de-100k (bilingual Czech+German)
• fav-kky/wav2vec2-base-en-de-100k (bilingual English+German)
• fav-kky/wav2vec2-base-cs-en-de-150k (trilingual Czech+English+German)

Citation

If you find this model useful, please cite our paper:

@inproceedings{lehecka24_interspeech,
  title     = {A Comparative Analysis of Bilingual and Trilingual Wav2Vec Models for Automatic Speech Recognition in Multilingual Oral History Archives},
  author    = {Jan Lehečka and Josef V. Psutka and Lubos Smidl and Pavel Ircing and Josef Psutka},
  year      = {2024},
  booktitle = {Interspeech 2024},
  pages     = {1285--1289},
  doi       = {10.21437/Interspeech.2024-472},
  issn      = {2958-1796},
}

Usage

This model does not have a tokenizer as it was pretrained on audio alone. In order to use this model for speech recognition, a tokenizer should be created and the model should be fine-tuned on labeled ASR data.

Inputs must be 16kHz mono audio files.

This model can be used e.g., to extract per-frame contextual embeddings from audio:

from transformers import Wav2Vec2Model, Wav2Vec2FeatureExtractor
import torchaudio

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("fav-kky/wav2vec2-base-en-de-100k")
model = Wav2Vec2Model.from_pretrained("fav-kky/wav2vec2-base-en-de-100k")

speech_array, sampling_rate = torchaudio.load("/path/to/audio/file.wav")
inputs = feature_extractor(
    speech_array, 
    sampling_rate=16_000, 
    return_tensors="pt"
)["input_values"][0]

output = model(inputs)
embeddings = output.last_hidden_state.detach().numpy()[0]

Related works