| | |
|---|
| | | ([简体中文](./README_zh.md)|English) |
|---|
| | | |
|---|
| | | # Speech Recognition |
|---|
| | | |
|---|
| | | > **Note**: |
|---|
| | |
|---|
| | | rec_result = inference_pipeline(audio_in='https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wav') |
|---|
| | | print(rec_result) |
|---|
| | | ``` |
|---|
| | | #### [Paraformer-long Model](https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) |
|---|
| | | ```python |
|---|
| | | from modelscope.pipelines import pipeline |
|---|
| | | from modelscope.utils.constant import Tasks |
|---|
| | | |
|---|
| | | inference_pipeline = pipeline( |
|---|
| | | task=Tasks.auto_speech_recognition, |
|---|
| | | model='damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch', |
|---|
| | | vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch', |
|---|
| | | #punc_model='damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch', |
|---|
| | | punc_model='damo/punc_ct-transformer_cn-en-common-vocab471067-large', |
|---|
| | | ) |
|---|
| | | |
|---|
| | | rec_result = inference_pipeline(audio_in='https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav', |
|---|
| | | batch_size_token=5000, batch_size_token_threshold_s=40, max_single_segment_time=6000) |
|---|
| | | print(rec_result) |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | Where, |
|---|
| | | - `batch_size_token` refs to dynamic batch_size and the total tokens of batch is `batch_size_token`, 1 token = 60 ms. |
|---|
| | | - `batch_size_token_threshold_s`: The batch_size is set to 1, when the audio duration exceeds the threshold value of `batch_size_token_threshold_s`, specified in `s`. |
|---|
| | | - `max_single_segment_time`: The maximum length for audio segmentation in VAD, specified in `ms`. |
|---|
| | | |
|---|
| | | Suggestion: When encountering OOM (Out of Memory) issues with long audio inputs, as the GPU memory usage increases with the square of the audio duration, there are three possible scenarios: |
|---|
| | | - a) In the initial inference stage, GPU memory usage primarily depends on `batch_size_token`. Reducing this value appropriately can help reduce memory usage. |
|---|
| | | - b) In the middle of the inference process, when encountering long audio segments segmented by VAD, if the total number of tokens is still smaller than `batch_size_token` but OOM issues persist, reducing `batch_size_token_threshold_s` can help. If the threshold is exceeded, forcing the batch size to 1 can be considered. |
|---|
| | | - c) Towards the end of the inference process, when encountering long audio segments segmented by VAD and the total number of tokens is smaller than `batch_size_token` but exceeds the threshold `batch_size_token_threshold_s`, forcing the batch size to 1 may still result in OOM errors. In such cases, reducing `max_single_segment_time` can be considered to shorten the duration of audio segments generated by VAD. |
|---|
| | | |
|---|
| | | #### [Paraformer-online Model](https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) |
|---|
| | | ##### Streaming Decoding |
|---|
| | | ```python |
|---|
| | | inference_pipeline = pipeline( |
|---|
| | | task=Tasks.auto_speech_recognition, |
|---|
| | | model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online', |
|---|
| | | model_revision='v1.0.6', |
|---|
| | | model_revision='v1.0.7', |
|---|
| | | update_model=False, |
|---|
| | | mode='paraformer_streaming' |
|---|
| | | ) |
|---|
| | | import soundfile |
|---|
| | | speech, sample_rate = soundfile.read("example/asr_example.wav") |
|---|
| | | |
|---|
| | | chunk_size = [5, 10, 5] #[5, 10, 5] 600ms, [8, 8, 4] 480ms |
|---|
| | | param_dict = {"cache": dict(), "is_final": False, "chunk_size": chunk_size} |
|---|
| | | chunk_size = [0, 10, 5] #[0, 10, 5] 600ms, [0, 8, 4] 480ms |
|---|
| | | encoder_chunk_look_back = 4 #number of chunks to lookback for encoder self-attention |
|---|
| | | decoder_chunk_look_back = 1 #number of encoder chunks to lookback for decoder cross-attention |
|---|
| | | param_dict = {"cache": dict(), "is_final": False, "chunk_size": chunk_size, |
|---|
| | | "encoder_chunk_look_back": encoder_chunk_look_back, "decoder_chunk_look_back": decoder_chunk_look_back} |
|---|
| | | chunk_stride = chunk_size[1] * 960 # 600ms、480ms |
|---|
| | | # first chunk, 600ms |
|---|
| | | speech_chunk = speech[0:chunk_stride] |
|---|
| | |
|---|
| | | inference_pipeline = pipeline( |
|---|
| | | task=Tasks.auto_speech_recognition, |
|---|
| | | model='damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online', |
|---|
| | | model_revision='v1.0.6', |
|---|
| | | model_revision='v1.0.7', |
|---|
| | | update_model=False, |
|---|
| | | mode="paraformer_fake_streaming" |
|---|
| | | ) |
|---|
| | |
|---|
| | | print(rec_result) |
|---|
| | | ``` |
|---|
| | | Full code of demo, please ref to [demo](https://github.com/alibaba-damo-academy/FunASR/discussions/241) |
|---|
| | | |
|---|
| | | #### [Paraformer-contextual Model](https://www.modelscope.cn/models/damo/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404/summary) |
|---|
| | | ```python |
|---|
| | | from modelscope.pipelines import pipeline |
|---|
| | | from modelscope.utils.constant import Tasks |
|---|
| | | |
|---|
| | | param_dict = dict() |
|---|
| | | # param_dict['hotword'] = "https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/hotword.txt" |
|---|
| | | param_dict['hotword']="邓郁松 王颖春 王晔君" |
|---|
| | | inference_pipeline = pipeline( |
|---|
| | | task=Tasks.auto_speech_recognition, |
|---|
| | | model="damo/speech_paraformer-large-contextual_asr_nat-zh-cn-16k-common-vocab8404", |
|---|
| | | param_dict=param_dict) |
|---|
| | | |
|---|
| | | rec_result = inference_pipeline(audio_in='https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_hotword.wav') |
|---|
| | | print(rec_result) |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | #### [UniASR Model](https://www.modelscope.cn/models/damo/speech_UniASR_asr_2pass-zh-cn-8k-common-vocab3445-pytorch-online/summary) |
|---|
| | | There are three decoding mode for UniASR model(`fast`、`normal`、`offline`), for more model details, please refer to [docs](https://www.modelscope.cn/models/damo/speech_UniASR_asr_2pass-zh-cn-8k-common-vocab3445-pytorch-online/summary) |
|---|
| | |
|---|
| | | ``` |
|---|
| | | The decoding mode of `fast` and `normal` is fake streaming, which could be used for evaluating of recognition accuracy. |
|---|
| | | Full code of demo, please ref to [demo](https://github.com/alibaba-damo-academy/FunASR/discussions/151) |
|---|
| | | |
|---|
| | | #### [Paraformer-Spk](https://modelscope.cn/models/damo/speech_paraformer-large-vad-punc-spk_asr_nat-zh-cn/summary) |
|---|
| | | This model allows user to get recognition results which contain speaker info of each sentence. Refer to [CAM++](https://modelscope.cn/models/damo/speech_campplus_speaker-diarization_common/summary) for detailed information about speaker diarization model. |
|---|
| | | ```python |
|---|
| | | from modelscope.pipelines import pipeline |
|---|
| | | from modelscope.utils.constant import Tasks |
|---|
| | | |
|---|
| | | if __name__ == '__main__': |
|---|
| | | audio_in = 'https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_speaker_demo.wav' |
|---|
| | | output_dir = "./results" |
|---|
| | | inference_pipeline = pipeline( |
|---|
| | | task=Tasks.auto_speech_recognition, |
|---|
| | | model='damo/speech_paraformer-large-vad-punc-spk_asr_nat-zh-cn', |
|---|
| | | model_revision='v0.0.2', |
|---|
| | | vad_model='damo/speech_fsmn_vad_zh-cn-16k-common-pytorch', |
|---|
| | | punc_model='damo/punc_ct-transformer_cn-en-common-vocab471067-large', |
|---|
| | | output_dir=output_dir, |
|---|
| | | ) |
|---|
| | | rec_result = inference_pipeline(audio_in=audio_in, batch_size_token=5000, batch_size_token_threshold_s=40, max_single_segment_time=6000) |
|---|
| | | print(rec_result) |
|---|
| | | ``` |
|---|
| | | |
|---|
| | | #### [RNN-T-online model]() |
|---|
| | | Undo |
|---|
| | | |
|---|
| | |
|---|
| | | if __name__ == '__main__': |
|---|
| | | params = modelscope_args(model="damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch", data_path="./data") |
|---|
| | | params.output_dir = "./checkpoint" # m模型保存路径 |
|---|
| | | params.data_path = "./example_data/" # 数据路径 |
|---|
| | | params.data_path = "speech_asr_aishell1_trainsets" # 数据路径 |
|---|
| | | params.dataset_type = "small" # 小数据量设置small，若数据量大于1000小时，请使用large |
|---|
| | | params.batch_bins = 2000 # batch size，如果dataset_type="small"，batch_bins单位为fbank特征帧数，如果dataset_type="large"，batch_bins单位为毫秒， |
|---|
| | | params.max_epoch = 20 # 最大训练轮数 |
|---|
| | |
|---|
| | | - `batch_bins`: batch size. For dataset_type is `small`, `batch_bins` indicates the feature frames. For dataset_type is `large`, `batch_bins` indicates the duration in ms |
|---|
| | | - `max_epoch`: number of training epoch |
|---|
| | | - `lr`: learning rate |
|---|
| | | - `init_param`: init model path, load modelscope model initialization by default. For example: ["checkpoint/20epoch.pb"] |
|---|
| | | - `freeze_param`: Freeze model parameters. For example：["encoder"] |
|---|
| | | - `ignore_init_mismatch`: Ignore size mismatch when loading pre-trained model |
|---|
| | | - `use_lora`: Fine-tuning model use lora, more detail please refer to [LORA](https://arxiv.org/pdf/2106.09685.pdf) |
|---|
| | | - `init_param`: `[]`(Default), init model path, load modelscope model initialization by default. For example: ["checkpoint/20epoch.pb"] |
|---|
| | | - `freeze_param`: `[]`(Default), Freeze model parameters. For example：["encoder"] |
|---|
| | | - `ignore_init_mismatch`: `True`(Default), Ignore size mismatch when loading pre-trained model |
|---|
| | | - `use_lora`: `False`(Default), Fine-tuning model use lora, more detail please refer to [LORA](https://arxiv.org/pdf/2106.09685.pdf) |
|---|
| | | |
|---|
| | | - Training data formats： |
|---|
| | | ```sh |
|---|
