FUNASR训练
blame | 历史 | 补丁 | 提交 | 提交对比 | ignore whitespace
游雁
2024-01-30 f1c1cb0773fca5e9d1ee595ef6ca2ff4bad9f2a4 
 README.md


@@ -3,11 +3,10 @@


([简体中文](./README_zh.md)|English)




# FunASR: A Fundamental End-to-End Speech Recognition Toolkit


<p align="left">


    <a href=""><img src="https://img.shields.io/badge/OS-Linux%2C%20Win%2C%20Mac-brightgreen.svg"></a>


    <a href=""><img src="https://img.shields.io/badge/Python->=3.7,<=3.10-aff.svg"></a>


    <a href=""><img src="https://img.shields.io/badge/Pytorch-%3E%3D1.11-blue"></a>


</p>






[![PyPI](https://img.shields.io/pypi/v/funasr)](https://pypi.org/project/funasr/)






<strong>FunASR</strong> hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model, researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun！




@@ -28,6 +27,8 @@




<a name="whats-new"></a>


## What's new:


- 2024/01/30：funasr-1.0 has been released ([docs](https://github.com/alibaba-damo-academy/FunASR/discussions/1319))


- 2024/01/25: Offline File Transcription Service 4.2, Offline File Transcription Service of English 1.3 released，optimized the VAD (Voice Activity Detection) data processing method, significantly reducing peak memory usage, memory leak optimization; Real-time Transcription Service 1.7 released，optimizatized the client-side；([docs](runtime/readme.md))


- 2024/01/09: The Funasr SDK for Windows version 2.0 has been released, featuring support for The offline file transcription service (CPU) of Mandarin 4.1, The offline file transcription service (CPU) of English 1.2, The real-time transcription service (CPU) of Mandarin 1.6. For more details, please refer to the official documentation or release notes([FunASR-Runtime-Windows](https://www.modelscope.cn/models/damo/funasr-runtime-win-cpu-x64/summary))


- 2024/01/03: File Transcription Service 4.0 released, Added support for 8k models, optimized timestamp mismatch issues and added sentence-level timestamps, improved the effectiveness of English word FST hotwords, supported automated configuration of thread parameters, and fixed known crash issues as well as memory leak problems, refer to ([docs](runtime/readme.md#file-transcription-service-mandarin-cpu)).


- 2024/01/03: Real-time Transcription Service 1.6 released，The 2pass-offline mode supports Ngram language model decoding and WFST hotwords, while also addressing known crash issues and memory leak problems, ([docs](runtime/readme.md#the-real-time-transcription-service-mandarin-cpu))


@@ -47,7 +48,19 @@


<a name="Installation"></a>


## Installation




Please ref to [installation docs](https://alibaba-damo-academy.github.io/FunASR/en/installation/installation.html)


```shell


pip3 install -U funasr


```


Or install from source code


``` sh


git clone https://github.com/alibaba/FunASR.git && cd FunASR


pip3 install -e ./


```


Install modelscope for the pretrained models (Optional)




```shell


pip3 install -U modelscope


```




## Model Zoo


FunASR has open-sourced a large number of pre-trained models on industrial data. You are free to use, copy, modify, and share FunASR models under the [Model License Agreement](./MODEL_LICENSE). Below are some representative models, for more models please refer to the [Model Zoo]().


@@ -55,16 +68,16 @@


(Note: 🤗 represents the Huggingface model zoo link, ⭐ represents the ModelScope model zoo link)






|                                                                             Model Name                                                                             |                                Task Details                                 |          Training Data           | Parameters |


|:------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------:|:--------------------------------:|:----------:|


|    paraformer-zh <br> ([⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary)  [🤗]() )    |             speech recognition, with timestamps, non-streaming              |      60000 hours, Mandarin       |    220M    |


|                paraformer-zh-spk <br> ( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large-vad-punc-spk_asr_nat-zh-cn/summary)  [🤗]() )                | speech recognition with speaker diarization, with timestamps, non-streaming |      60000 hours, Mandarin       |    220M    |


| <nobr>paraformer-zh-online <br> ( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗]() )</nobr> |                        speech recognition, streaming                        |      60000 hours, Mandarin       |    220M    |


|         paraformer-en <br> ( [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020/summary) [🤗]() )         |             speech recognition, with timestamps, non-streaming              |       50000 hours, English       |    220M    |


|                     conformer-en <br> ( [⭐](https://modelscope.cn/models/damo/speech_conformer_asr-en-16k-vocab4199-pytorch/summary) [🤗]() )                      |                      speech recognition, non-streaming                      |       50000 hours, English       |    220M    |


|                     ct-punc <br> ( [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗]() )                      |                           punctuation restoration                           |    100M, Mandarin and English    |    1.1G    | 


|                          fsmn-vad <br> ( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗]() )                          |                          voice activity detection                           | 5000 hours, Mandarin and English |    0.4M    | 


|                          fa-zh <br> ( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗]() )                           |                            timestamp prediction                             |       5000 hours, Mandarin       |    38M     | 


|                                                                             Model Name                                                                             |                    Task Details                    |          Training Data           | Parameters |


|:------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:--------------------------------------------------:|:--------------------------------:|:----------:|


|    paraformer-zh <br> ([⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary)  [🤗]() )    | speech recognition, with timestamps, non-streaming |      60000 hours, Mandarin       |    220M    |


| <nobr>paraformer-zh-online <br> ( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗]() )</nobr> |           speech recognition, streaming            |      60000 hours, Mandarin       |    220M    |


|         paraformer-en <br> ( [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020/summary) [🤗]() )         | speech recognition, with timestamps, non-streaming |       50000 hours, English       |    220M    |


|                     conformer-en <br> ( [⭐](https://modelscope.cn/models/damo/speech_conformer_asr-en-16k-vocab4199-pytorch/summary) [🤗]() )                      |         speech recognition, non-streaming          |       50000 hours, English       |    220M    |


|                     ct-punc <br> ( [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗]() )                      |              punctuation restoration               |    100M, Mandarin and English    |    1.1G    | 


|                          fsmn-vad <br> ( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗]() )                          |              voice activity detection              | 5000 hours, Mandarin and English |    0.4M    | 


|                          fa-zh <br> ( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗]() )                           |                timestamp prediction                |       5000 hours, Mandarin       |    38M     | 


|                cam++ <br> ( [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗]() )                                             |        speaker verification/diarization            |            5000 hours            |    7.2M    | 








@@ -91,9 +104,9 @@


from funasr import AutoModel


# paraformer-zh is a multi-functional asr model


# use vad, punc, spk or not as you need


model = AutoModel(model="paraformer-zh", model_revision="v2.0.2",


                  vad_model="fsmn-vad", vad_model_revision="v2.0.2",


                  punc_model="ct-punc-c", punc_model_revision="v2.0.2",


model = AutoModel(model="paraformer-zh", model_revision="v2.0.4",


                  vad_model="fsmn-vad", vad_model_revision="v2.0.4",


                  punc_model="ct-punc-c", punc_model_revision="v2.0.4",


                  # spk_model="cam++", spk_model_revision="v2.0.2",


                  )


res = model.generate(input=f"{model.model_path}/example/asr_example.wav", 


@@ -111,7 +124,7 @@


encoder_chunk_look_back = 4 #number of chunks to lookback for encoder self-attention


decoder_chunk_look_back = 1 #number of encoder chunks to lookback for decoder cross-attention




model = AutoModel(model="paraformer-zh-streaming", model_revision="v2.0.2")


model = AutoModel(model="paraformer-zh-streaming", model_revision="v2.0.4")




import soundfile


import os


@@ -134,7 +147,7 @@


```python


from funasr import AutoModel




model = AutoModel(model="fsmn-vad", model_revision="v2.0.2")


model = AutoModel(model="fsmn-vad", model_revision="v2.0.4")


wav_file = f"{model.model_path}/example/asr_example.wav"


res = model.generate(input=wav_file)


print(res)


@@ -144,7 +157,7 @@


from funasr import AutoModel




chunk_size = 200 # ms


model = AutoModel(model="fsmn-vad", model_revision="v2.0.2")


model = AutoModel(model="fsmn-vad", model_revision="v2.0.4")




import soundfile




@@ -165,7 +178,7 @@


```python


from funasr import AutoModel




model = AutoModel(model="ct-punc", model_revision="v2.0.2")


model = AutoModel(model="ct-punc", model_revision="v2.0.4")


res = model.generate(input="那今天的会就到这里吧 happy new year 明年见")


print(res)


```


@@ -173,7 +186,7 @@


```python


from funasr import AutoModel




model = AutoModel(model="fa-zh", model_revision="v2.0.2")


model = AutoModel(model="fa-zh", model_revision="v2.0.4")


wav_file = f"{model.model_path}/example/asr_example.wav"


text_file = f"{model.model_path}/example/text.txt"


res = model.generate(input=(wav_file, text_file), data_type=("sound", "text"))