FUNASR训练
blame | 历史 | 补丁 | 提交 | 提交对比 | ignore whitespace
zhifu gao
2023-04-24 754fd0b1ff704411731d32457e802c07f3cdf29b 
 README.md


@@ -1,69 +1,73 @@


<div align="left"><img src="docs/images/funasr_logo.jpg" width="400"/></div>


[//]: # (<div align="left"><img src="docs/images/funasr_logo.jpg" width="400"/></div>)




# FunASR: A Fundamental End-to-End Speech Recognition Toolkit


<p align="left">


    <a href=""><img src="https://img.shields.io/badge/OS-Linux%2C%20Win%2C%20Mac-brightgreen.svg"></a>


    <a href=""><img src="https://img.shields.io/badge/Python->=3.7,<=3.10-aff.svg"></a>


    <a href=""><img src="https://img.shields.io/badge/Pytorch-%3E%3D1.11-blue"></a>


</p>




<strong>FunASR</strong> hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model released on [ModelScope](https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition), researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun！[Model Zoo](docs/modelscope_models.md)


<strong>FunASR</strong> hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model released on [ModelScope](https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition), researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun！




## Release Notes: 


### 2023.1.16, funasr-0.1.6


- We release a new version model [Paraformer-large-long](https://modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary), which integrate the [VAD](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) model, [ASR](https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary),


 [Punctuation](https://www.modelscope.cn/models/damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch/summary) model and timestamp together. The model could take in several hours long inputs.


- We release a new type model, [VAD](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary), which could predict the duration of none-silence speech. It could be freely integrated with any ASR models in [Model Zoo](docs/modelscope_models.md).


- We release a new type model, [Punctuation](https://www.modelscope.cn/models/damo/punc_ct-transformer_zh-cn-common-vocab272727-pytorch/summary), which could predict the punctuation of ASR models's results. It could be freely integrated with any ASR models in [Model Zoo](docs/modelscope_models.md).


- We release a new model, [Data2vec](https://www.modelscope.cn/models/damo/speech_data2vec_pretrain-zh-cn-aishell2-16k-pytorch/summary), an unsupervised pretraining model which could be finetuned on ASR and other downstream tasks.


- We release a new model, [Paraformer-Tiny](https://www.modelscope.cn/models/damo/speech_paraformer-tiny-commandword_asr_nat-zh-cn-16k-vocab544-pytorch/summary), a lightweight Paraformer model which supports Mandarin command words recognition.


- We release a new type model, [SV](https://www.modelscope.cn/models/damo/speech_xvector_sv-zh-cn-cnceleb-16k-spk3465-pytorch/summary), which could extract speaker embeddings and further perform speaker verification on paired utterances. It will be supported for speaker diarization in the future version.


- We improve the pipeline of modelscope to speedup the inference, by integrating the process of build model into build pipeline.


- Various new types of audio input types are now supported by modelscope inference pipeline, including wav.scp, wav format, audio bytes, wave samples...


[**News**](https://github.com/alibaba-damo-academy/FunASR#whats-new) 


| [**Highlights**](#highlights)


| [**Installation**](#installation)


| [**Docs**](https://alibaba-damo-academy.github.io/FunASR/en/index.html)


| [**Tutorial**](https://github.com/alibaba-damo-academy/FunASR/wiki#funasr%E7%94%A8%E6%88%B7%E6%89%8B%E5%86%8C)


| [**Papers**](https://github.com/alibaba-damo-academy/FunASR#citations)


| [**Runtime**](https://github.com/alibaba-damo-academy/FunASR/tree/main/funasr/runtime)


| [**Model Zoo**](https://github.com/alibaba-damo-academy/FunASR/blob/main/docs/modelscope_models.md)


| [**Contact**](#contact)


|


[**M2MET2.0 Guidence_CN**](https://alibaba-damo-academy.github.io/FunASR/m2met2_cn/index.html)


| [**M2MET2.0 Guidence_EN**](https://alibaba-damo-academy.github.io/FunASR/m2met2/index.html)




## Key Features


- Many types of typical models are supported, e.g., [Tranformer](https://arxiv.org/abs/1706.03762), [Conformer](https://arxiv.org/abs/2005.08100), [Paraformer](https://arxiv.org/abs/2206.08317).


## Multi-Channel Multi-Party Meeting Transcription 2.0 (M2MET2.0) Challenge


We are pleased to announce that the M2MeT2.0 challenge will be held in the near future. The baseline system is conducted on FunASR and is provided as a receipe of AliMeeting corpus. For more details you can see the guidence of M2MET2.0 ([CN](https://alibaba-damo-academy.github.io/FunASR/m2met2_cn/index.html)/[EN](https://alibaba-damo-academy.github.io/FunASR/m2met2/index.html)).


## What's new: 




For the release notes, please ref to [news](https://github.com/alibaba-damo-academy/FunASR/releases)




## Highlights


- FunASR supports speech recognition(ASR), Multi-talker ASR, Voice Activity Detection(VAD), Punctuation Restoration, Language Models, Speaker Verification and Speaker diarization.   


- We have released large number of academic and industrial pretrained models on [ModelScope](https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition)


- The pretrained model [Paraformer-large](https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) obtains the best performance on many tasks in [SpeechIO leaderboard](https://github.com/SpeechColab/Leaderboard)


- FunASR supplies a easy-to-use pipeline to finetune pretrained models from [ModelScope](https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition)


- Compared to [Espnet](https://github.com/espnet/espnet) framework, the training speed of large-scale datasets in FunASR is much faster owning to the optimized dataloader.




## Installation(Training and Developing)


## Installation




- Install Conda:


``` sh


wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh


sh Miniconda3-latest-Linux-x86_64.sh


source ~/.bashrc


conda create -n funasr python=3.7


conda activate funasr


Install from pip


```shell


pip install -U funasr


# For the users in China, you could install with the command:


# pip install -U funasr -i https://mirror.sjtu.edu.cn/pypi/web/simple


```




- Install Pytorch (version >= 1.7.0): 


``` sh


pip3 install torch torchvision torchaudio


```


For more versions, please see [https://pytorch.org/get-started/locally](https://pytorch.org/get-started/locally)


Or install from source code






If you are in the area of China, you could set the source to speed the downloading.




``` sh


pip config set global.index-url https://mirror.sjtu.edu.cn/pypi/web/simple


```




- Install ModelScope:


``` sh


pip install "modelscope[audio]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html


```




For more details about modelscope, please see [modelscope installation](https://modelscope.cn/docs/%E7%8E%AF%E5%A2%83%E5%AE%89%E8%A3%85)




- Install FunASR and other packages: 




``` sh


git clone https://github.com/alibaba/FunASR.git && cd FunASR


pip install --editable ./


pip install -e ./


# For the users in China, you could install with the command:


# pip install -e ./ -i https://mirror.sjtu.edu.cn/pypi/web/simple




```


If you want to use the pretrained models in ModelScope, you should install the modelscope:




```shell


pip install -U modelscope


# For the users in China, you could install with the command:


# pip install -U modelscope -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html -i https://mirror.sjtu.edu.cn/pypi/web/simple


```




## Pretrained Model Zoo


For more details, please ref to [installation](https://github.com/alibaba-damo-academy/FunASR/wiki)




We have trained many academic and industrial models, [model hub](docs/modelscope_models.md)


[//]: # ()


[//]: # (## Usage)




[//]: # (For users who are new to FunASR and ModelScope, please refer to FunASR Docs&#40;[CN]&#40;https://alibaba-damo-academy.github.io/FunASR/cn/index.html&#41; / [EN]&#40;https://alibaba-damo-academy.github.io/FunASR/en/index.html&#41;&#41;)




## Contact




@@ -71,15 +75,23 @@




- email: [funasr@list.alibaba-inc.com](funasr@list.alibaba-inc.com)




- Dingding group:


<div align="left"><img src="docs/images/dingding.jpg" width="250"/>!<img src="docs/images/wechat.png" width="222"/></div>


|Dingding group |                     Wechat group                      |


|:---:|:-----------------------------------------------------:|


|<div align="left"><img src="docs/images/dingding.jpg" width="250"/> | <img src="docs/images/wechat.png" width="232"/></div> |




## Contributors




| <div align="left"><img src="docs/images/damo.png" width="180"/> | <div align="left"><img src="docs/images/nwpu.png" width="260"/> | <img src="docs/images/China_Telecom.png" width="200"/> </div>  | <img src="docs/images/RapidAI.png" width="200"/> </div> | <img src="docs/images/DeepScience.png" width="200"/> </div> |


|:---------------------------------------------------------------:|:---------------------------------------------------------------:|:--------------------------------------------------------------:|:-------------------------------------------------------:|:-----------------------------------------------------------:|




## Acknowledge




1. We borrowed a lot of code from [Kaldi](http://kaldi-asr.org/) for data preparation.


2. We borrowed a lot of code from [ESPnet](https://github.com/espnet/espnet). FunASR follows up the training and finetuning pipelines of ESPnet.


3. We referred [Wenet](https://github.com/wenet-e2e/wenet) for building dataloader for large scale data training.


4. We acknowledge [ChinaTelecom](https://github.com/zhuzizyf/damo-fsmn-vad-infer-httpserver) for contributing the VAD runtime. 


5. We acknowledge [RapidAI](https://github.com/RapidAI) for contributing the Paraformer and CT_Transformer-punc runtime.


6. We acknowledge [DeepScience](https://www.deepscience.cn) for contributing the grpc service.




## License


This project is licensed under the [The MIT License](https://opensource.org/licenses/MIT). FunASR also contains various third-party components and some code modified from other repos under other open source licenses.


@@ -87,17 +99,22 @@


## Citations




``` bibtex


@inproceedings{gao2022paraformer,


  title={Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition},


  author={Gao, Zhifu and Zhang, Shiliang and McLoughlin, Ian and Yan, Zhijie},


  booktitle={INTERSPEECH},


  year={2022}


}


@inproceedings{gao2020universal,


  title={Universal ASR: Unifying Streaming and Non-Streaming ASR Using a Single Encoder-Decoder Model},


  author={Gao, Zhifu and Zhang, Shiliang and Lei, Ming and McLoughlin, Ian},


  booktitle={arXiv preprint arXiv:2010.14099},


  year={2020}


}




@inproceedings{gao2022paraformer,


  title={Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition},


  author={Gao, Zhifu and Zhang, Shiliang and McLoughlin, Ian and Yan, Zhijie},


  booktitle={INTERSPEECH},


  year={2022}


@inproceedings{Shi2023AchievingTP,


  title={Achieving Timestamp Prediction While Recognizing with Non-Autoregressive End-to-End ASR Model},


  author={Xian Shi and Yanni Chen and Shiliang Zhang and Zhijie Yan},


  booktitle={arXiv preprint arXiv:2301.12343}


  year={2023}


}


```