python/FunASR-XL.git

			@@ -9,31 +9,32 @@
			<a href=""><img src="https://img.shields.io/badge/Pytorch-%3E%3D1.11-blue"></a>
			</p>

			<strong>FunASR</strong> hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model released on [ModelScope](https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition), researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun！
			<strong>FunASR</strong> hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model, researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun！

			[Highlights](#highlights)
			\| [News](https://github.com/alibaba-damo-academy/FunASR#whats-new)
			\| [Installation](#installation)
			\| [Quick Start](#quick-start)
			\| [Runtime](./funasr/runtime/readme.md)
			\| [Model Zoo](./docs/model_zoo/modelscope_models.md)
			\| [Runtime](./runtime/readme.md)
			\| [Model Zoo](#model-zoo)
			\| [Contact](#contact)


			<a name="highlights"></a>
			## Highlights
			- FunASR is a fundamental speech recognition toolkit that offers a variety of features, including speech recognition (ASR), Voice Activity Detection (VAD), Punctuation Restoration, Language Models, Speaker Verification, Speaker Diarization and multi-talker ASR. FunASR provides convenient scripts and tutorials, supporting inference and fine-tuning of pre-trained models.
			- We have released a vast collection of academic and industrial pretrained models on the [ModelScope](https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition), which can be accessed through our [Model Zoo](https://github.com/alibaba-damo-academy/FunASR/blob/main/docs/model_zoo/modelscope_models.md). The representative [Paraformer-large](https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary), a non-autoregressive end-to-end speech recognition model, has the advantages of high accuracy, high efficiency, and convenient deployment, supporting the rapid construction of speech recognition services. For more details on service deployment, please refer to the [service deployment document](funasr/runtime/readme_cn.md).
			- We have released a vast collection of academic and industrial pretrained models on the [ModelScope](https://www.modelscope.cn/models?page=1&tasks=auto-speech-recognition) and [huggingface](https://huggingface.co/FunASR), which can be accessed through our [Model Zoo](https://github.com/alibaba-damo-academy/FunASR/blob/main/docs/model_zoo/modelscope_models.md). The representative [Paraformer-large](https://www.modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary), a non-autoregressive end-to-end speech recognition model, has the advantages of high accuracy, high efficiency, and convenient deployment, supporting the rapid construction of speech recognition services. For more details on service deployment, please refer to the [service deployment document](runtime/readme_cn.md).


			<a name="whats-new"></a>
			## What's new:
			- 2023/10/17: The offline file transcription service (CPU) of English has been released. For more details, please refer to ([Deployment documentation](funasr/runtime/docs/SDK_tutorial_en.md)).
			- 2023/11/08: The offline file transcription service 3.0 (CPU) of Mandarin has been released, adding punctuation large model, Ngram language model, and wfst hot words. For detailed information, please refer to [docs](runtime#file-transcription-service-mandarin-cpu).
			- 2023/10/17: The offline file transcription service (CPU) of English has been released. For more details, please refer to ([docs](runtime#file-transcription-service-english-cpu)).
			- 2023/10/13: [SlideSpeech](https://slidespeech.github.io/): A large scale multi-modal audio-visual corpus with a significant amount of real-time synchronized slides.
			- 2023/10/10: The ASR-SpeakersDiarization combined pipeline [Paraformer-VAD-SPK](https://github.com/alibaba-damo-academy/FunASR/blob/main/egs_modelscope/asr_vad_spk/speech_paraformer-large-vad-punc-spk_asr_nat-zh-cn/demo.py) is now released. Experience the model to get recognition results with speaker information.
			- 2023/10/07: [FunCodec](https://github.com/alibaba-damo-academy/FunCodec): A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec.
			- 2023/09/01: The offline file transcription service 2.0 (CPU) of Mandarin has been released, with added support for ffmpeg, timestamp, and hotword models. For more details, please refer to ([Deployment documentation](funasr/runtime/docs/SDK_tutorial.md)).
			- 2023/08/07: The real-time transcription service (CPU) of Mandarin has been released. For more details, please refer to ([Deployment documentation](funasr/runtime/docs/SDK_tutorial_online.md)).
			- 2023/09/01: The offline file transcription service 2.0 (CPU) of Mandarin has been released, with added support for ffmpeg, timestamp, and hotword models. For more details, please refer to ([docs](runtime#file-transcription-service-mandarin-cpu)).
			- 2023/08/07: The real-time transcription service (CPU) of Mandarin has been released. For more details, please refer to ([docs](runtime#the-real-time-transcription-service-mandarin-cpu)).
			- 2023/07/17: BAT is released, which is a low-latency and low-memory-consumption RNN-T model. For more details, please refer to ([BAT](egs/aishell/bat)).
			- 2023/06/26: ASRU2023 Multi-Channel Multi-Party Meeting Transcription Challenge 2.0 completed the competition and announced the results. For more details, please refer to ([M2MeT2.0](https://alibaba-damo-academy.github.io/FunASR/m2met2/index.html)).

			@@ -43,19 +44,98 @@

			Please ref to [installation docs](https://alibaba-damo-academy.github.io/FunASR/en/installation/installation.html)

			## Deployment Service
			## Model Zoo
			FunASR has open-sourced a large number of pre-trained models on industrial data. You are free to use, copy, modify, and share FunASR models under the [Model License Agreement](./MODEL_LICENSE). Below are some representative models, for more models please refer to the [Model Zoo]().

			FunASR supports pre-trained or further fine-tuned models for deployment as a service. The CPU version of the Chinese offline file conversion service has been released, details can be found in [docs](funasr/runtime/docs/SDK_tutorial.md). More detailed information about service deployment can be found in the [deployment roadmap](funasr/runtime/readme_cn.md).
			(Note: 🤗 represents the Huggingface model zoo link, ⭐ represents the ModelScope model zoo link)


			\| Model Name \| Task Details \| Training Date \| Parameters \|
			\|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------:\|:---------------------------------------------------------------------------:\|:--------------------------------:\|:----------:\|
			\| <nobr>paraformer-zh ([⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch/summary) [🤗]() )</nobr> \| speech recognition, with timestamps, non-streaming \| 60000 hours, Mandarin \| 220M \|
			\| <nobr>paraformer-zh-spk ( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large-vad-punc-spk_asr_nat-zh-cn/summary) [🤗]() )</nobr> \| speech recognition with speaker diarization, with timestamps, non-streaming \| 60000 hours, Mandarin \| 220M \|
			\| <nobr>paraformer-zh-online ( [⭐](https://modelscope.cn/models/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online/summary) [🤗]() )</nobr> \| speech recognition, streaming \| 60000 hours, Mandarin \| 220M \|
			\| <nobr>paraformer-en ( [⭐](https://www.modelscope.cn/models/damo/speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020/summary) [🤗]() )</nobr> \| speech recognition, with timestamps, non-streaming \| 50000 hours, English \| 220M \|
			\| <nobr>paraformer-en-spk ([🤗]() [⭐]() )</nobr> \| speech recognition with speaker diarization, non-streaming \| 50000 hours, English \| 220M \|
			\| <nobr>conformer-en ( [⭐](https://modelscope.cn/models/damo/speech_conformer_asr-en-16k-vocab4199-pytorch/summary) [🤗]() )</nobr> \| speech recognition, non-streaming \| 50000 hours, English \| 220M \|
			\| <nobr>ct-punc ( [⭐](https://modelscope.cn/models/damo/punc_ct-transformer_cn-en-common-vocab471067-large/summary) [🤗]() )</nobr> \| punctuation restoration \| 100M, Mandarin and English \| 1.1G \|
			\| <nobr>fsmn-vad ( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗]() )</nobr> \| voice activity detection \| 5000 hours, Mandarin and English \| 0.4M \|
			\| <nobr>fa-zh ( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗]() )</nobr> \| timestamp prediction \| 5000 hours, Mandarin \| 38M \|




			[//]: # ()
			[//]: # (FunASR supports pre-trained or further fine-tuned models for deployment as a service. The CPU version of the Chinese offline file conversion service has been released, details can be found in [docs](funasr/runtime/docs/SDK_tutorial.md). More detailed information about service deployment can be found in the [deployment roadmap](funasr/runtime/readme_cn.md).)


			<a name="quick-start"></a>
			## Quick Start
			Quick start for new users（[tutorial](https://alibaba-damo-academy.github.io/FunASR/en/funasr/quick_start.html)）

			FunASR supports inference and fine-tuning of models trained on industrial data for tens of thousands of hours. For more details, please refer to [modelscope_egs](https://alibaba-damo-academy.github.io/FunASR/en/modelscope_pipeline/quick_start.html). It also supports training and fine-tuning of models on academic standard datasets. For more information, please refer to [egs](https://alibaba-damo-academy.github.io/FunASR/en/academic_recipe/asr_recipe.html).

			FunASR supports inference and fine-tuning of models trained on industrial datasets of tens of thousands of hours. For more details, please refer to ([modelscope_egs](https://alibaba-damo-academy.github.io/FunASR/en/modelscope_pipeline/quick_start.html)). It also supports training and fine-tuning of models on academic standard datasets. For more details, please refer to([egs](https://alibaba-damo-academy.github.io/FunASR/en/academic_recipe/asr_recipe.html)). The models include speech recognition (ASR), speech activity detection (VAD), punctuation recovery, language model, speaker verification, speaker separation, and multi-party conversation speech recognition. For a detailed list of models, please refer to the [Model Zoo](https://github.com/alibaba-damo-academy/FunASR/blob/main/docs/model_zoo/modelscope_models.md):
			Below is a quick start tutorial. Test audio files ([Mandarin](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav), [English]()).

			<a name="Community Communication"></a>
			### Command-line usage

			```shell
			funasr --model paraformer-zh asr_example_zh.wav
			```

			Notes: Support recognition of single audio file, as well as file list in Kaldi-style wav.scp format: `wav_id wav_pat`

			### Speech Recognition (Non-streaming)
			```python
			from funasr import infer

			p = infer(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc", model_hub="ms")

			res = p("asr_example_zh.wav", batch_size_token=5000)
			print(res)
			```
			Note: `model_hub`: represents the model repository, `ms` stands for selecting ModelScope download, `hf` stands for selecting Huggingface download.

			### Speech Recognition (Streaming)
			```python
			from funasr import infer

			p = infer(model="paraformer-zh-streaming", model_hub="ms")

			chunk_size = [0, 10, 5] #[0, 10, 5] 600ms, [0, 8, 4] 480ms
			param_dict = {"cache": dict(), "is_final": False, "chunk_size": chunk_size, "encoder_chunk_look_back": 4, "decoder_chunk_look_back": 1}

			import torchaudio
			speech = torchaudio.load("asr_example_zh.wav")[0][0]
			speech_length = speech.shape[0]

			stride_size = chunk_size[1] * 960
			sample_offset = 0
			for sample_offset in range(0, speech_length, min(stride_size, speech_length - sample_offset)):
			param_dict["is_final"] = True if sample_offset + stride_size >= speech_length - 1 else False
			input = speech[sample_offset: sample_offset + stride_size]
			rec_result = p(input=input, param_dict=param_dict)
			print(rec_result)
			```
			Note: `chunk_size` is the configuration for streaming latency.` [0,10,5]` indicates that the real-time display granularity is `1060=600ms`, and the lookahead information is `560=300ms`. Each inference input is `600ms` (sample points are `16000*0.6=960`), and the output is the corresponding text. For the last speech segment input, `is_final=True` needs to be set to force the output of the last word.

			Quick start for new users can be found in [docs](https://alibaba-damo-academy.github.io/FunASR/en/funasr/quick_start_zh.html)


			[//]: # (FunASR supports inference and fine-tuning of models trained on industrial datasets of tens of thousands of hours. For more details, please refer to ([modelscope_egs](https://alibaba-damo-academy.github.io/FunASR/en/modelscope_pipeline/quick_start.html)). It also supports training and fine-tuning of models on academic standard datasets. For more details, please refer to([egs](https://alibaba-damo-academy.github.io/FunASR/en/academic_recipe/asr_recipe.html)). The models include speech recognition (ASR), speech activity detection (VAD), punctuation recovery, language model, speaker verification, speaker separation, and multi-party conversation speech recognition. For a detailed list of models, please refer to the [Model Zoo](https://github.com/alibaba-damo-academy/FunASR/blob/main/docs/model_zoo/modelscope_models.md):)

			## Deployment Service
			FunASR supports deploying pre-trained or further fine-tuned models for service. Currently, it supports the following types of service deployment:
			- File transcription service, Mandarin, CPU version, done
			- The real-time transcription service, Mandarin (CPU), done
			- File transcription service, English, CPU version, done
			- File transcription service, Mandarin, GPU version, in progress
			- and more.

			For more detailed information, please refer to the [service deployment documentation](runtime/readme.md).


			<a name="contact"></a>
			## Community Communication
			If you encounter problems in use, you can directly raise Issues on the github page.

			@@ -67,8 +147,8 @@

			## Contributors

			\| <div align="left"><img src="docs/images/damo.png" width="180"/> \| <div align="left"><img src="docs/images/nwpu.png" width="260"/> \| <img src="docs/images/China_Telecom.png" width="200"/> </div> \| <img src="docs/images/RapidAI.png" width="200"/> </div> \| <img src="docs/images/aihealthx.png" width="200"/> </div> \| <img src="docs/images/XVERSE.png" width="250"/> </div> \|
			\|:---------------------------------------------------------------:\|:---------------------------------------------------------------:\|:--------------------------------------------------------------:\|:-------------------------------------------------------:\|:-----------------------------------------------------------:\|:------------------------------------------------------:\|
			\| <div align="left"><img src="docs/images/nwpu.png" width="260"/> \| <img src="docs/images/China_Telecom.png" width="200"/> </div> \| <img src="docs/images/RapidAI.png" width="200"/> </div> \| <img src="docs/images/aihealthx.png" width="200"/> </div> \| <img src="docs/images/XVERSE.png" width="250"/> </div> \|
			\|:---------------------------------------------------------------:\|:--------------------------------------------------------------:\|:-------------------------------------------------------:\|:-----------------------------------------------------------:\|:------------------------------------------------------:\|

			The contributors can be found in [contributors list](./Acknowledge.md)

			@@ -90,12 +170,6 @@
			title={BAT: Boundary aware transducer for memory-efficient and low-latency ASR},
			year={2023},
			booktitle={INTERSPEECH},
			}
			@inproceedings{wang2023told,
			author={Jiaming Wang and Zhihao Du and Shiliang Zhang},
			title={{TOLD:} {A} Novel Two-Stage Overlap-Aware Framework for Speaker Diarization},
			year={2023},
			booktitle={ICASSP},
			}
			@inproceedings{gao22b_interspeech,
			author={Zhifu Gao and ShiLiang Zhang and Ian McLoughlin and Zhijie Yan},