python/FunASR-XL.git

			@@ -14,6 +14,7 @@
			\| [News](https://github.com/alibaba-damo-academy/FunASR#whats-new)
			\| [Installation](#installation)
			\| [Quick Start](#quick-start)
			\| [Tutorial](https://github.com/alibaba-damo-academy/FunASR/blob/main/docs/tutorial/README.md)
			\| [Runtime](./runtime/readme.md)
			\| [Model Zoo](#model-zoo)
			\| [Contact](#contact)
			@@ -27,6 +28,8 @@

			<a name="whats-new"></a>
			## What's new:
			- 2024/05/15：emotion recognition models are new supported. [emotion2vec+large](https://modelscope.cn/models/iic/emotion2vec_plus_large/summary)，[emotion2vec+base](https://modelscope.cn/models/iic/emotion2vec_plus_base/summary)，[emotion2vec+seed](https://modelscope.cn/models/iic/emotion2vec_plus_seed/summary). currently supports the following categories: 0: angry 1: happy 2: neutral 3: sad 4: unknown.
			- 2024/05/15: Offline File Transcription Service 4.5, Offline File Transcription Service of English 1.6，Real-time Transcription Service 1.10 released，adapting to FunASR 1.0 model structure；([docs](runtime/readme.md))
			- 2024/03/05：Added the Qwen-Audio and Qwen-Audio-Chat large-scale audio-text multimodal models, which have topped multiple audio domain leaderboards. These models support speech dialogue, [usage](examples/industrial_data_pretraining/qwen_audio).
			- 2024/03/05：Added support for the Whisper-large-v3 model, a multitasking model that can perform multilingual speech recognition, speech translation, and language identification. It can be downloaded from the[modelscope](examples/industrial_data_pretraining/whisper/demo.py), and [openai](examples/industrial_data_pretraining/whisper/demo_from_openai.py).
			- 2024/03/05: Offline File Transcription Service 4.4, Offline File Transcription Service of English 1.5，Real-time Transcription Service 1.9 released，docker image supports ARM64 platform, update modelscope；([docs](runtime/readme.md))
			@@ -67,7 +70,7 @@
			```

			## Model Zoo
			FunASR has open-sourced a large number of pre-trained models on industrial data. You are free to use, copy, modify, and share FunASR models under the [Model License Agreement](./MODEL_LICENSE). Below are some representative models, for more models please refer to the [Model Zoo]().
			FunASR has open-sourced a large number of pre-trained models on industrial data. You are free to use, copy, modify, and share FunASR models under the [Model License Agreement](./MODEL_LICENSE). Below are some representative models, for more models please refer to the [Model Zoo](./model_zoo).

			(Note: ⭐ represents the ModelScope model zoo, 🤗 represents the Huggingface model zoo, 🍀 represents the OpenAI model zoo)

			@@ -82,10 +85,11 @@
			\| fsmn-vad <br> ( [⭐](https://modelscope.cn/models/damo/speech_fsmn_vad_zh-cn-16k-common-pytorch/summary) [🤗](https://huggingface.co/funasr/fsmn-vad) ) \| voice activity detection \| 5000 hours, Mandarin and English \| 0.4M \|
			\| fa-zh <br> ( [⭐](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) [🤗](https://huggingface.co/funasr/fa-zh) ) \| timestamp prediction \| 5000 hours, Mandarin \| 38M \|
			\| cam++ <br> ( [⭐](https://modelscope.cn/models/iic/speech_campplus_sv_zh-cn_16k-common/summary) [🤗](https://huggingface.co/funasr/campplus) ) \| speaker verification/diarization \| 5000 hours \| 7.2M \|
			\| Whisper-large-v2 <br> ([⭐](https://www.modelscope.cn/models/iic/speech_whisper-large_asr_multilingual/summary) [🍀](https://github.com/openai/whisper) ) \| speech recognition, with timestamps, non-streaming \| multilingual \| 1.5G \|
			\| Whisper-large-v3 <br> ([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3/summary) [🍀](https://github.com/openai/whisper) ) \| speech recognition, with timestamps, non-streaming \| multilingual \| 1.5G \|
			\| Qwen-Audio <br> ([⭐](examples/industrial_data_pretraining/qwen_audio/demo.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio) ) \| audio-text multimodal models (pretraining) \| multilingual \| 8B \|
			\| Qwen-Audio-Chat <br> ([⭐](examples/industrial_data_pretraining/qwen_audio/demo_chat.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio-Chat) ) \| audio-text multimodal models (chat) \| multilingual \| 8B \|
			\| Whisper-large-v2 <br> ([⭐](https://www.modelscope.cn/models/iic/speech_whisper-large_asr_multilingual/summary) [🍀](https://github.com/openai/whisper) ) \| speech recognition, with timestamps, non-streaming \| multilingual \| 1550 M \|
			\| Whisper-large-v3 <br> ([⭐](https://www.modelscope.cn/models/iic/Whisper-large-v3/summary) [🍀](https://github.com/openai/whisper) ) \| speech recognition, with timestamps, non-streaming \| multilingual \| 1550 M \|
			\| Qwen-Audio <br> ([⭐](examples/industrial_data_pretraining/qwen_audio/demo.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio) ) \| audio-text multimodal models (pretraining) \| multilingual \| 8B \|
			\| Qwen-Audio-Chat <br> ([⭐](examples/industrial_data_pretraining/qwen_audio/demo_chat.py) [🤗](https://huggingface.co/Qwen/Qwen-Audio-Chat) ) \| audio-text multimodal models (chat) \| multilingual \| 8B \|
			\| emotion2vec+large <br> ([⭐](https://modelscope.cn/models/iic/emotion2vec_plus_large/summary) [🤗](https://huggingface.co/emotion2vec/emotion2vec_plus_large) ) \| speech emotion recongintion \| 40000 hours \| 300M \|



			@@ -97,7 +101,7 @@
			<a name="quick-start"></a>
			## Quick Start

			Below is a quick start tutorial. Test audio files ([Mandarin](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav), [English]()).
			Below is a quick start tutorial. Test audio files ([Mandarin](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/vad_example.wav), [English](https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_en.wav)).

			### Command-line usage

			@@ -112,7 +116,7 @@
			from funasr import AutoModel
			# paraformer-zh is a multi-functional asr model
			# use vad, punc, spk or not as you need
			model = AutoModel(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc-c",
			model = AutoModel(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc",
			# spk_model="cam++",
			)
			res = model.generate(input=f"{model.model_path}/example/asr_example.wav",
			@@ -154,7 +158,7 @@
			from funasr import AutoModel

			model = AutoModel(model="fsmn-vad")
			wav_file = f"{model.model_path}/example/asr_example.wav"
			wav_file = f"{model.model_path}/example/vad_example.wav"
			res = model.generate(input=wav_file)
			print(res)
			```
			@@ -207,10 +211,40 @@
			res = model.generate(input=(wav_file, text_file), data_type=("sound", "text"))
			print(res)
			```
			More usages ref to [docs](docs/tutorial/README_zh.md),
			more examples ref to [demo](https://github.com/alibaba-damo-academy/FunASR/tree/main/examples/industrial_data_pretraining)

			More examples ref to [docs](https://github.com/alibaba-damo-academy/FunASR/tree/main/examples/industrial_data_pretraining)

			[//]: # (FunASR supports inference and fine-tuning of models trained on industrial datasets of tens of thousands of hours. For more details, please refer to ([modelscope_egs](https://alibaba-damo-academy.github.io/FunASR/en/modelscope_pipeline/quick_start.html)). It also supports training and fine-tuning of models on academic standard datasets. For more details, please refer to([egs](https://alibaba-damo-academy.github.io/FunASR/en/academic_recipe/asr_recipe.html)). The models include speech recognition (ASR), speech activity detection (VAD), punctuation recovery, language model, speaker verification, speaker separation, and multi-party conversation speech recognition. For a detailed list of models, please refer to the [Model Zoo](https://github.com/alibaba-damo-academy/FunASR/blob/main/docs/model_zoo/modelscope_models.md):)
			## Export ONNX

			### Command-line usage
			```shell
			funasr-export ++model=paraformer ++quantize=false ++device=cpu
			```

			### Python
			```python
			from funasr import AutoModel

			model = AutoModel(model="paraformer", device="cpu")

			res = model.export(quantize=False)
			```

			### Test ONNX
			```python
			# pip3 install -U funasr-onnx
			from funasr_onnx import Paraformer
			model_dir = "damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch"
			model = Paraformer(model_dir, batch_size=1, quantize=True)

			wav_path = ['~/.cache/modelscope/hub/damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch/example/asr_example.wav']

			result = model(wav_path)
			print(result)
			```

			More examples ref to [demo](runtime/python/onnxruntime)

			## Deployment Service
			FunASR supports deploying pre-trained or further fine-tuned models for service. Currently, it supports the following types of service deployment:
			@@ -229,9 +263,9 @@

			You can also scan the following DingTalk group or WeChat group QR code to join the community group for communication and discussion.

			\|DingTalk group \| WeChat group \|
			\|:---:\|:-----------------------------------------------------:\|
			\|<div align="left"><img src="docs/images/dingding.jpg" width="250"/> \| <img src="docs/images/wechat.png" width="215"/></div> \|
			\| DingTalk group \| WeChat group \|
			\|:-------------------------------------------------------------------:\|:-----------------------------------------------------:\|
			\| <div align="left"><img src="docs/images/dingding.png" width="250"/> \| <img src="docs/images/wechat.png" width="215"/></div> \|

			## Contributors