From 4ab0de7e5fbb91751a817eda826c5f71b825be98 Mon Sep 17 00:00:00 2001 From: zhifu gao <zhifu.gzf@alibaba-inc.com> Date: 星期二, 25 四月 2023 15:19:59 +0800 Subject: [PATCH] Merge pull request #414 from alibaba-damo-academy/main --- egs_modelscope/tp/TEMPLATE/README.md | 102 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 102 insertions(+), 0 deletions(-) diff --git a/egs_modelscope/tp/TEMPLATE/README.md b/egs_modelscope/tp/TEMPLATE/README.md new file mode 100644 index 0000000..2678a7f --- /dev/null +++ b/egs_modelscope/tp/TEMPLATE/README.md @@ -0,0 +1,102 @@ +# Timestamp Prediction (FA) + +## Inference + +### Quick start +#### [Use TP-Aligner Model Simply](https://modelscope.cn/models/damo/speech_timestamp_prediction-v1-16k-offline/summary) +```python +from modelscope.pipelines import pipeline +from modelscope.utils.constant import Tasks + +inference_pipline = pipeline( + task=Tasks.speech_timestamp, + model='damo/speech_timestamp_prediction-v1-16k-offline', + output_dir=None) + +rec_result = inference_pipline( + audio_in='https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_timestamps.wav', + text_in='涓� 涓� 涓� 澶� 骞� 娲� 鍥� 瀹� 涓� 浠� 涔� 璺� 鍒� 瑗� 澶� 骞� 娲� 鏉� 浜� 鍛�',) +print(rec_result) +``` + +Timestamp pipeline can also be used after ASR pipeline to compose complete ASR function, ref to [demo](https://github.com/alibaba-damo-academy/FunASR/discussions/246). + + + +#### API-reference +##### Define pipeline +- `task`: `Tasks.speech_timestamp` +- `model`: model name in [model zoo](https://alibaba-damo-academy.github.io/FunASR/en/modelscope_models.html#pretrained-models-on-modelscope), or model path in local disk +- `ngpu`: `1` (Default), decoding on GPU. If ngpu=0, decoding on CPU +- `ncpu`: `1` (Default), sets the number of threads used for intraop parallelism on CPU +- `output_dir`: `None` (Default), the output path of results if set +- `batch_size`: `1` (Default), batch size when decoding +##### Infer pipeline +- `audio_in`: the input speech to predict, which could be: + - wav_path, `e.g.`: asr_example.wav (wav in local or url), + - wav.scp, kaldi style wav list (`wav_id wav_path`), `e.g.`: + ```text + asr_example1 ./audios/asr_example1.wav + asr_example2 ./audios/asr_example2.wav + ``` + In this case of `wav.scp` input, `output_dir` must be set to save the output results +- `text_in`: the input text to predict锛� splited by blank, which could be: + - text string, `e.g.`: `浠� 澶� 澶� 姘� 鎬� 涔� 鏍穈 + - text.scp, kaldi style text file (`wav_id transcription`), `e.g.`: + ```text + asr_example1 浠� 澶� 澶� 姘� 鎬� 涔� 鏍� + asr_example2 娆� 杩� 浣� 楠� 杈� 鎽� 闄� 璇� 闊� 璇� 鍒� 妯� 鍨� + ``` +- `audio_fs`: audio sampling rate, only set when audio_in is pcm audio +- `output_dir`: None (Default), the output path of results if set, containing + - output_dir/timestamp_prediction/tp_sync, timestamp in second containing silence periods, `wav_id# token1 start_time end_time;`, `e.g.`: + ```text + test_wav1# <sil> 0.000 0.500;娓� 0.500 0.680;宸� 0.680 0.840;鍖� 0.840 1.040;宸� 1.040 1.280;浠� 1.280 1.520;<sil> 1.520 1.680;搴� 1.680 1.920;<sil> 1.920 2.160;璧� 2.160 2.380;鐏� 2.380 2.580;娈� 2.580 2.760;鍙� 2.760 2.920;闄� 2.920 3.100;杩� 3.100 3.340;<sil> 3.340 3.400;娌� 3.400 3.640;<sil> 3.640 3.700;娴� 3.700 3.940;<sil> 3.940 4.240;澶� 4.240 4.400;閲� 4.400 4.520;姝� 4.520 4.680;楸� 4.680 4.920;<sil> 4.920 4.940;婕� 4.940 5.120;娴� 5.120 5.300;娌� 5.300 5.500;闈� 5.500 5.900;<sil> 5.900 6.240; + ``` + - output_dir/timestamp_prediction/tp_time, timestamp list in ms of same length as input text without silence `wav_id# [[start_time, end_time],]`, `e.g.`: + ```text + test_wav1# [[500, 680], [680, 840], [840, 1040], [1040, 1280], [1280, 1520], [1680, 1920], [2160, 2380], [2380, 2580], [2580, 2760], [2760, 2920], [2920, 3100], [3100, 3340], [3400, 3640], [3700, 3940], [4240, 4400], [4400, 4520], [4520, 4680], [4680, 4920], [4940, 5120], [5120, 5300], [5300, 5500], [5500, 5900]] + ``` + +### Inference with multi-thread CPUs or multi GPUs +FunASR also offer recipes [egs_modelscope/vad/TEMPLATE/infer.sh](https://github.com/alibaba-damo-academy/FunASR/blob/main/egs_modelscope/vad/TEMPLATE/infer.sh) to decode with multi-thread CPUs, or multi GPUs. + +- Setting parameters in `infer.sh` + - `model`: model name in [model zoo](https://alibaba-damo-academy.github.io/FunASR/en/modelscope_models.html#pretrained-models-on-modelscope), or model path in local disk + - `data_dir`: the dataset dir **must** include `wav.scp` and `text.scp` + - `output_dir`: output dir of the recognition results + - `batch_size`: `64` (Default), batch size of inference on gpu + - `gpu_inference`: `true` (Default), whether to perform gpu decoding, set false for CPU inference + - `gpuid_list`: `0,1` (Default), which gpu_ids are used to infer + - `njob`: only used for CPU inference (`gpu_inference`=`false`), `64` (Default), the number of jobs for CPU decoding + - `checkpoint_dir`: only used for infer finetuned models, the path dir of finetuned models + - `checkpoint_name`: only used for infer finetuned models, `valid.cer_ctc.ave.pb` (Default), which checkpoint is used to infer + +- Decode with multi GPUs: +```shell + bash infer.sh \ + --model "damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch" \ + --data_dir "./data/test" \ + --output_dir "./results" \ + --batch_size 64 \ + --gpu_inference true \ + --gpuid_list "0,1" +``` +- Decode with multi-thread CPUs: +```shell + bash infer.sh \ + --model "damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch" \ + --data_dir "./data/test" \ + --output_dir "./results" \ + --gpu_inference false \ + --njob 64 +``` + +## Finetune with pipeline + +### Quick start + +### Finetune with your data + +## Inference with your finetuned model + -- Gitblit v1.9.1