From ff477cf1487a074d9febcf7472ecb397a83a8964 Mon Sep 17 00:00:00 2001 From: onlybetheone <iriszhangchong@gmail.com> Date: 星期四, 22 十二月 2022 17:30:13 +0800 Subject: [PATCH] FunTextProcessing --- fun_text_processing/README.md | 26 ++++++++++++++++++++++++-- 1 files changed, 24 insertions(+), 2 deletions(-) diff --git a/fun_text_processing/README.md b/fun_text_processing/README.md index 878601b..9778493 100644 --- a/fun_text_processing/README.md +++ b/fun_text_processing/README.md @@ -10,16 +10,38 @@ - FunTextProcessing supports inverse text processing (ITN), text processing (TN). - FunTextProcessing supports multilingual, 10+ languages for ITN, 5 languages for TN. +### Installation + +Fun Text Processing, specifically (Inverse) Text Normalization, requires Pynini to be installed. +``` +bash fun_text_processing/install_pynini.sh +``` ### Example #### Inverse Text Processing (ITN) Given text inputs, such as speech recognition results, use `fun_text_processing/inverse_text_normalization/inverse_normalize.py` to output ITN results. You may refer to the following example scripts. +- ITN example for English ``` -test_file=fun_text_processing/inverse_text_normalization/id/id_itn_test_input.txt +python fun_text_processing/inverse_text_normalization/inverse_normalize.py --text="one hundred twenty three" --language=en +``` -python fun_text_processing/inverse_text_normalization/inverse_normalize.py --input_file $test_file --cache_dir ./itn_model/ --output_file output.txt --language=id +- ITN example for Indonesian ``` +python fun_text_processing/inverse_text_normalization/inverse_normalize.py --text="ratus dua puluh tiga" --language=id --cache_dir ./model/ --output_file output.txt +cat output.txt +``` + +Arguments: +- text - Input text. Should not exceed 500 words. +- input_file - Input file with lines of input text. Only one of text or input_file is accepted. +- output_file - Output file to save normalizations. Needed if input_file is specified. +- language - language id. +- input_case - Only for text normalization. lower_cased or cased. +- verbose - Outputs intermediate information. +- cache_dir - Specifies a cache directory for compiled grammars. If grammars exist, this significantly improves speed. +- overwrite_cache - Updates grammars in cache. +- whitelist - TSV file with custom mappings of written text to spoken form. ### Acknowledge -- Gitblit v1.9.1