FUNASR训练
parent: ea64830e | 补丁 | 提交 | ignore whitespace
zhifu gao
2023-02-14 b3bfea34ade9cb640db11c041fea9e4c61169168 
Merge pull request #103 from alibaba-damo-academy/dev_lhn

fix persian text segment bug
1个文件已修改
2 ■■■■■ 已修改文件
funasr/datasets/preprocessor.py 2 ●●●●● 补丁 | 查看 | 原始文档 | blame | 历史 
 funasr/datasets/preprocessor.py


@@ -363,7 +363,7 @@


            if self.split_with_space:


                tokens = text.strip().split(" ")


                if self.seg_dict is not None:


                    tokens = forward_segment("".join(tokens).lower(), self.seg_dict)


                    tokens = forward_segment("".join(tokens), self.seg_dict)


                    tokens = seg_tokenize(tokens, self.seg_dict)


            else:


                tokens = self.tokenizer.text2tokens(text)