修复chunk_size可能传入str或list结构 (#1357)

dudulu

2024-02-05 9c5f00794e7c474d1079ff78e5c69e8c53a30e6e

1
{"bos_token": "<|endoftext|>", "eos_token": "<|endoftext|>", "unk_token": "<|endoftext|>"}