add check for chunk_size_vec

雾聪

2023-10-24 78e4f729be82b515480759addb64bd8d4e124dca

1
{"bos_token": "<|endoftext|>", "eos_token": "<|endoftext|>", "unk_token": "<|endoftext|>"}