AttributeError: 'ChatGLMTokenizer' object has no attribute 'sp_tokenizer'

[DuFangY]

No description provided.

[JimmyJIA-02]

+1, is that solved?

[JimmyJIA-02]

alright I solved this just now, by reinstall 4.27.1 transformers, problem here in my environment can be solved.

[DuFangY]

alright I solved this just now, by reinstall 4.27.1 transformers, problem here in my environment can be solved.
I solved the problem by following your method, thank you!

[shituo123456]

4.27.1

thank you，solved

[dingqiangliu]

move line "self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens) " before "super().init(" at "init" of class ChatGLMTokenizer can solve this issue.

[Yancy456]

check if you are using ChatGLM1 but no ChatGLM2

[ovjust]

4.27.1 not run on my computer.

Exception has occurred: RuntimeError
Internal: D:\a\sentencepiece\sentencepiece\src\sentencepiece_processor.cc(1102) [model_proto->ParseFromArray(serialized.data(), serialized.size())]
File "C:\Users\Administrator.cache\huggingface\modules\transformers_modules\visualglm-6b-model-gitee\tokenization_chatglm.py", line 22, in init
self.sp.Load(model_path)
File "C:\Users\Administrator.cache\huggingface\modules\transformers_modules\visualglm-6b-model-gitee\tokenization_chatglm.py", line 64, in init
self.text_tokenizer = TextTokenizer(vocab_file)
File "C:\Users\Administrator.cache\huggingface\modules\transformers_modules\visualglm-6b-model-gitee\tokenization_chatglm.py", line 221, in init
self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens)
File "D:\1MyFiles\code\python\visualglm-6b-main\2test.py", line 4, in
tokenizer = AutoTokenizer.from_pretrained(r"D:\1MyFiles\code\python\visualglm-6b-model-gitee", trust_remote_code=True)
RuntimeError: Internal: D:\a\sentencepiece\sentencepiece\src\sentencepiece_processor.cc(1102) [model_proto->ParseFromArray(serialized.data(), serialized.size())]