How to re-download tokenizer for huggingface?

I have the exact same problem as https://github.com/huggingface/transformers/issues/11243, except it only does not work in Jupyter lab. It does work in python in my shell.

I downloaded the cardiffnlp/twitter-roberta-base-emotion model using:

model_name = "cardiffnlp/twitter-roberta-base-emotion"
model = AutoModelForSequenceClassification.from_pretrained(model_name)

I saved the model with model.save_pretrained(model_name) and now I can’t load the tokenizer. If I run:

tokenizer = AutoTokenizer.from_pretrained(model_name)

it gives the error:

OSError: Can't load tokenizer for 'cardiffnlp/twitter-roberta-base-emotion'. Make sure that:

- 'cardiffnlp/twitter-roberta-base-emotion' is a correct model identifier listed on 'https://huggingface.co/models'
(make sure 'cardiffnlp/twitter-roberta-base-emotion' is not a path to a local directory with something else, in that case)

- or 'cardiffnlp/twitter-roberta-base-emotion' is the correct path to a directory containing relevant tokenizer files

Because I saved the model and not the tokenizer yesterday, I can’t load the tokenizer anymore. What can I do to fix this? I don’t understand how to save the tokenizer if I can’t load the tokenizer. Why does this only not work in Jupyter?

Source: Python Questions

LEAVE A COMMENT