Add a token counter similar to automatic1111

It can now be found in the Default and Notebook tabs
This commit is contained in:
oobabooga 2023-08-20 19:13:01 -07:00
parent 120fb86c6a
commit 429cacd715
4 changed files with 38 additions and 14 deletions

View file

@ -46,6 +46,6 @@ def load_instruction_prompt_simple(fname):
def count_tokens(text):
try:
tokens = get_encoded_length(text)
return f'{tokens} tokens in the input.'
return str(tokens)
except:
return 'Couldn\'t count the number of tokens. Is a tokenizer loaded?'
return '-1'