mmarch/text-generation-webui
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Failed attempt at evaluating exllama_hf perplexity

Browse source
This commit is contained in:
oobabooga 2023-06-24 12:02:25 -03:00
parent e356f69b36
commit cec5fb0ef6
2 changed files with 21 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

2
modules/evaluate.py
View file

@ -100,7 +100,7 @@ def calculate_perplexity(models, input_dataset, stride, _max_length):
target_ids[:, :-trg_len] = -100
with torch.no_grad():
outputs = shared.model(input_ids, labels=target_ids)
outputs = shared.model(input_ids=input_ids, labels=target_ids)
# loss is calculated using CrossEntropyLoss which averages over valid labels
# N.B. the model only calculates loss over trg_len - 1 labels, because it internally shifts the labels