pytorch/examples

Query on loss calculation in word language model

Open

#1,082 opened on Oct 14, 2022

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Python (9,429 forks)batch import
help wanted

Repository metrics

Stars
 (21,634 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

In the main.py of word language model, I find that in the evaluate function the total_loss is getting multiplied by length of data https://github.com/pytorch/examples/blob/ca1bd9167f7216e087532160fc5b98643d53f87e/word_language_model/main.py#L163

However in the train function, total_loss is not getting multiplied by length of data https://github.com/pytorch/examples/blob/ca1bd9167f7216e087532160fc5b98643d53f87e/word_language_model/main.py#L194

Is this proper?

Contributor guide

Research direction
Compare the loss calculation in the train and evaluate functions. In train, total loss is not multiplied by the sequence length, while in evaluate it is. Determine if this is intentional by checking the batch handling and loss averaging. If inconsistent, propose to align them.
Tech stack
python
Domain
machine learning
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Under 1 hour
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PyTorch
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
75