huawei-noah/vega

searching valid metric is better than re-train valid metric in autoFIS?

Open

#107 opened on Apr 8, 2021

View on GitHub
 (2 comments) (0 reactions) (1 assignee)Python (177 forks)auto 404
good first issuequestion

Repository metrics

Stars
 (848 stars)
PR merge metrics
 (PR metrics pending)

Description

Thank you for your excellent codes. I run your autoFIS codes (autoDeepFM), but found that the valid metric after searching (auc=0.7896248002165958) is better than the retrain valid metric (auc=0.7851094769386633) in the log. i.e. [searching valid metric] > [re-train valid metric] > [official valid metric], it looks like the performance improvement is achieved by BN but autofis damages the valid metric. Something wrong ?

the valid metric after searching: image the valid metric after re-train: image

Contributor guide

Research direction
Investigate the effect of Batch Normalization (BN) during the search phase vs. retrain phase in autoFIS. Compare the model architecture and training configuration between the two phases, focusing on whether BN is applied differently. Check if the search phase overfits or uses different data splits.
Tech stack
python
Domain
aimachine learning
Issue type
Research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Pythondeep learning basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70