huawei-noah/vega

CARS Accuracy Metric Question

Open

#248 opened on Jun 8, 2022

 (1 comment) (0 reactions) (1 assignee)Python (177 forks)auto 404
good first issue

Repository metrics

Stars
 (848 stars)
PR merge metrics
 (PR metrics pending)

Description

I am using a custom dataset with CARS and it runs fine, but I am confused because it shows a different accuracy at the end of the search part than during the search itself. During the 500 epochs of searching it for example finds this NN with 89% Val accuracy: 2022-06-08 03:07:34.605 INFO ############## KEEP ############ 2022-06-08 03:07:34.609 INFO KEEP Valid_acc for invidual 1 0.890625, size 0.111890, genotype Genotype(normal=[['max_pool_3x3', 2, 0], ['max_pool_3x3', 2, 1], ['max_pool_3x3', 3, 0], ['max_pool_3x3', 3, 1], ['max_pool_3x3', 4, 2], ['max_pool_3x3', 4, 0], ['avg_pool_3x3', 5, 0], ['avg_pool_3x3', 5, 1]], normal_concat=range(2, 6), reduce=[['sep_conv_5x5', 2, 0], ['max_pool_3x3', 2, 1], ['avg_pool_3x3', 3, 0], ['skip_connect', 3, 1], ['max_pool_3x3', 4, 1], ['max_pool_3x3', 4, 0], ['avg_pool_3x3', 5, 0], ['max_pool_3x3', 5, 1]], reduce_concat=range(2, 6))

but then at the end of the search it shows the best NN had 84% accuracy: 022-06-08 03:11:18.300 INFO worker id [0], epoch [500/500], train step [10/17], loss [ 0.004, 0.100], lr [ 0.0010004], time pre batch [0.582s] , total mean time per batch [0.569s] 2022-06-08 03:11:23.176 INFO Finished the unified trainer successfully. 2022-06-08 03:11:23.191 INFO Update Success. step_name=nas, worker_id=0 2022-06-08 03:11:23.194 INFO Best values: [{'worker_id': 2, 'performance': {'accuracy': 0.84375}}] 2022-06-08 03:11:23.218 INFO Clean worker folder /content/tasks/0608.012736.396/workers/nas.

Is this because it is picking the best NNs based on the Val score, and then at the end showing what the Test accuracy was for it?

Contributor guide