Repository metrics
- Stars
- (3,127 stars)
- PR merge metrics
- (No merged PRs in 30d)
Description
I'm getting memory limit exceeded even after allocating 125GB for the training of ntu_xview.
Getting this error
[11.14.18|20:57:54] Parameters: {'print_log': True, 'log_interval': 100, 'step': [10, 50], 'save_result': False, 'test_feeder_args': {'data_path': '/scratch/neeraj.b/data/NTU-RGB-D/xview/val_data.npy', 'label_path': '/scratch/neeraj.b/data/NTU-RGB-D/xview/val_label.pkl'}, 'optimizer': 'SGD', 'start_epoch': 0, 'batch_size': 64, 'phase': 'train', 'base_lr': 0.1, 'num_worker': 4, 'debug': False, 'weights': None, 'save_interval': 10, 'pavi_log': False, 'ignore_weights': [], 'save_log': True, 'num_epoch': 80, 'use_gpu': True, 'weight_decay': 0.0001, 'test_batch_size': 64, 'device': [0, 1, 2, 3], 'nesterov': True, 'feeder': 'feeder.feeder.Feeder', 'train_feeder_args': {'data_path': '/scratch/neeraj.b/data/NTU-RGB-D/xview/train_data.npy', 'label_path': '/scratch/neeraj.b/data/NTU-RGB-D/xview/train_label.pkl', 'debug': False}, 'eval_interval': 5, 'config': 'config/st_gcn/ntu-xview/train.yaml', 'work_dir': '/scratch/neeraj.b/data/work_dir', 'model_args': {'num_class': 60, 'graph_args': {'strategy': 'spatial', 'layout': 'ntu-rgb+d'}, 'edge_importance_weighting': True, 'dropout': 0.5, 'in_channels': 3}, 'show_topk': [1, 5], 'model': 'net.st_gcn.Model'}
[11.14.18|20:57:54] Training epoch: 0 slurmstepd: Job 157430 exceeded memory limit (131272228 > 131072000), being killed slurmstepd: Exceeded job memory limit