huawei-noah/SMARTS

Fine tuning pre-trained agent

Open

#1,471 opened on Jun 17, 2022

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Python (220 forks)auto 404
help wanted

Repository metrics

Stars
 (1,129 stars)
PR merge metrics
 (PR metrics pending)

Description

High Level Description Fine-tune the "RL-Agent" provided inside the model zoo, by specifying a new OpenAI Gym environment with customly-defined rewards. We tried several configurations, from understanding how to load the pre-trained weights inside Ray, to instantiate a model and replace the weights with the pre-trained ones, or by trying to registering the model inside Ray, but no strategies leaded us to anything working.

Desired SMARTS version 0.6.1rc1

Operating System Docker, Ubuntu 16.04

Problems The agent is correctly instantiated, but it is not clear how to fine-tune the created pretrained model. We tried to customize the rllib.py by instantiating the agent and passing it to Ray, through tune.run, but we were unsuccessful. Also, inside the rlagent script, there is a warning from developers that the zoo model is not trainable, but it is provided for evaluation only. No further explanations are given.

Contributor guide

Research direction
Inspect the SMARTS model zoo to load the pre trained agent checkpoint, then use Ray's tune.run with a custom training configuration that loads the checkpoint. Ensure the environment is registered with Gym and the reward function is defined. Refer to Ray RLlib documentation on restoring checkpoints and continuing training.
Tech stack
python
Domain
ai
Issue type
Research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Python
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
45