Freeze network parameters of some nodes

During training in a virtual environment, it is possible to freeze the network parameters of certain nodes and prevent them from being updated during gradient descent. This is particularly useful when loading an existing environment model.

For example, in a robot control task, suppose we have trained a good virtual environment model on a type-A robot task. Now we want to quickly transfer the virtual environment model of type-A robot to a new type-B robot task. The two robot systems share most of the same mechanical structure, but there are also some differences. In this case, we can use the virtual environment model of the type-A robot as a pre-trained model, and then protect the parts of the model that are the same as the type-A robot by freezing their parameters using the parameter freezing feature. This way, these nodes’ parameter settings will not be disturbed during the training process. Then, we can continue training on the type-B robot, only fine-tuning or relearning the parts that are different from the type-A robot.

The benefit of doing this is that it can save training time and computing resources while also avoiding overfitting to a certain extent, because the pre-trained model has learned some common features, which can avoid overfitting during the training process from scratch. In addition, since the model has been pre-trained, the size of the new dataset needed to be fitted during transfer learning does not need to be too large.

It should be noted that when freezing parameters, we need to decide which nodes need to be frozen and which nodes need to be retrained according to specific problems. If too many nodes are frozen, the robot in the new task may not be able to learn enough, while if too few nodes are frozen, the new task robot may not be able to take advantage of the pre-trained model, and the training time will be long. Therefore, reasonable trade-offs need to be made in specific operations.

When writing the .yaml file to implement this functionality, we need to specify the freeze attribute of the relevant nodes as true.

Here is an example of how to write the .yaml file:

metadata:

   graph:
     action:
     - observation
     next_observation:
     - action
     - observation

   columns:
   ...

In the following example, we set the freeze attribute of the action node to true in the .yaml file. During training, the network parameters of the action node will not be updated.

metadata:

   graph:
     action:
     - observation
     next_observation:
     - action
     - observation

   columns:
   ...

   nodes:
     action:
       freeze: true

Important

Do NOT freeze all network nodes at the same time. Make sure at least one network node can be updated, or REVIVE will report an error.