9 hours ago
https://www.eurekalert.org/news-releases/1142246
INTRO: Robots and other embodied agents learn by acting in the physical world, but the world does not remain fixed. A controller changes as training proceeds; hardware wears; payloads vary; and friction, gravity, wind, sensor noise and contact conditions can shift. At the same time, real-world interaction is costly, so an agent cannot simply discard its past and collect a fresh on-policy dataset whenever conditions change.
This creates a fundamental problem for standard off-policy reinforcement learning. Historical transitions stored in a replay buffer may have been generated by older policies and under older physical dynamics. When those stale transitions are used as if they still described the current environment, value estimates can become biased and the policy may learn actions whose predicted outcomes are no longer physically possible.
A research team led by Tsinghua University has proposed Transition Occupancy Matching as a common mathematical view of these two sources of change. A policy shift changes which action is selected in a state, while a dynamics shift changes the next state produced by that action. Both therefore change the joint distribution of state, action and next state, which the researchers call transition occupancy.
The team turned this principle into Occupancy-Matching Policy Optimization, or OMPO. Instead of treating all replayed data equally, OMPO maintains a large global buffer containing accumulated experience and a small first-in, first-out local buffer containing the most recent interactions. A discriminator compares transitions from the two buffers and estimates which historical transitions remain compatible with the current physical regime. Compatible experience is retained, while stale or mismatched transitions are downweighted.
OMPO also uses a sign-free logarithmic link to reformulate the matching objective as a stable min-max optimization. This design allows the method to work with reward signals that contain both bonuses and penalties, without adding a task-specific reward shift. A distributional critic models the full distribution of possible returns rather than only their mean, helping the agent account for randomness from action noise, sensing and contact dynamics. For visual tasks, the policy, critic and discriminator share a co-trained visual encoder, while an ODE-based flow actor represents potentially multimodal actions.
The researchers evaluated OMPO across three forms of distribution shift.. (MORE - details, no ads)
PAPER: http://dx.doi.org/10.1093/nsr/nwag506
INTRO: Robots and other embodied agents learn by acting in the physical world, but the world does not remain fixed. A controller changes as training proceeds; hardware wears; payloads vary; and friction, gravity, wind, sensor noise and contact conditions can shift. At the same time, real-world interaction is costly, so an agent cannot simply discard its past and collect a fresh on-policy dataset whenever conditions change.
This creates a fundamental problem for standard off-policy reinforcement learning. Historical transitions stored in a replay buffer may have been generated by older policies and under older physical dynamics. When those stale transitions are used as if they still described the current environment, value estimates can become biased and the policy may learn actions whose predicted outcomes are no longer physically possible.
A research team led by Tsinghua University has proposed Transition Occupancy Matching as a common mathematical view of these two sources of change. A policy shift changes which action is selected in a state, while a dynamics shift changes the next state produced by that action. Both therefore change the joint distribution of state, action and next state, which the researchers call transition occupancy.
The team turned this principle into Occupancy-Matching Policy Optimization, or OMPO. Instead of treating all replayed data equally, OMPO maintains a large global buffer containing accumulated experience and a small first-in, first-out local buffer containing the most recent interactions. A discriminator compares transitions from the two buffers and estimates which historical transitions remain compatible with the current physical regime. Compatible experience is retained, while stale or mismatched transitions are downweighted.
OMPO also uses a sign-free logarithmic link to reformulate the matching objective as a stable min-max optimization. This design allows the method to work with reward signals that contain both bonuses and penalties, without adding a task-specific reward shift. A distributional critic models the full distribution of possible returns rather than only their mean, helping the agent account for randomness from action noise, sensing and contact dynamics. For visual tasks, the policy, critic and discriminator share a co-trained visual encoder, while an ODE-based flow actor represents potentially multimodal actions.
The researchers evaluated OMPO across three forms of distribution shift.. (MORE - details, no ads)
PAPER: http://dx.doi.org/10.1093/nsr/nwag506