research

PG-DPO — Pontryagin-Guided Direct Policy Optimization.

PG-DPO: Pontryagin-Guided Direct Policy Optimization

Forward simulation. BPTT costates. Hamiltonian recovery.

▶ Interactive demo

PG-DPO is our framework for solving high-dimensional, continuous-time stochastic control problems. Instead of learning an entire value landscape, it learns the policy path, estimates the costate, and recovers the control locally from the Hamiltonian optimality condition.

PG-DPO two-stage sketch in the Merton model
A two-stage sketch in the Merton model: warm-up, then costate estimation and control recovery.

The two stages

Stage 1 — Warm-up. A feasible policy network is trained by direct simulation of the controlled dynamics.

Stage 2 — Costate estimation + control recovery. Backpropagation through time (BPTT) over continuation rollouts produces pathwise costate estimates; Monte-Carlo averaging stabilizes them into an adjoint signal. The estimated costate then enters the Hamiltonian optimality condition and the optimal control is recovered directly.

The backward pass computes the pathwise discrete adjoint of the time-discretized control problem. Under stationarity and martingale-projection consistency, it converges to the adjoint BSDE — the continuous-time Pontryagin costate. See Why BPTT ≈ Costate?.

Core idea

Classical Hamilton–Jacobi–Bellman (HJB) methods provide verification but face the curse of dimensionality; deep reinforcement learning scales better but loses structural optimality. PG-DPO combines all three desiderata:

  • the scalability of neural policies,
  • the structural discipline of Pontryagin’s maximum principle, and
  • the numerical precision of local Hamiltonian control recovery.

Why Pontryagin-guided?

Many hard control problems have delicate intermediate structure rather than a complicated final policy — settings where global value-function learning is unstable or expensive:

  • high-dimensional portfolio choice,
  • hard constraints,
  • parameter uncertainty,
  • non-Markovian or delay-driven dynamics,
  • non-exponential discounting,
  • transaction costs with no-trade regions.

PG-DPO uses simulated rollouts and adjoint sensitivities to enforce local optimality conditions instead of solving a global PDE.

PG-DPO problem landscape

Extensions

The framework extends across continuous-time control. For constrained problems the Hamiltonian recovery becomes a local KKT / barrier / QP-style decoder; for transaction-cost problems the costate-to-control map recovers buy / hold / sell regimes and no-trade regions. Further extensions cover non-Markovian dynamics and non-exponential discounting.

References

  • Breaking the Dimensional Barrier: A Pontryagin-Guided Direct Policy Optimization for Continuous-Time Multi-Asset Portfolio — arXiv:2504.11116
  • Breaking the Dimensional Barrier for Constrained Dynamic Portfolio Choice — under revision in Mathematical Finance, 2026, arXiv:2501.12600
  • Breaking the Dimensional Barrier: Dynamic Portfolio Choice with Parameter Uncertainty via Pontryagin Projection — arXiv:2601.03175
  • Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting — accepted in International Conference on Machine Learning (ICML), 2026, arXiv:2605.20996
  • Rec-ve-ing the Ki--s in D--ay C-nt-ol: A Str-ct-re-Aw-re O-timal Con--ol So-ve- wit- Pon-ry-gin -roj-ction
  • Rec-ve-ing No-Tr-de Re-i-ns: Pont--a-in-Gui--d Po--cy Proj--tion f-r Tr--action-C-st -ont-ol

See Papers and Papers in Progress for the full list.