RL-10-TD算法-ActorCritic03-连续动作控制01-DPG：DPG03【Actor与Critic协同训练机制：A的∇Jᶿ梯度组成：A的θ梯度×C的动作a梯度】【训练C的∇Jʷ：w梯度】

Deterministic Actor-Critic 的梯度/Gradient：

∇θJ(θ)=∑s∈Sρμ(s)∇θμ(s)(∇aqμ(s,a))∣a=μ(s)=ES∼ρμ[∇θμ(S)(∇aqμ(S,a))∣a=μ(S)]\color{red}{\begin{aligned} \nabla_{\theta} J(\theta) & =\sum_{s \in \mathcal{S}} \rho_{\mu}(s) \nabla_{\theta} \mu(s)\left(\nabla_{a} q_{\mu}(s, a)\right)|_{a=\mu(s)} \\ & =\mathbb{E}_{S \sim \rho_{\mu}}\left[\left.\nabla_{\theta} \mu(S)\left(\nabla_{a} q_{\mu}(S, a)\right)\right|_{a=\mu(S)}\right] \end{aligned}}