😇
Deep Multi-Agent Reinforcement Learning
  • Deep Multi-Agent Reinforcement Learning
  • Abstract & Contents
    • Abstract
  • 1. Introduction
    • 1. INTRODUCTION
      • 1.1 The Industrial Revolution, Cognition, and Computers
      • 1.2 Deep Multi-Agent Reinforcement-Learning
      • 1.3 Overall Structure
  • 2. Background
    • 2. BACKGROUND
      • 2.1 Reinforcement Learning
      • 2.2 Multi-Agent Settings
      • 2.3 Centralized vs Decentralized Control
      • 2.4 Cooperative, Zero-sum, and General-Sum
      • 2.5 Partial Observability
      • 2.6 Centralized Training, Decentralized Execution
      • 2.7 Value Functions
      • 2.8 Nash Equilibria
      • 2.9 Deep Learning for MARL
      • 2.10 Q-Learning and DQN
      • 2.11 Reinforce and Actor-Critic
  • I Learning to Collaborate
    • 3. Counterfactual Multi-Agent Policy Gradients
      • 3.1 Introduction
      • 3.2 Related Work
      • 3.3 Multi-Agent StarCraft Micromanagement
      • 3.4 Methods
        • 3.4.1 Independent Actor-Critic
        • 3.4.2 Counterfactual Multi-Agent Policy Gradients
        • 3.4.2.1 baseline lemma
        • 3.4.2.2 COMA Algorithm
      • 3.5 Results
      • 3.6 Conclusions & Future Work
    • 4 Multi-Agent Common Knowledge Reinforcement Learning
      • 4.1 Introduction
      • 4.2 Related Work
      • 4.3 Dec-POMDP and Features
      • 4.4 Common Knowledge
      • 4.5 Multi-Agent Common Knowledge Reinforcement Learning
      • 4.6 Pairwise MACKRL
      • 4.7 Experiments and Results
      • 4.8 Conclusion & Future Work
    • 5 Stabilizing Experience Replay
      • 5.1 Introduction
      • 5.2 Related Work
      • 5.3 Methods
        • 5.3.1 Multi-Agent Importance Sampling
        • 5.3.2 Multi-Agent Fingerprints
      • 5.4 Experiments
        • 5.4.1 Architecture
      • 5.5 Results
        • 5.5.1 Importance Sampling
        • 5.5.2 Fingerprints
        • 5.5.3 Informative Trajectories
      • 5.6 Conclusion & Future Work
  • II Learning to Communicate
    • 6. Learning to Communicate with Deep Multi-Agent ReinforcementLearning
      • 6.1 Introduction
      • 6.2 Related Work
      • 6.3 Setting
      • 6.4 Methods
        • 6.4.1 Reinforced Inter-Agent Learning
        • 6.4.2 Differentiable Inter-Agent Learning
      • 6.5 DIAL Details
      • 6.6 Experiments
        • 6.6.1 Model Architecture
        • 6.6.2 Switch Riddle
        • 6.6.3 MNIST Games
        • 6.6.4 Effect of Channel Noise
      • 6.7 Conclusion & Future Work
    • 7. Bayesian Action Decoder
      • 7.1 Introduction
      • 7.2 Setting
      • 7.3 Method
        • 7.3.1 Public belief
        • 7.3.2 Public Belief MDP
        • 7.3.3 Sampling Deterministic Partial Policies
        • 7.3.4 Factorized Belief Updates
        • 7.3.5 Self-Consistent Beliefs
      • 7.4 Experiments and Results
        • 7.4.1 Matrix Game
        • 7.4.2 Hanabi
        • 7.4.3 Observations and Actions
        • 7.4.4 Beliefs in Hanabi
        • 7.4.5 Architecture Details for Baselines and Method
        • 7.4.6 Hyperparamters
        • 7.4.7 Results on Hanabi
      • 7.5 Related Work
        • 7.5.1 Learning to Communicate
        • 7.5.2 Research on Hanabi
        • 7.5.3 Belief State Methods
      • 7.6 Conclusion & Future Work
  • III Learning to Reciprocate
    • 8. Learning with Opponent-Learning Awareness
      • 8.1 Introduction
      • 8.2 Related Work
      • 8.3 Methods
        • 8.3.1 Naive Learner
        • 8.3.2 Learning with Opponent Learning Awareness
        • 8.3.3. Learning via Policy gradient
        • 8.3.4 LOLA with Opponent modeling
        • 8.3.5 Higher-Order LOLA
      • 8.4 Experimental Setup
        • 8.4.1 Iterated Games
        • 8.4.2 Coin Game
        • 8.4.3 Training Details
      • 8.5 Results
        • 8.5.1 Iterated Games
        • 8.5.2 Coin Game
        • 8.5.3 Exploitability of LOLA
      • 8.6 Conclusion & Future Work
    • 9. DiCE: The Infinitely Differentiable Monte Carlo Estimator
      • 9.1 Introduction
      • 9.2 Background
        • 9.2.1 Stochastic Computation Graphs
        • 9.2.2 Surrogate Losses
      • 9.3 Higher Order Gradients
        • 9.3.1 Higher Order Gradient Estimators
        • 9.3.2 Higher Order Surrogate Losses
        • 9.3.3. Simple Failing Example
      • 9.4 Correct Gradient Estimators with DiCE
        • 9.4.1 Implement of DiCE
        • 9.4.2 Casuality
        • 9.4.3 First Order Variance Reduction
        • 9.4.4 Hessian-Vector Product
      • 9.5 Case Studies
        • 9.5.1 Empirical Verification
        • 9.5.2 DiCE For multi-agent RL
      • 9.6 Related Work
      • 9.7 Conclusion & Future Work
  • Reference
    • Reference
  • After
    • 보충
    • 역자 후기
Powered by GitBook
On this page

Was this helpful?

  1. III Learning to Reciprocate
  2. 9. DiCE: The Infinitely Differentiable Monte Carlo Estimator
  3. 9.4 Correct Gradient Estimators with DiCE

9.4.1 Implement of DiCE

DiCE의 처음부터 실용적인 논문임을 강조했는데, 이는 단순한 구현방법에 있습니다. MagicBox는 두가지 특성을 만족하면 됐었는데, 이는 다음과 같이 정의함으로써 두가지 성질을 다 가져갈 수 있습니다. 아래선 위를 확인하도록 합니다.

□(W)=exp⁡(τ−⊥(τ)) \square (\mathcal{W}) = \exp(\tau - \bot(\tau))□(W)=exp(τ−⊥(τ))

τ=∑w∈Wlog⁡(p(w;θ)) \tau = \sum_{w \in \mathcal{W}} \log(p(w;\theta))τ=∑w∈W​log(p(w;θ))

⊥ \bot ⊥은 ∇x⊥(x)=0 \nabla_x \bot(x) = 0 ∇x​⊥(x)=0이 되도록 하는 gradient를 안흐르도록 하는 operator로 pytorch의 detach같은 역할을 합니다. ⊥(x)→x\bot(x) \rightarrow x ⊥(x)→x이므로, □(W)→1\square (\mathcal{W}) \rightarrow 1□(W)→1임이 자명합니다. 이로써 첫번째 성질이 증명되었습니다. 바로 두번째 성질을 증명하면 다음과 같습니다.

∇θ□(W)=∇θexp⁡(τ−⊥(τ)) \nabla_\theta \square (\mathcal{W}) = \nabla_\theta\exp(\tau-\bot(\tau))∇θ​□(W)=∇θ​exp(τ−⊥(τ))

=exp⁡(τ−⊥(τ))∇θ(τ−⊥(τ))= \exp(\tau-\bot(\tau))\nabla_\theta(\tau-\bot(\tau))=exp(τ−⊥(τ))∇θ​(τ−⊥(τ))

=□(W)(∇θτ−0)=\square(\mathcal{W})(\nabla_\theta\tau-0)=□(W)(∇θ​τ−0)

=□(W)∑w∈W∇θlog⁡(p(w;θ))=\square(\mathcal{W})\sum_{w \in \mathcal{W}}\nabla_\theta \log(p(w;\theta))=□(W)∑w∈W​∇θ​log(p(w;θ))

그리고 magicbox operator를 구현하게되면, 주로 objective와 바로 연관지어 구현하는게 가장 간단한데, 일반적인 RL에선 J=E[∑rt]J = \mathbb{E}[\sum r_t]J=E[∑rt​]로 나타낼 때, DiCE의 objective는 J□=∑t□({at′,t′≤t})rt J_\square= \sum_t \square(\{a_{t'},t'\leq t\})r_tJ□​=∑t​□({at′​,t′≤t})rt​로 나타내야합니다. (이는 이전의 action에 따라 reward에 stochastic하게 영향을 주기 때문입니다.

Previous9.4 Correct Gradient Estimators with DiCENext9.4.2 Casuality

Last updated 4 years ago

Was this helpful?