😇
Deep Multi-Agent Reinforcement Learning
  • Deep Multi-Agent Reinforcement Learning
  • Abstract & Contents
    • Abstract
  • 1. Introduction
    • 1. INTRODUCTION
      • 1.1 The Industrial Revolution, Cognition, and Computers
      • 1.2 Deep Multi-Agent Reinforcement-Learning
      • 1.3 Overall Structure
  • 2. Background
    • 2. BACKGROUND
      • 2.1 Reinforcement Learning
      • 2.2 Multi-Agent Settings
      • 2.3 Centralized vs Decentralized Control
      • 2.4 Cooperative, Zero-sum, and General-Sum
      • 2.5 Partial Observability
      • 2.6 Centralized Training, Decentralized Execution
      • 2.7 Value Functions
      • 2.8 Nash Equilibria
      • 2.9 Deep Learning for MARL
      • 2.10 Q-Learning and DQN
      • 2.11 Reinforce and Actor-Critic
  • I Learning to Collaborate
    • 3. Counterfactual Multi-Agent Policy Gradients
      • 3.1 Introduction
      • 3.2 Related Work
      • 3.3 Multi-Agent StarCraft Micromanagement
      • 3.4 Methods
        • 3.4.1 Independent Actor-Critic
        • 3.4.2 Counterfactual Multi-Agent Policy Gradients
        • 3.4.2.1 baseline lemma
        • 3.4.2.2 COMA Algorithm
      • 3.5 Results
      • 3.6 Conclusions & Future Work
    • 4 Multi-Agent Common Knowledge Reinforcement Learning
      • 4.1 Introduction
      • 4.2 Related Work
      • 4.3 Dec-POMDP and Features
      • 4.4 Common Knowledge
      • 4.5 Multi-Agent Common Knowledge Reinforcement Learning
      • 4.6 Pairwise MACKRL
      • 4.7 Experiments and Results
      • 4.8 Conclusion & Future Work
    • 5 Stabilizing Experience Replay
      • 5.1 Introduction
      • 5.2 Related Work
      • 5.3 Methods
        • 5.3.1 Multi-Agent Importance Sampling
        • 5.3.2 Multi-Agent Fingerprints
      • 5.4 Experiments
        • 5.4.1 Architecture
      • 5.5 Results
        • 5.5.1 Importance Sampling
        • 5.5.2 Fingerprints
        • 5.5.3 Informative Trajectories
      • 5.6 Conclusion & Future Work
  • II Learning to Communicate
    • 6. Learning to Communicate with Deep Multi-Agent ReinforcementLearning
      • 6.1 Introduction
      • 6.2 Related Work
      • 6.3 Setting
      • 6.4 Methods
        • 6.4.1 Reinforced Inter-Agent Learning
        • 6.4.2 Differentiable Inter-Agent Learning
      • 6.5 DIAL Details
      • 6.6 Experiments
        • 6.6.1 Model Architecture
        • 6.6.2 Switch Riddle
        • 6.6.3 MNIST Games
        • 6.6.4 Effect of Channel Noise
      • 6.7 Conclusion & Future Work
    • 7. Bayesian Action Decoder
      • 7.1 Introduction
      • 7.2 Setting
      • 7.3 Method
        • 7.3.1 Public belief
        • 7.3.2 Public Belief MDP
        • 7.3.3 Sampling Deterministic Partial Policies
        • 7.3.4 Factorized Belief Updates
        • 7.3.5 Self-Consistent Beliefs
      • 7.4 Experiments and Results
        • 7.4.1 Matrix Game
        • 7.4.2 Hanabi
        • 7.4.3 Observations and Actions
        • 7.4.4 Beliefs in Hanabi
        • 7.4.5 Architecture Details for Baselines and Method
        • 7.4.6 Hyperparamters
        • 7.4.7 Results on Hanabi
      • 7.5 Related Work
        • 7.5.1 Learning to Communicate
        • 7.5.2 Research on Hanabi
        • 7.5.3 Belief State Methods
      • 7.6 Conclusion & Future Work
  • III Learning to Reciprocate
    • 8. Learning with Opponent-Learning Awareness
      • 8.1 Introduction
      • 8.2 Related Work
      • 8.3 Methods
        • 8.3.1 Naive Learner
        • 8.3.2 Learning with Opponent Learning Awareness
        • 8.3.3. Learning via Policy gradient
        • 8.3.4 LOLA with Opponent modeling
        • 8.3.5 Higher-Order LOLA
      • 8.4 Experimental Setup
        • 8.4.1 Iterated Games
        • 8.4.2 Coin Game
        • 8.4.3 Training Details
      • 8.5 Results
        • 8.5.1 Iterated Games
        • 8.5.2 Coin Game
        • 8.5.3 Exploitability of LOLA
      • 8.6 Conclusion & Future Work
    • 9. DiCE: The Infinitely Differentiable Monte Carlo Estimator
      • 9.1 Introduction
      • 9.2 Background
        • 9.2.1 Stochastic Computation Graphs
        • 9.2.2 Surrogate Losses
      • 9.3 Higher Order Gradients
        • 9.3.1 Higher Order Gradient Estimators
        • 9.3.2 Higher Order Surrogate Losses
        • 9.3.3. Simple Failing Example
      • 9.4 Correct Gradient Estimators with DiCE
        • 9.4.1 Implement of DiCE
        • 9.4.2 Casuality
        • 9.4.3 First Order Variance Reduction
        • 9.4.4 Hessian-Vector Product
      • 9.5 Case Studies
        • 9.5.1 Empirical Verification
        • 9.5.2 DiCE For multi-agent RL
      • 9.6 Related Work
      • 9.7 Conclusion & Future Work
  • Reference
    • Reference
  • After
    • 보충
    • 역자 후기
Powered by GitBook
On this page

Was this helpful?

  1. III Learning to Reciprocate
  2. 9. DiCE: The Infinitely Differentiable Monte Carlo Estimator
  3. 9.3 Higher Order Gradients

9.3.3. Simple Failing Example

Previous9.3.2 Higher Order Surrogate LossesNext9.4 Correct Gradient Estimators with DiCE

Last updated 4 years ago

Was this helpful?

이번 chapter에서는 x∼Ber(θ) x \sim \mathrm{Ber}(\theta)x∼Ber(θ)를 따르는 xxx에 대한 간단한 예제를 보겠습니다.

f(x,θ)=x(1−θ)+(1−x)(1+θ) f(x,\theta) = x(1-\theta) + (1-x)(1+\theta) f(x,θ)=x(1−θ)+(1−x)(1+θ)일 때, L\mathcal{L} L은 다음과 같이 정의됩니다.

L=Ex[f(x;θ)] \mathcal{L} = \mathbb{E}_x[f(x;\theta)]L=Ex​[f(x;θ)]

=∑xp(x;θ)f(x)= \sum_x{p(x;\theta)}f(x)=∑x​p(x;θ)f(x)

=∑xp(x;θ)(x(1−θ)+(1−x)(1+θ)) = \sum_x p(x;\theta)(x(1-\theta)+(1-x)(1+\theta))=∑x​p(x;θ)(x(1−θ)+(1−x)(1+θ))

=(θ(1(1−θ)+0(1+θ)))+((1−θ)(0(1−θ)+1(1+θ)) = (\theta (1(1-\theta)+0(1+\theta))) + ((1-\theta)(0(1-\theta)+ 1(1+\theta))=(θ(1(1−θ)+0(1+θ)))+((1−θ)(0(1−θ)+1(1+θ))

=−2θ2+θ+1 =-2 \theta^2+\theta + 1=−2θ2+θ+1

이 때, 그냥 2차 미분까지 한다면 다음과 같습니다.

∇θL=−4θ+1 \nabla_{\theta}\mathcal{L} = -4\theta+1 ∇θ​L=−4θ+1

∇θ2L=−4\nabla^2_\theta\mathcal{L} = -4 ∇θ2​L=−4

SL을 사용한 미분은 다음과 같습니다.

(∇θL)SL=−4θ+1( \nabla_{\theta}\mathcal{L})_{\mathrm{SL}} = -4\theta + 1(∇θ​L)SL​=−4θ+1

(∇θ2L)SL=−2( \nabla^2_{\theta}\mathcal{L})_{\mathrm{SL}} = -2(∇θ2​L)SL​=−2

(f^\hat{f}f^​를 어떻게 세워야 나올지 개념은 이해했는데, 여기에 적용이 어려웠습니다.)

sampling을 아무리 많이한다해도 SL estimator는 잘못된 2차 미분값을 내놓습니다. 만약 이런 잘못된 estimate이 2차 미분을 이용하는 Newton-Raphson method같은 optimization method와 결합된다면 절대 θ\thetaθ는 바른값에 수렴하지 못할 것입니다. 반대로 정확한 gradient를 갖는다면 이는 단한번에도 수렴할 수 있습니다.

이번 예제에서 보여주는 점은 θ\thetaθ가 stochastic sample에 의해 regularization될 때, 비슷하게 일어날 것이라는 것을 보여줍니다. 예를들면 soft Q-learning에서도 reward에 entropy penalty를 줌으로써 policy를 regularization하는데, 이 penalty는 policy parameter θ \thetaθ에 의존적입니다. 또한 state에 대해서도 의존적인데, 이는 차례로 stochastically sampled action에 의해 영향을 받습니다. 결과적으로, entropy를 통해 regularization을 하는 RL objective는 모두 위에서의 문제처럼 SL의 접근이 실패할 것임을 암시합니다. 이는 차 gradient에서만 일어나는 것이 아닌, θ\thetaθ의 복잡성에 따라도 달려있음을 보입니다. 이처럼 regularized objective를 가진 이차 gradient를 사용하는 method는 다른 대체 방법을 찾아야합니다.