DRL Algos Collection

A colleciotn of implements of classical DRL algorithms.

The repository contain the implementation of A3C, A2C, DDQN, and REINFORCE(naive) with Tensorflow2.0. Some of them have been demostrated in the OpenAI Cart Pole environment.

In additon, it modulizes the API of environments(Cart Pole, Flappy Bird, and remote environment), exploration stratgies(although I still working on it). The remote environment even allows the agent to connect to the external server and interact with them.

---

DRL Models

A3C

Still working on modulizing, but here is the DEMO on OpenAI Cart Pole. I use Master-Slave strategy(which is similar to the parameter server strategy in TensorFlow1) with TensorFlow2.0 and Multiprocessing.py to implement. Worker send the updated gradients to the master, and the master receive the updated gradients and apply them to the global model. The master also keeps send the lastest model variables to the workers.

However, TensorFlow2.0 has removed the tf.Session() which can allocate the computation task to specific device. Therefore, I use with tf.device() to specify the device of the task.

For more detail, please read the doc: Tricks of A3C on TensorFlow2 + Multiprocessing

and here is the DEMO

Tensorflow DEMO on Cart Pole Tensorflow DEMO on Flappy Bird

A2C

Implementation of Actor-Critic Network

Tensorflow DEMO on Cart Pole

Tensorflow Code

DDQN

Implementation of Doubly Deep Q-Network with Tensorflow

Tensorflow Code

REINFORCE

Tensorflow Code

DDPG

Working on it

PPO

Working on it

---

Environments

Integrate the API of different environments.

Cart Pole

From OpenAI Cart Pole.

Python Code

Flappy Bird

From PLE Flappy Bird

Python Code

Remote Environment

Create a TCP client and connect to the provided server. You can see the DEMO and the details in another repo(RL Java Integral) which we implement a Java multi-threading server and interact with the A3C model. Thanks tom1236868 for implementing the Java server.

Python Code

---

With LSTM

Reference:

• Keras LSTM layer
• Keras TimeDistributed layer
• Github flyyufelix/VizDoom-Keras-RL

Reference:

• 当我们在谈论 DRL：从AC、PG 到 A3C、DDPG

• Deep Deterministic Policy Gradient (DDPG)

• Actor Critic Method

• Deriving Policy Gradients and Implementing REINFORCE

• Movan Python - Policy Gradient

• Deep Reinforcement Learning: Playing CartPole through Asynchronous Advantage Actor Critic (A3C) with tf.keras and eager execution

• flappy bird REINFORCE

• kevin-fang/reinforced-flappy-bird

• Tensorflow 2.0 Pitfalls

• 分布式训练（理论篇）

• Ceruleanacg/Learning-Notes

• 重拾基础 - A3C & DPPO

• Day 23: Tensorflow 2.0: 再造訪 Distribute Strategy API

• tf.distribute.cluster_resolver.ClusterResolver

• 【04】tensorflow 到底該用 name scope 還是 variable scope

• 在Flask使用TensorFlow的几个常见错误

• keras 或 tensorflow 调用GPU报错：Blas GEMM launch failed

• How to run Keras.model() for prediction inside a tensorflow session?

• Day 23: Tensorflow 2.0: 再造訪 Distribute Strategy API

• [第 27 天] 深度學習 TensorFlow

• 一句一句读Pytorch（更新中）

• 并行处理最佳实践