Create an environment with a discrete action space, and obtain its observation and action specifications. For this example, load the environment used in the example Create Agent Using Deep Network Designer and Train Using Image Observations. This environment has two observations: a 50-by-50 grayscale image and a scalar (the angular velocity of the pendulum). The action is a scalar with five possible elements (a torque of either -2, -1, 0, 1, or 2 Nm applied to a swinging pole).

% load predefined environment
env = rlPredefinedEnv("SimplePendulumWithImage-Discrete");

% obtain observation and action specifications
obsInfo = getObservationInfo(env);
actInfo = getActionInfo(env);

The agent creation function initializes the actor and critic networks randomly. You can ensure reproducibility by fixing the seed of the random generator. To do so, uncomment the following line.

% rng(0)

Create a deep Q-network agent from the environment observation and action specifications.

agent = rlDQNAgent(obsInfo,actInfo);

To check your agent, use getAction to return the action from a random observation.

getAction(agent,{rand(obsInfo(1).Dimension),rand(obsInfo(2).Dimension)})

ans = 1

You can now test and train the agent within the environment.

Create an environment with a discrete action space, and obtain its observation and action specifications. For this example, load the environment used in the example Create Agent Using Deep Network Designer and Train Using Image Observations. This environment has two observations: a 50-by-50 grayscale image and a scalar (the angular velocity of the pendulum). The action is a scalar with five possible elements (a torque of either -2, -1, 0, 1, or 2 Nm applied to a swinging pole).

% load predefined environment
env = rlPredefinedEnv("SimplePendulumWithImage-Discrete");

% obtain observation and action specifications
obsInfo = getObservationInfo(env);
actInfo = getActionInfo(env);

Create an agent initialization option object, specifying that each hidden fully connected layer in the network must have 128 neurons (instead of the default number, 256).

initOpts = rlAgentInitializationOptions('NumHiddenUnit',128);

The agent creation function initializes the actor and critic networks randomly. You can ensure reproducibility by fixing the seed of the random generator. To do so, uncomment the following line.

% rng(0)

Create a policy gradient agent from the environment observation and action specifications.

agent = rlPGAgent(obsInfo,actInfo,initOpts);

Reduce the critic learning rate to 1e-3.

critic = getCritic(agent);
critic.Options.LearnRate = 1e-3;
agent  = setCritic(agent,critic);

Extract the deep neural network from both the critic.

criticNet = getModel(getCritic(agent));

The default DQN agent uses a multi-output Q-value critic approximator. A multi-output approximator has observations as inputs and state-action values as outputs. Each output element represents the expected cumulative long-term reward for taking the corresponding discrete action from the state indicated by the observation inputs.

Display the layers of the critic network, and verify that each hidden fully connected layer has 128 neurons

criticNet.Layers

ans = 
  12x1 Layer array with layers:

     1   'concat'               Concatenation       Concatenation of 2 inputs along dimension 3
     2   'relu_body'            ReLU                ReLU
     3   'fc_body'              Fully Connected     128 fully connected layer
     4   'body_output'          ReLU                ReLU
     5   'input_1'              Image Input         50x50x1 images
     6   'conv_1'               Convolution         64 3x3x1 convolutions with stride [1  1] and padding [0  0  0  0]
     7   'relu_input_1'         ReLU                ReLU
     8   'fc_1'                 Fully Connected     128 fully connected layer
     9   'input_2'              Image Input         1x1x1 images
    10   'fc_2'                 Fully Connected     128 fully connected layer
    11   'output'               Fully Connected     1 fully connected layer
    12   'RepresentationLoss'   Regression Output   mean-squared-error

Plot the critic network

plot(criticNet)

To check your agent, use getAction to return the action from a random observation.

getAction(agent,{rand(obsInfo(1).Dimension),rand(obsInfo(2).Dimension)})

ans = 1x1 cell array
    {[2]}

You can now test and train the agent within the environment.

Create an environment interface and obtain its observation and action specifications. For this example load the predefined environment used for the Train DQN Agent to Balance Cart-Pole System example. This environment has a continuous four-dimensional observation space (the positions and velocities of both cart and pole) and a discrete one-dimensional action space consisting on the application of two possible forces, -10N or 10N.

% load predefined environment
env = rlPredefinedEnv("CartPole-Discrete");

% get observation and action specification objects
obsInfo = getObservationInfo(env);
actInfo = getActionInfo(env);

For an agent with a discrete action space, you have the option to create a multi-output critic representation, which is generally more efficient than a comparable single-output critic representation.

A multi-output critic has only the observation as input, and an output vector having as many elements as the number of possible discrete actions. Each output element represents the expected cumulative long-term reward following from the observation given as input, when the corresponding discrete action is taken.

Create the multi-output critic representation using a deep neural network approximator.

% create a deep neural network approximator 
% the observation input layer must have 4 elements (obsInfo.Dimension(1))
% the action output layer must have 2 elements (length(actInfo.Elements))
dnn = [
    imageInputLayer([obsInfo.Dimension(1) 1 1], 'Normalization', 'none', 'Name', 'state')
    fullyConnectedLayer(24, 'Name', 'CriticStateFC1')
    reluLayer('Name', 'CriticRelu1')
    fullyConnectedLayer(24, 'Name', 'CriticStateFC2')
    reluLayer('Name','CriticCommonRelu')
    fullyConnectedLayer(length(actInfo.Elements), 'Name', 'output')];

% set some options for the critic
criticOpts = rlRepresentationOptions('LearnRate',0.01,'GradientThreshold',1);

% create the critic based on the network approximator
critic = rlQValueRepresentation(dnn,obsInfo,actInfo,'Observation',{'state'},criticOpts);

Specify agent options, and create a DQN agent using the critic.

agentOpts = rlDQNAgentOptions(...
    'UseDoubleDQN',false, ...    
    'TargetUpdateMethod',"periodic", ...
    'TargetUpdateFrequency',4, ...   
    'ExperienceBufferLength',100000, ...
    'DiscountFactor',0.99, ...
    'MiniBatchSize',256);

agent = rlDQNAgent(critic,agentOpts)

agent = 
  rlDQNAgent with properties:

        AgentOptions: [1x1 rl.option.rlDQNAgentOptions]
    ExperienceBuffer: [1x1 rl.util.ExperienceBuffer]

To check your agent, use getAction to return the action from a random observation.

getAction(agent,{rand(4,1)})

ans = 10

You can now test and train the agent against the environment.

Create an environment interface and obtain its observation and action specifications. For this example load the predefined environment used for the Train DQN Agent to Balance Cart-Pole System example. This environment has a continuous four-dimensional observation space (the positions and velocities of both cart and pole) and a discrete one-dimensional action space consisting on the application of two possible forces, -10N or 10N.

% load predefined environment
env = rlPredefinedEnv("CartPole-Discrete");

% get observation and specification info
obsInfo = getObservationInfo(env);
actInfo = getActionInfo(env);

Create a single-output critic representation using a deep neural network approximator. It must have both observations and action as input layers, and a single scalar output representing the expected cumulative long-term reward following from the given observation and action.

% create a deep neural network approximator 
% the observation input layer must have 4 elements (obsInfo.Dimension(1))
% the action input layer must have 1 element (actInfo.Dimension(1))
% the output must be a scalar
statePath = [
    featureInputLayer(obsInfo.Dimension(1), 'Normalization', 'none', 'Name', 'state')
    fullyConnectedLayer(24, 'Name', 'CriticStateFC1')
    reluLayer('Name', 'CriticRelu1')
    fullyConnectedLayer(24, 'Name', 'CriticStateFC2')];
actionPath = [
    featureInputLayer(actInfo.Dimension(1), 'Normalization', 'none', 'Name', 'action')
    fullyConnectedLayer(24, 'Name', 'CriticActionFC1')];
commonPath = [
    additionLayer(2,'Name', 'add')
    reluLayer('Name','CriticCommonRelu')
    fullyConnectedLayer(1, 'Name', 'output')];
criticNetwork = layerGraph(statePath);
criticNetwork = addLayers(criticNetwork, actionPath);
criticNetwork = addLayers(criticNetwork, commonPath);    
criticNetwork = connectLayers(criticNetwork,'CriticStateFC2','add/in1');
criticNetwork = connectLayers(criticNetwork,'CriticActionFC1','add/in2');

% set some options for the critic
criticOpts = rlRepresentationOptions('LearnRate',0.01,'GradientThreshold',1);

% create the critic based on the network approximator
critic = rlQValueRepresentation(criticNetwork,obsInfo,actInfo,...
    'Observation',{'state'},'Action',{'action'},criticOpts);

Specify agent options, and create a DQN agent using the critic.

agentOpts = rlDQNAgentOptions(...
    'UseDoubleDQN',false, ...    
    'TargetUpdateMethod',"periodic", ...
    'TargetUpdateFrequency',4, ...   
    'ExperienceBufferLength',100000, ...
    'DiscountFactor',0.99, ...
    'MiniBatchSize',256);

agent = rlDQNAgent(critic,agentOpts)

agent = 
  rlDQNAgent with properties:

        AgentOptions: [1x1 rl.option.rlDQNAgentOptions]
    ExperienceBuffer: [1x1 rl.util.ExperienceBuffer]

To check your agent, use getAction to return the action from a random observation.

getAction(agent,{rand(4,1)})

ans = 10

You can now test and train the agent against the environment.

Create an environment and obtain observation and action information.

env = rlPredefinedEnv('CartPole-Discrete');
obsInfo = getObservationInfo(env);
actInfo = getActionInfo(env);
numObs = obsInfo.Dimension(1);
numDiscreteAct = numel(actInfo.Elements);

Create a recurrent deep neural network for your critic. To create a recurrent neural network, use a sequenceInputLayer as the input layer and include an lstmLayer as one of the other network layers.

For DQN agents, only the multi-output Q-value function representation supports recurrent neural networks.

criticNetwork = [
    sequenceInputLayer(numObs,'Normalization','none','Name','state')
    fullyConnectedLayer(50, 'Name', 'CriticStateFC1')
    reluLayer('Name','CriticRelu1')
    lstmLayer(20,'OutputMode','sequence','Name','CriticLSTM');
    fullyConnectedLayer(20,'Name','CriticStateFC2')
    reluLayer('Name','CriticRelu2')
    fullyConnectedLayer(numDiscreteAct,'Name','output')];

Create a representation for your critic using the recurrent neural network.

criticOptions = rlRepresentationOptions('LearnRate',1e-3,'GradientThreshold',1);
critic = rlQValueRepresentation(criticNetwork,obsInfo,actInfo,...
    'Observation','state',criticOptions);

Specify options for creating the DQN agent. To use a recurrent neural network, you must specify a SequenceLength greater than 1.

agentOptions = rlDQNAgentOptions(...
    'UseDoubleDQN',false, ...
    'TargetSmoothFactor',5e-3, ...
    'ExperienceBufferLength',1e6, ...
    'SequenceLength',20);
agentOptions.EpsilonGreedyExploration.EpsilonDecay = 1e-4;
agent = rlDQNAgent(critic,agentOptions);