%% Create Q-Value Function Critic from Custom Basis Function
% Create an observation specification object (or alternatively use getObservationInfo to extract the specification object from an environment). For this example, define the observation space as a continuous four-dimensional space, so that a single observation is a column vector containing 3 doubles.

obsInfo = rlNumericSpec([3 1]);

%
% Create an action specification object (or alternatively use getActionInfo to extract the specification object from an environment). For this example, define the action space as a continuous two-dimensional space, so that a single action is a column vector containing 2 doubles.

actInfo = rlNumericSpec([2 1]);

%
% Create a custom basis function to approximate the value function within the critic. The custom basis function must return a column vector. Each vector element must be a function of the observations and actions respectively defined by obsInfo and actInfo.

myBasisFcn = @(myobs,myact) [myobs(2)^2; myobs(1)+exp(myact(1)); abs(myact(2)); myobs(3)]

%
% The output of the critic is the scalar W'*myBasisFcn(myobs,myact), where W is a weight column vector which must have the same size of the custom basis function output. This output is the expected cumulative long term reward when the agent starts from the given observation and takes the best possible action. The elements of W are the learnable parameters.
% Define an initial parameter vector.

W0 = [1;4;4;2];

%
% Create the critic. The first argument is a two-element cell containing both the handle to the custom function and the initial weight vector. The second and third arguments are, respectively, the observation and action specification objects.

critic = rlQValueRepresentation({myBasisFcn,W0},obsInfo,actInfo)

%
% To check your critic, use the getValue function to return the value of a given observation-action pair, using the current parameter vector.

v = getValue(critic,{[1 2 3]'},{[4 5]'})

% You can now use the critic (along with an with an actor) to create an agent relying on a Q-value function critic (such as an rlQAgent, rlDQNAgent, rlSARSAAgent, or rlDDPGAgent agent).
% Copyright 2019 The MathWorks, Inc.