rlValueFunction error: The number of network input layers must be e... (2024)

Table of Contents
Accepted Answer More Answers (0) See Also Categories Tags Products Community Treasure Hunt

25 views (last 30 days)

Show older comments

Anna on 18 Jun 2024 at 15:12

  • Link

    Direct link to this question

    https://jmaab.mathworks.com/matlabcentral/answers/2129741-rlvaluefunction-error-the-number-of-network-input-layers-must-be-equal-to-the-number-of-observation

⋮

  • Link

    Direct link to this question

    https://jmaab.mathworks.com/matlabcentral/answers/2129741-rlvaluefunction-error-the-number-of-network-input-layers-must-be-equal-to-the-number-of-observation

Commented: Anna on 21 Jun 2024 at 16:04

Accepted Answer: Ayush Aniket

Open in MATLAB Online

Hi,

I am currently training the biped robot with PPO algorithm. I used rlValueFunction to create critic, but it keep shows this error:

Error using rl.internal.validate.mapFunctionObservationInput (line 5)

The number of network input layers must be equal to the number of observation channels in the

environment specification object.

Error in rlValueFunction (line 81)

model = rl.internal.validate.mapFunctionObservationInput(model,observationInfo,...

Error in createPPONetworks (line 219)

critic = rlValueFunction(criticNetwork,env.getObservationInfo)

The critic network code and plot image is below:

criticLayerSizes = [600,400]; %400,300

statePath = [

imageInputLayer([31 1 1],'Normalization','none','Name', 'observation')

fullyConnectedLayer(criticLayerSizes(1), 'Name', 'CriticStateFC1', ...

'Weights',2/sqrt(31)*(rand(criticLayerSizes(1),31)-0.5), ...

'Bias',2/sqrt(31)*(rand(criticLayerSizes(1),1)-0.5))

reluLayer('Name','CriticStateRelu1')

fullyConnectedLayer(criticLayerSizes(2), 'Name', 'CriticStateFC2', ...

'Weights',2/sqrt(criticLayerSizes(1))*(rand(criticLayerSizes(2),criticLayerSizes(1))-0.5), ...

'Bias',2/sqrt(criticLayerSizes(1))*(rand(criticLayerSizes(2),1)-0.5))

];

actionPath = [

imageInputLayer([6 1 1],'Normalization','none', 'Name', 'action') %numAct

fullyConnectedLayer(criticLayerSizes(2), 'Name', 'CriticActionFC1', ...

'Weights',2/sqrt(6)*(rand(criticLayerSizes(2),6)-0.5), ...

'Bias',2/sqrt(6)*(rand(criticLayerSizes(2),1)-0.5))

];

commonPath = [

additionLayer(2,'Name','add')

reluLayer('Name','CriticCommonRelu1')

fullyConnectedLayer(1, 'Name', 'CriticOutput',...

'Weights',2*5e-3*(rand(1,criticLayerSizes(2))-0.5), ...

'Bias',2*5e-3*(rand(1,1)-0.5))

];

% Connect the layer graph

criticNetwork = layerGraph(statePath);

criticNetwork = addLayers(criticNetwork, actionPath);

criticNetwork = addLayers(criticNetwork, commonPath);

criticNetwork = connectLayers(criticNetwork,'CriticStateFC2','add/in1');

criticNetwork = connectLayers(criticNetwork,'CriticActionFC1','add/in2');

% Create critic representation

criticOptions = rlRepresentationOptions('Optimizer','adam','LearnRate',1e-3, ...

'GradientThreshold',1,'L2RegularizationFactor',2e-4);

if useGPU

%criticOptions.UseDevice = 'gpu';

end

critic = rlValueFunction(criticNetwork,env.getObservationInfo)

rlValueFunction error: The number of network input layers must be e... (2)

I couldn't find why that error occurs. Can anyone please offer me some guidance regarding this problem? Thank you very much.

0 Comments

Show -2 older commentsHide -2 older comments

Sign in to comment.

Sign in to answer this question.

Accepted Answer

Ayush Aniket on 19 Jun 2024 at 4:21

  • Link

    Direct link to this answer

    https://jmaab.mathworks.com/matlabcentral/answers/2129741-rlvaluefunction-error-the-number-of-network-input-layers-must-be-equal-to-the-number-of-observation#answer_1474016

⋮

  • Link

    Direct link to this answer

    https://jmaab.mathworks.com/matlabcentral/answers/2129741-rlvaluefunction-error-the-number-of-network-input-layers-must-be-equal-to-the-number-of-observation#answer_1474016

Open in MATLAB Online

Hi Anna,

The error message you are encountering indicates a mismatch between the number of input layers in your critic network and the number of observation channels provided by your environment's observation specifications. Referring to the Biped Robot example as here: https://www.mathworks.com/help/reinforcement-learning/ug/train-biped-robot-to-walk-using-reinforcement-learning-agents.html#TrainBipedRobotToWalkUsingReinforcementLearningAgentsExample-2 ,

the dimension of ObservationInfo of the environment is [29 1], while the input layer of your critic network expects an image with the dimension [31 1 1]. The use of imageInputLayer is typically for image data inputs. If your observations and actions are not image data (which seems to be the case given their dimensions), consider using featureInputLayer instead for a more appropriate representation. This is especially true if your observations and actions are simply vectors of numerical values.

You can adjust the observation input layer as shown below:

statePath = [

featureInputLayer(obsInfo.Dimension(1),'Normalization','none','Name', 'observation')

% ... rest of your layers

];

For the action layer, you should use rlDiscreteCategoricalActor function as the action space is discrete for the environment. You can read more about the function here: https://www.mathworks.com/help/reinforcement-learning/ref/rl.function.rldiscretecategoricalactor.html

Refer to the following link to know the process of creating critic and actor network using rlvalueFunction: https://www.mathworks.com/help/reinforcement-learning/ref/rl.function.rlvaluefunction.html#mw_810cde6a-16f5-47da-b472-acabe4917d18

3 Comments

Show 1 older commentHide 1 older comment

Anna on 19 Jun 2024 at 14:43

Direct link to this comment

https://jmaab.mathworks.com/matlabcentral/answers/2129741-rlvaluefunction-error-the-number-of-network-input-layers-must-be-equal-to-the-number-of-observation#comment_3190816

⋮

  • Link

    Direct link to this comment

    https://jmaab.mathworks.com/matlabcentral/answers/2129741-rlvaluefunction-error-the-number-of-network-input-layers-must-be-equal-to-the-number-of-observation#comment_3190816

Open in MATLAB Online

Hi Aniket,

I truly appreciate your response. I have revised the code as you adviced, like this:

%% CRITIC

% Create the critic network layers

criticLayerSizes = [600,400]; %400,300

See Also
Using dlarray with betarnd/randg

statePath = [

featureInputLayer(observationInfo.Dimension(1),'Normalization','none','Name', 'observation')

fullyConnectedLayer(criticLayerSizes(1), 'Name', 'CriticStateFC1', ...

'Weights',2/sqrt(observationInfo.Dimension(1))*(rand(criticLayerSizes(1),observationInfo.Dimension(1))-0.5), ...

'Bias',2/sqrt(observationInfo.Dimension(1))*(rand(criticLayerSizes(1),1)-0.5))

reluLayer('Name','CriticStateRelu1')

fullyConnectedLayer(criticLayerSizes(2), 'Name', 'CriticStateFC2', ...

'Weights',2/sqrt(criticLayerSizes(1))*(rand(criticLayerSizes(2),criticLayerSizes(1))-0.5), ...

'Bias',2/sqrt(criticLayerSizes(1))*(rand(criticLayerSizes(2),1)-0.5))

%reluLayer('Name','CriticStateRelu2') %added

%fullyConnectedLayer(1, 'Name','CriticStateFC3')

];

actionPath = [

featureInputLayer(actionInfo.Dimension(1),'Normalization','none','Name', 'action')%numAct

fullyConnectedLayer(criticLayerSizes(2), 'Name', 'CriticActionFC1', ...

'Weights',2/sqrt(actionInfo.Dimension(1))*(rand(criticLayerSizes(2),actionInfo.Dimension(1))-0.5), ...

'Bias',2/sqrt(actionInfo.Dimension(1))*(rand(criticLayerSizes(2),1)-0.5))

];

commonPath = [

additionLayer(2,'Name','add')

reluLayer('Name','CriticCommonRelu1')

fullyConnectedLayer(1, 'Name', 'CriticOutput',...

'Weights',2*5e-3*(rand(1,criticLayerSizes(2))-0.5), ...

'Bias',2*5e-3*(rand(1,1)-0.5))

];

% Connect the layer graph

criticNetwork = layerGraph(statePath);

criticNetwork = addLayers(criticNetwork, actionPath);

criticNetwork = addLayers(criticNetwork, commonPath);

criticNetwork = connectLayers(criticNetwork,'CriticStateFC2','add/in1');

criticNetwork = connectLayers(criticNetwork,'CriticActionFC1','add/in2');

figure(2)

plot(criticNetwork)

hold on

% Create critic representation

criticOptions = rlRepresentationOptions('Optimizer','adam','LearnRate',1e-3, ...

'GradientThreshold',1,'L2RegularizationFactor',2e-4);

if useGPU

%criticOptions.UseDevice = 'gpu';

end

critic = rlValueFunction(criticNetwork,env.getObservationInfo)%,env.getActionInfo)%,...

However, it keeps showing the same error:

The number of network input layers must be equal to the number of observation channels in the

environment specification object.

Can you tell me if there are any suspicious parts in this code? Thank you for help again.

Ayush Aniket on 20 Jun 2024 at 6:33

Direct link to this comment

https://jmaab.mathworks.com/matlabcentral/answers/2129741-rlvaluefunction-error-the-number-of-network-input-layers-must-be-equal-to-the-number-of-observation#comment_3191251

⋮

  • Link

    Direct link to this comment

    https://jmaab.mathworks.com/matlabcentral/answers/2129741-rlvaluefunction-error-the-number-of-network-input-layers-must-be-equal-to-the-number-of-observation#comment_3191251

Open in MATLAB Online

Hi Anna,

For creating a critic netowk for PPO agent, we dont need the action information. Your criticNetwork has an additional channel for action inputs which is not required. Refer following documentation link to read about the requirements for rlValueFunction function : https://www.mathworks.com/help/reinforcement-learning/ug/create-policy-and-value-functions.html#mw_90f2c37b-252f-4ce1-902d-1ed95616f1ee

Your code can be modified as follows to create a critic network:

%% CRITIC

% Create the critic network layers

criticLayerSizes = [600,400]; %400,300

statePath = [

featureInputLayer(obsInfo.Dimension(1),'Normalization','none','Name', 'observation')

fullyConnectedLayer(criticLayerSizes(1), 'Name', 'CriticStateFC1', ...

'Weights',2/sqrt(obsInfo.Dimension(1))*(rand(criticLayerSizes(1),obsInfo.Dimension(1))-0.5), ...

'Bias',2/sqrt(obsInfo.Dimension(1))*(rand(criticLayerSizes(1),1)-0.5))

reluLayer('Name','CriticStateRelu1')

fullyConnectedLayer(criticLayerSizes(2), 'Name', 'CriticStateFC2', ...

'Weights',2/sqrt(criticLayerSizes(1))*(rand(criticLayerSizes(2),criticLayerSizes(1))-0.5), ...

'Bias',2/sqrt(criticLayerSizes(1))*(rand(criticLayerSizes(2),1)-0.5))

reluLayer('Name','CriticCommonRelu1')

fullyConnectedLayer(1, 'Name', 'CriticOutput',...

'Weights',2*5e-3*(rand(1,criticLayerSizes(2))-0.5), ...

'Bias',2*5e-3*(rand(1,1)-0.5))

];

% Connect the layer graph

criticNetwork = layerGraph(statePath);

figure(2)

plot(criticNetwork)

hold on

% Create critic representation

criticOptions = rlRepresentationOptions('Optimizer','adam','LearnRate',1e-3, ...

'GradientThreshold',1,'L2RegularizationFactor',2e-4);

critic = rlValueFunction(criticNetwork,env.getObservationInfo)%,env.getactInfo)%,...

In the above code, I have removed the actionPath layer and combined the statePath and commponPath layers. The actionPath layer can used to create the Actor required for the PPO agent using rlDiscreteCategoricalActor function. Refer to the following documentation to read about PPO Agents and the process of creating actor and critic network for the same: https://www.mathworks.com/help/reinforcement-learning/ug/proximal-policy-optimization-agents.html#mw_06cbb093-e547-44e0-9c06-61d829a6c113

Additionaly, MATLAB also has a rlPPOAgent function. You can read about the function here: https://www.mathworks.com/help/reinforcement-learning/ref/rl.agent.rlppoagent.html

Anna on 21 Jun 2024 at 16:04

Direct link to this comment

https://jmaab.mathworks.com/matlabcentral/answers/2129741-rlvaluefunction-error-the-number-of-network-input-layers-must-be-equal-to-the-number-of-observation#comment_3192721

⋮

  • Link

    Direct link to this comment

    https://jmaab.mathworks.com/matlabcentral/answers/2129741-rlvaluefunction-error-the-number-of-network-input-layers-must-be-equal-to-the-number-of-observation#comment_3192721

Hi Aniket,

Thank you so much for your help. I tried the things you recommended, including rlDiscreteCategoricalActor.

However the training didn't work properly. The episode reward didn't increase and remained under 0.

I thought my task is not proper to PPO algorithm, so I'm going to try other algorithms.

I appreciate your time and assistance, and I feel sorry about I couldn't solve the problem. Thank you:)

Sign in to comment.

More Answers (0)

Sign in to answer this question.

See Also

Categories

AI, Data Science, and StatisticsDeep Learning ToolboxSequence and Numeric Feature Data Workflows

Find more on Sequence and Numeric Feature Data Workflows in Help Center and File Exchange

Tags

  • rlvaluefunction
  • reinforcement learning
  • rl
  • rlppoagent
  • ppo

Products

  • Reinforcement Learning Toolbox

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

An Error Occurred

Unable to complete the action because of changes made to the page. Reload the page to see its updated state.


rlValueFunction error: The number of network input layers must be e... (7)

Select a Web Site

Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .

You can also select a web site from the following list

Americas

  • América Latina (Español)
  • Canada (English)
  • United States (English)

Europe

  • Belgium (English)
  • Denmark (English)
  • Deutschland (Deutsch)
  • España (Español)
  • Finland (English)
  • France (Français)
  • Ireland (English)
  • Italia (Italiano)
  • Luxembourg (English)
  • Netherlands (English)
  • Norway (English)
  • Österreich (Deutsch)
  • Portugal (English)
  • Sweden (English)
  • Switzerland
    • Deutsch
    • English
    • Français
  • United Kingdom(English)

Asia Pacific

Contact your local office

rlValueFunction error: The number of network input layers must be e... (2024)
Top Articles
35+ Grandma Tattoo Ideas To Honor Your Favorite Person!
49 Memorial Tattoo Ideas | Ever Loved
Balkonkraftwerk mit Speicher zum Preis einer Powerstation: Oubo Mentech im Test
Cups to Ounces Converter - Convert Cups to Oz
R.H. Shumway Seedsman (Firm) | Open Library
R.H. Shumway's annual illustrated garden guide and seed catalogue
20 Best Things to Do in Tiffin, OH - Travel Lens
White Round M 2
Black Adam Showtimes Near Regal Coldwater Crossing
Causa With Shrimp and Avocado Recipe
Gir Cow | Milk Per Day, Facts & Price
Gir Cow Price, Milk Per Day, Images, Food, Adaptability,
Latest Posts
Unlocking The Heartbeat of Generations: Grandchildren Tattoos for Grandma - TattooQuestions.com
11+ Grandma Tattoo Ideas That Will Blow Your Mind!
Article information

Author: Jerrold Considine

Last Updated:

Views: 5752

Rating: 4.8 / 5 (78 voted)

Reviews: 93% of readers found this page helpful

Author information

Name: Jerrold Considine

Birthday: 1993-11-03

Address: Suite 447 3463 Marybelle Circles, New Marlin, AL 20765

Phone: +5816749283868

Job: Sales Executive

Hobby: Air sports, Sand art, Electronics, LARPing, Baseball, Book restoration, Puzzles

Introduction: My name is Jerrold Considine, I am a combative, cheerful, encouraging, happy, enthusiastic, funny, kind person who loves writing and wants to share my knowledge and understanding with you.