- Notifications
You must be signed in to change notification settings - Fork4.4k
POCA trainer#5005
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
POCA trainer#5005
Uh oh!
There was an error while loading.Please reload this page.
Conversation
This reverts commit292b6ce.
Integrate into CC
* simple rl multiagent env* runs but does not train* assemble terminal steps* seems to train* fix final reward* Merge changes* fix multiple discrete actions* Lots of small fixes for multiagent env* Fix just_died* Add simple RL tests* Add LSTM simple_rl for COMA* adding comments to multiagent rl* Address commentsCo-authored-by: Ervin Teng <ervin@unity3d.com>
erge branch 'develop-poca-trainer' into develop-coma2-trainer
ervteng left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
LGTM, will wait for at least one more reviewer
vincentpierre left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I approve once all comments have been resolved.
Including those that were wrongfully marked as outdated by github.
Uh oh!
There was an error while loading.Please reload this page.
| ) | ||
| returnvalue_outputs,critic_mem_out | ||
| defforward( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Remove this method. It has no reason to be public.
| # Convert to tensors | ||
| current_obs= [ModelUtils.list_to_tensor(obs)forobsincurrent_obs] | ||
| group_obs=GroupObsUtil.from_buffer(batch,n_obs) | ||
| group_obs= [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Some of my comments got lost. Please review them in the conversation tab :#5005 (comment)
| if ( | ||
| BufferKey.GROUPMATE_REWARDSinmini_batch | ||
| andBufferKey.GROUP_REWARDinmini_batch | ||
| ): | ||
| ifself.add_groupmate_rewards: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Invert these 2 ifs. No need to check the first one if there are no groumaterewards
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
These if conditions could be better:
if self.add_groupmate_rewards and BufferKey.GROUPMATE_REWARDS in mini_batch : do the groupmate reward
if BufferKey.GROUP_REWARD in mini_batch : Do the group reward
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Updated
Uh oh!
There was an error while loading.Please reload this page.
| returnrsa,x_self_encoder | ||
| @staticmethod | ||
| defencode_observations( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Stand by my statement, make create_residual_self_attention a module with encode_observations its forward method
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Call it ObservationEncoder
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Uh oh!
There was an error while loading.Please reload this page.
…nologies/ml-agents into develop-coma2-trainer
* Make Observation Encoder a module* Fix copy normalize
Uh oh!
There was an error while loading.Please reload this page.
Proposed change(s)
This PR adds the POCA trainer and associated tests. In addition it makes changes to the extrinsic reward provider to enable team-based rewards to work.
Useful links (Github issues, JIRA tickets, ML-Agents forum threads etc.)
PR for documentation - to be merged after this one#5056
Explanation of some of the design choices:
Types of change(s)
Checklist
Other comments