Repository files navigation

Linux	Mac OS X	Windows

This package provides a core interface for working withMarkov decision processes (MDPs) andpartially observable Markov decision processes (POMDPs).ThePOMDPTools package acts as a "standard library" for the POMDPs.jl interface, providing implementations of commonly-used components such as policies, belief updaters, distributions, and simulators.

Our goal is to provide a common programming vocabulary for:

1. Expressing problems as MDPs and POMDPs.
2. Writing solver software.
3. Running simulations efficiently.

POMDPs.jl integrates with other ecosystems:

• Python can be used to define and solve MDPs and POMDPs via thequickpomdps package or through tables directly viapyjulia.
• POMDPTools providestwo-way integration withCommonRLInterface and therefore with theJuliaReinforcementLearning packages.
• TheSymbolicMDPs package provides an interface to work with PDDL models.

For a detailed introduction, check out ourJulia Academy course! For help, please post inGitHub Discussions tab. We welcome contributions from anyone! SeeCONTRIBUTING.md for information about contributing.

POMDPs.jl and associated solver packages can be installed usingJulia's package manager. For example, to install POMDPs.jl and the QMDP solver package, type the following in the Julia REPL:

using Pkg; Pkg.add("POMDPs"); Pkg.add("QMDP")

To run a simple simulation of the classicTiger POMDP using a policy created by the QMDP solver, you can use the following code (note that POMDPs.jl is not limited to discrete problems with explicitly-defined distributions like this):

using POMDPs, QuickPOMDPs, POMDPTools, QMDPm=QuickPOMDP(    states= ["left","right"],    actions= ["left","right","listen"],    observations= ["left","right"],    initialstate=Uniform(["left","right"]),    discount=0.95,    transition=function (s, a)if a=="listen"returnDeterministic(s)# tiger stays behind the same doorelse# a door is openedreturnUniform(["left","right"])# resetendend,    observation=function (s, a, sp)if a=="listen"if sp=="left"returnSparseCat(["left","right"], [0.85,0.15])# sparse categorical distributionelsereturnSparseCat(["right","left"], [0.85,0.15])endelsereturnUniform(["left","right"])endend,    reward=function (s, a)if a=="listen"return-1.0elseif s== a# the tiger was foundreturn-100.0else# the tiger was escapedreturn10.0endend)solver=QMDPSolver()policy=solve(solver, m)rsum=0.0for (s,b,a,o,r)instepthrough(m, policy,"s,b,a,o,r", max_steps=10)println("s:$s, b:$([s=>pdf(b,s)for sinstates(m)]), a:$a, o:$o")global rsum+= rendprintln("Undiscounted reward was$rsum.")

For more examples and examples with visualizations, reference theExamples andGallery of POMDPs.jl Problems sections of the documentaiton.

In addition to the above-mentionedJulia Academy course, detailed documentation and examples can be foundhere.

Many packages use the POMDPs.jl interface, including MDP and POMDP solvers, support tools, and extensions to the POMDPs.jl interface. POMDPs.jl and all packages in the JuliaPOMDP project are fully supported on Linux. OSX and Windows are supported for all native solvers*, and most non-native solvers should work, but may require additional configuration.

POMDPs.jl itself contains only the core interface for communicating about problem definitions; these packages contain implementations of commonly-used components:

Package	Build	Coverage
POMDPTools (hosted in this repository)
ParticleFilters

Many models have been implemented using the POMDPs.jl interface for various projects. This list contains a few commonly used models:

Package	Build	Coverage
POMDPModels
LaserTag
RockSample
TagPOMDPProblem
DroneSurveillance
ContinuumWorld
VDPTag2
RoombaPOMDPs (Roomba Localization)

Package	Build/Coverage	Online/ Offline	Continuous States - Actions	Rating3
DiscreteValueIteration		Offline	N-N	★★★★★
LocalApproximationValueIteration		Offline	Y-N	★★
GlobalApproximationValueIteration		Offline	Y-N	★★
MCTS (Monte Carlo Tree Search)		Online	Y (DPW)-Y (DPW)	★★★★

Package	Build/Coverage	Online/ Offline	Continuous States-Actions-Observations	Rating3
QMDP (suboptimal)		Offline	N-N-N	★★★★★
FIB (suboptimal)		Offline	N-N-N	★★
BeliefGridValueIteration		Offline	N-N-N	★★
SARSOP*		Offline	N-N-N	★★★★
NativeSARSOP		Offline	N-N-N	★★★★
ParticleFilterTrees (SparsePFT, PFT-DPW)		Online	Y-Y2-Y	★★★
BasicPOMCP		Online	Y-N-N1	★★★★
ARDESPOT		Online	Y-N-N1	★★★★
AdaOPS		Online	Y-N-Y	★★★★
MCVI		Offline	Y-N-Y	★★
POMDPSolve*		Offline	N-N-N	★★★
IncrementalPruning		Offline	N-N-N	★★★
POMCPOW		Online	Y-Y2-Y	★★★
AEMS		Online	N-N-N	★★
PointBasedValueIteration		Offline	N-N-N	★★

¹: Will run, but will not converge to optimal solution

²: Will run, but convergence to optimal solution is not proven, and it will likely not work well on multidimensional action spaces. See alsohttps://github.com/michaelhlim/VOOTreeSearch.jl.

Package	Build/Coverage	Continuous States	Continuous Actions	Rating3
TabularTDLearning		N	N	★★
DeepQLearning		Y1	N	★★★

¹: For POMDPs, it will use the observation instead of the state as input to the policy.

³ Subjective rating; File an issue if you believe one should be changed

• ★★★★★: Reliably Computes solution for every problem.
• ★★★★: Works well for most problems. May require some configuration, or not support every edge of interface.
• ★★★: May work well, but could require difficult or significant configuration.
• ★★: Not recently used (unknown condition). May not conform to interface exactly, or may have package compatibility issues
• ★: Not known to run

*These packages require non-Julia dependencies

If POMDPs is useful in your research and you would like to acknowledge it, please cite thispaper:

@article{egorov2017pomdps,  author  = {Maxim Egorov and Zachary N. Sunberg and Edward Balaban and Tim A. Wheeler and Jayesh K. Gupta and Mykel J. Kochenderfer},  title   = {{POMDP}s.jl: A Framework for Sequential Decision Making under Uncertainty},  journal = {Journal of Machine Learning Research},  year    = {2017},  volume  = {18},  number  = {26},  pages   = {1-5},  url     = {http://jmlr.org/papers/v18/16-300.html}}