Short Description of the Tutorial:

In many real-world applications, agents must make sequential decisions in environments where conditions are subject to change due to various exogenous factors. These non-stationary environments pose significant challenges to traditional decision-making models, which typically assume stationary dynamics. Non-stationary Markov decision processes (NS-MDPs) offer a framework to model and solve decision problems under such changing conditions. In this tutorial, we will introduce the fundamental concepts of non-stationary MDPs, discuss prior work and formulations, algorithmic frameworks for solving such problems, recent trends, and directions for future work. We will also discuss the NS-Gym, the first simulation toolkit designed explicitly for NS-MDPs, integrated within the popular Gymnasium framework.

Organizer:

Ayan Mukhopadhyay is a senior research scientist at Vanderbilt University, leading several NSF, DoE, and DoT-funded projects. His research focuses on developing scalable and adaptive algorithmic methods for decision-making under uncertainty for positive societal impact. Ayan’s work has won best paper awards at multiple top-tier venues, such as IJCAI, ICCPS, INFORMS, ICLR’s AI for Good Workshop, and the Google Research AI for Good program. His work has also been successfully deployed by multiple community partners in several countries. Before this, he was a post-doctoral fellow at Stanford University, where he was awarded the Center of Automotive Research post-doctoral fellowship. He holds a doctorate in Computer Science from Vanderbilt University. His doctoral dissertation on scalable algorithmic approaches for semi-Markovian decision processes was nominated for the IFAAMAS Victor Lesser Distinguished Dissertation Award.