A Stochastic Game Theory Framework For Multi-Agent Decision-Making In Clinical Healthcare Settings

Abstract

The study focused on a stochastic game theory framework for multi-agent decision-making in clinical healthcare settings, driven by advancement in a clinical and simulation-based environment. Despite advancements in single-agent models, there remains a notable knowledge gap in incorporating multi-agent strategic interactions within stochastic frameworks that adequately addressed uncertainty. The objectives of the study were to: develop a stochastic game-theoretic framework for modeling dynamic, interactive decision-making in clinical healthcare settings, incorporate Stochastic game models based on Markov Decision Processes (MDPs) and Partially Observable Markov Decision Processes (POMDPs), and evaluate outcomes under realistic clinical constraints, including limited resources, diagnostic uncertainty, and time-sensitive interventions, using simulation-based analysis. This study addressed patient progression through distinct health states (critical, serious, stable and recovered), influenced by healthcare interventions and a simulated patient summary table. The study considered the patients as the principal agents; characterized by initial severity (Mild, Moderate, Severe) and risk (Low, Medium, High) which directly influenced their initial states and potential health progression. The study adopted simulation-based analysis framework which was implemented in Python. The computed value function, expected reward for each health state, derived via the Bellman equation. The simulated-based analysis was conducted using  transition probability matrices on table 2 and table 4 with a discount factor of 0.95. The method of data analysis was based on Markov Decision Process and Partially Observable Markov Decision Processes. Two (4) matrices were created from simulated-based data for transition probability on “treat” and “wait” actions where the analysis revealed that all states eventually absorbed into ‘recovery with probability 1.0. Based on Markov Decision Process, the expected time to recovery from Critical was 3.25 compared to 6.0 from Serious and 7.75 from Stable. Using a reward structure penalizing critical states (0) and rewarding recovery (+8.5), the expected cumulative reward from each initial state was computed as: Critical = 3.25, Serious = 6.0, Stable = 7.75. Also, using Partially Observable Markov Decision Process on table 9 revealed the values of belief update of 20 stimulated patients which ranged approximately from 152.87 to 167.59, showing moderate variability across patient beliefs The relative closeness of patient value of belief, (v(b)) on table 9 indicated that each patient health state had the potential to recover with an average payoff of 6.8, improving recovery odds by 15-20%. However, the study recommended that the healthcare agents and systems should improve clinical decision-making under uncertainty by applying Markov Decision Processes and Partially Observable Markov Decision Processes to minimize patient times spent in critical or serious health states, delays and costs of care in order to ensure evidenced-based support and overall system performance.