Deep Reinforcement Learning for Downlink User Scheduling in LEO Satellite Massive MIMO Systems

Low earth orbit (LEO) satellite communication system is expected to play a key role in the 6G wireless networks. Due to the limited communication resource and inter-user interference (IUI), the user equipments (UEs) must be well scheduled to maximize spectral efficiency. However, rapid channel variations and high mobility in LEO system make the conventional user scheduling strategies relying on instantaneous channel state information impractical. To address this challenge, this paper proposes a deep reinforcement learning (DRL) framework based on batch normalization in CrossQ algorithm to optimize the downlink user scheduling in a LEO satellite massive multi-input multi-output (MIMO) system. The scheduling task is modeled as a Markov decision process (MDP), where the observation space integrates high-dimensional features including large-scale fading (LSF) coefficients, channel matrix, and temporal features capturing the orbital progress of the satellite. Unlike traditional DRL methods, the CrossQ-based algorithm applies batch normalization (BN) within the critic networks to eliminate the need for slowly-moving target networks, fostering training stability and sample efficiency. A continuous priority score vector is mapped to discrete scheduling decisions through a top-z selection logic, allowing the agent to effectively identify spatially orthogonal user subsets to suppress the IUI. To guide the agent toward the true optimization direction, a differential reward mechanism is designed. Simulation results demonstrate that the proposed CrossQ-based scheduler consistently achieves nearly optimal performance at different signal-to-noise ratios (SNR), reaching at least 91% and 94% of the Monte Carlo search ceiling under different user densities. Moreover, the trained model provides scheduling strategy in millisecond inference time, making it a practical and robust solution for future 6G non-terrestrial networks.