Deep Reinforcement-based Phase Design for RIS-aided Massive MIMO Systems

The burgeoning technology of Reconfigurable Intelligent Surface (RIS) holds the promise of facilitating high data rates while simultaneously minimizing costs and energy consumption. Moreover, its capability to strategically reflect signals from the Base Station (BS) to users addresses the prevalent blockage challenges in urban areas. Given these advantages, RIS emerges as a potentially energy-efficient and cost-effective adjunct to traditional massive Multiple Input Multiple Output (MIMO) systems. The design and implementation of a RIS-aided system present unique challenges, particularly in the acquisition of Channel State Information (CSI). The inherent passive nature of RIS limits its ability to actively send and receive pilot signals, making traditional methods of CSI estimation impractical. This thesis introduces a novel approach to address the challenge of imperfect channel knowledge in the context of RIS design. Leveraging the capabilities of deep Reinforcement Learning (RL), the proposed methodology aims to enhance the adaptability and performance of RIS systems by incorporating learning mechanisms that enable efficient decision-making in response to dynamically changing and uncertain channel conditions. Through the integration of deep RL techniques, the thesis endeavors to optimize RIS configurations, mitigating the impact of imperfect channel knowledge and advancing the practical implementation of RIS technology in real-world scenarios.

First, a deep RL-based RIS design for a Multi-user Multiple Input and Single Output (MU-MISO) system is proposed. In this case, to reduce the overhead of channel estimation and RIS design, it is assumed that both the active beamforming at the BS and passive beamforming at the RIS are simultaneously designed based on the Statistical Channel State Information (S-CSI). A closed-form expression for the downlink data rate is derived and the sum rate maximization problem is solved using Proximal Policy Optimization (PPO).

Second, a two-timescale design for a Cell-free Massive MIMO (CF-mMIMO) system is proposed. Initially, the uplink channel is estimated through the LMMSE method to estimate the aggregated channel from users to access points. Subsequently, based on the estimated channel, each Access Point (AP) independently detects the uplink data transmitted by each user using the Maximum Ratio Combining (MRC) method which utilizes the instantaneous information of the aggregated channels, that combine the direct and reflected channels from the RISs and relays the information to the Central Processing Unit (CPU). To alleviate the fronthaul load between APs and the CPU, it is assumed that the CPU has access solely to the S-CSI. Subsequently, the closed-form expressions of the achievable uplink spectral efficiency (SE) is derived, which is a function of S-CSI elements including distance-dependent path loss, Rician factors, number of RIS elements, and AP antennas. Then, phase shifts of the RISs are optimized to maximize the uplink SE of the users, utilizing Soft Actor-Critic (SAC) which is a deep RL method, and relying on the derived closed-form expressions. Simulation results show that despite the presence of imperfect CSI, the use of RISs in cell-free systems can lead to significant performance improvement.

Third, a cell-free network merged with active RIS is investigated. Based on the imperfect CSI, the aggregated channel from the user to the AP is initially estimated using the Linear Minimal Mean Square Error (LMMSE) technique. The CPU then detects uplink data from individual users through the utilization of the MRC approach, relying on the estimated channel. Then, a closed-form expression for uplink Spectral Efficiency (SE) is derived which demonstrates its reliance on S-CSI alone. The amplitude gain of each active RIS element is derived in a closed-form expression as a function of the number of active RIS elements, the number of users, and the size of each reflecting element. A SAC algorithm is utilized to design the phase shift of the active RIS to maximize the uplink SE. Simulation results emphasize the robustness of the proposed SAC algorithm, showcasing its effectiveness in cell-free networks under the influence of imperfect CSI.