arXiv:2611.14232v1 Announce Type: new Abstract: Base stations account for a dominant share of energy consumption and operational carbon emissions in radio access networks. In O-RAN deployments with heterogeneous energy sources, however, minimizing energy consumption alone does not necessarily minimize carbon emissions. In this work, we investigate carbon-aware DRL for RU sleep control. Under the adopted source-matched operational accounting model, renewable-matched RUs are assigned zero direct operational emissions, whereas non-renewable-supplied RUs incur operational emissions. The problem is formulated as a Markov decision process and evaluated using two carbon-aware reward formulations and a conventional energy--QoS reward baseline. We implement and compare TD3, PPO, and a heuristic method under realistic mobility and 3GPP channel models across multiple renewable-matched RU layouts and traffic loads. Simulation results show that TD3 learns more energy-efficient activation policies than PPO, achieving approximately 3%--6% lower energy consumption in representative scenarios. TD3 also achieves 8%--12% lower operational emissions than PPO in scenarios with a low proportion of renewable-matched RUs and up to a 20% reduction relative to the heuristic baseline under moderate renewable availability and high traffic load. Moreover, the joint energy,carbon and QoS reward achieves approximately 12%--28% lower operational emissions than the energy and QoS reward, while maintaining comparable energy consumption.