arXiv:2611.24271v1 Announce Type: new Abstract: Emerging integrated sensing and communications (ISAC) systems reuse the communications signal for radar-like sensing to provide environmental awareness. However, this integrated design imposes conflicting constraints on the modulation format, as communications systems use higher-order modulation formats while sensing systems use unit-modulus signals, making constellation shaping a key performance driver in ISAC systems. This paper develops a unified analytical and experimental framework for constellation shaping in monostatic orthogonal frequency division multiplexing ISAC systems and systematically analyzes its interaction with the three canonical sensing channel estimators (SCEs): the matched filter, the reciprocal filter, and the Wiener filter (WF). We begin by deriving constellation-dependent image signal-to-interference-and-noise ratio (ISINR) expressions for the SCEs that account for practical effects such as windowing and multi target scenarios. Based on this, we derive the corresponding entropy-maximizing channel input distributions. We generalize an autoencoder framework to optimize geometric, probabilistic, including conventional and generalized low-complexity probabilistic amplitude shaping (PAS), and joint constellation shaping across all three SCEs. Extensive simulations and hardware-in-the-loop experiments validate the derived ISINR and show that sensing performance is governed by the SCE choice, while communications performance is dominated by the shaping method. Notably, the WF combined with joint shaping achieves the best ISAC trade-off. Low-complexity PAS variants closely approach this performance, offering a flexible trade-off and outperforming legacy modulation formats.