In engineering scenarios such as deep rock excavation and shallow instability of slopes, rock masses are commonly subjected to a unilateral confined stress state, with one side exposed to a free surface and the other constrained by the in-situ rock mass. The resulting failure behavior differs significantly from that under conventional uniaxial compression. However, existing studies have paid insufficient attention to the quantitative variation of fracture angles under unilateral confined conditions. In this study, numerical simulations were conducted using the discrete element software PFC^2D. Four rock-forming materials, namely potassium feldspar, plagioclase, quartz, and mica, were selected. First, direct shear tests under different normal stresses were performed to inversely determine the internal friction angle and cohesion of each material. Subsequently, uniaxial compression (UC) and unilateral confined compression (UCC) numerical tests were carried out to systematically compare the stress–strain responses, spatiotemporal evolution of cracks, and failure modes under the two loading conditions. A calculation formula for the fracture angle under unilateral confined conditions was further established. The results show that, compared with UC, rocks under UCC exhibit pronounced post-peak dilation, earlier crack initiation, and lower peak strength, while the fracture development process lasts longer. Cracks in UCC specimens are mainly concentrated near the free surface, and the macroscopic failure pattern presents a “>”-shaped morphology, whereas UC specimens exhibit an “X”-shaped conjugate shear failure pattern. Under both loading conditions, the macroscopic shear bands are dominated by tensile cracks; however, the proportion of shear cracks increases under UCC, indicating an enhanced tendency toward shear failure. The fracture angle of rocks under UCC shows a good linear relationship with the internal friction angle, based on which a prediction model was proposed. Furthermore, conventional triaxial compression and unilateral confined compression laboratory tests were conducted on standard red sandstone specimens. The relative error between the measured fracture angles and the values predicted by the proposed formula was less than 1%, verifying the applicability of the model. The findings provide a theoretical basis for unilateral confined instability analysis and support design of surrounding rock in deep roadways and shallow slope layers.