Catenary insulator defects in railway UAV inspection images are usually small in scale and weak in texture, and are easily affected by complex backgrounds such as cables, towers and vegetation. These factors often lead to false positives, missed detections and unstable bounding-box localization. To address these problems, a background-aware feature enhancement YOLO method, termed BAFE-YOLO, is proposed for insulator defect detection. First, a residual adaptive feature fusion module, termed AFF-Res, is designed in the neck cross-scale fusion stage to replace the original Concat fusion nodes, thereby enhancing the cross-scale representation of weak and small defects. Second, a background-aware feature enhancement module, termed BAFE-P3, is introduced into the P3 high-resolution detection branch output by the neck to perform low-disturbance feature calibration and alleviate feature confusion between weak-texture defects and complex backgrounds. Finally, a lightweight distance-aware regression loss, LiteRegLoss, is constructed to improve the bounding-box localization stability of small-scale defects. Experiments on the public Insulator-Defect Detection dataset show that BAFE-YOLO achieves a Precision of 94.68%, a Recall of 88.93%, an mAP@0.5 of 92.03%, and an mAP@0.5:0.95 of 64.18%. Compared with the baseline model under the same training settings, the false discovery rate of pollution-flashover defects decreases from 18.92% to 8.76%, and the missed detection rate of broken defects decreases from 12.50% to 4.17%. The AP@0.5 values for pollution-flashover and broken defects are improved by 3.00 and 2.60 percentage points, respectively. The proposed method improves the detection of small and weak insulator defects, such as breakage and pollution flashover, in UAV-based railway inspection images, while reducing false detections, missed detections, and localization deviations under complex backgrounds.