This study presents a theoretical analysis of NEPCM behavior under natural convection within a porous octagonal cavity featuring a Y-shaped fin. The study primarily examines the impact of different fin heights on convection-driven heat transfer. To identify the optimal fin height and evaluate the effects of key governing parameters, a hybrid machine learning framework was implemented, incorporating a genetic algorithm (GA) to fine-tune weight and bias parameters. Due to the highly nonlinear nature of the dataset, initial predictions exhibited suboptimal Mean Squared Error (MSE) and R2 values; however, GA optimization significantly improved predictive accuracy, highlighting its effectiveness in refining parameter selection. The governing dimensionless equations are solved using the finite difference method, while a parametric analysis explores the influence of fusion temperature, Rayleigh number, Hartmann number, and Darcy number on fluid flow behavior within the system. Results demonstrate that, regardless of fin height configuration, increasing the nanoparticle volume fraction substantially enhances heat transfer rate. Notably, when the fin height is set at 0.2, elevating the nanoparticle concentration from 1% to 5% results in a substantial 52.66% improvement in thermal performance. A higher heat transfer rate is observed at a fusion temperature of 0.6 compared to other fusion temperatures. A fin height of 0.2 achieves a superior heat transfer rate compared to other fin height configurations. Machine learning analysis indicates that, at this height, thermal buoyancy forces dominate at 29.7%, surpassing those observed at fin heights of 0.4, 0.5, and 0.6 by 8.3%, 5.9%, and 7.4%, respectively. This study provides crucial insights into enhancing NEPCM-based heat transfer systems while highlighting the effectiveness of GA-integrated machine learning in accurately predicting complex, nonlinear thermal behavior.