Accurate trajectory prediction of traffic agents is critical for enabling safer and more reliable autonomous driving, particularly in urban driving scenarios where close-range interactions are most safety critical. High-definition (HD) and standard-definition (SD) maps play a vital role in this process by providing lane topology and directional cues for forecasting agent movements. However, HD maps are expensive and resource-intensive to create, often requiring specialized sensors, while SD maps lack the precision needed for reliable autonomous navigation. To address this, we propose a novel framework for trajectory prediction that leverages online reconstruction of HD maps using vehicle-mounted cameras, offering a scalable and cost-effective alternative. Our method achieves improvements in predicting accuracy, particularly in close-range scenarios, the most crucial for urban driving, while also performing robustly in settings without pre-built maps. Furthermore, we introduce a new safety-aware evaluation metric that incorporates heuristic weights based on agent relevance and distance, enhancing traditional metrics like Brier-minFDE with a stronger focus on safety-critical scenarios. Extensive experiments demonstrate that our approach outperforms state-of-the-art map-less methods, particularly in close-range prediction, while our proposed metric establishes a more domain-relevant benchmark for assessing trajectory prediction in autonomous driving.