The modern technologies of light-emitting devices, light harvesting, and quantum information processing requires understanding of structure-function relations at spatial scales below the optical diffraction limit and the timescales of energy and information flows [1]. Here, we uniquely combine cathodoluminescence (CL) with ultrafast electron microscopy (UEM) [2-4]. A synergistic use of the two methodologies is essential because CL and UEM pose with required spectral and spatiotemporal sensitivity, respectively [4,5]. For color centers in nanodiamonds, we demonstrate the measurement of CL lifetime with the local sensitivity of 50 nm and the time resolution of 100 ps. It is revealed that the emitting states of the color centers can be populated by charge transfer among the color centers across diamond lattices upon high-energy electron beam excitation. The technical advance achieved in this study will deliver new concepts for specific control over energy conversion at nanoscales relevant to quantum dots and single-photon sources.