As an energy filter in transmission electron microscopy has improved the precision of structural determination by filtering out inelastic imaging electrons, the introduction of the energy filter to ultrafast electron microscopy (UEM) can advance the time resolution to the domain of atomic motion. Imaging transient structures with femtosecond temporal precision was made possible by gating imaging electrons of narrow energy distribution from dense chirped photoelectron packets, thus, typically posing picosecond duration. Presented are the concept and proof-of-principle demonstration of the energy-filtered UEM achieving the temporal resolution limited by the briefness of an optical excitation pulse, i.e., 500 fs in this study, filming ultrafast insulator-to-metal phase transition of vanadium dioxide. Our approach leads the access of electron microscopy to the timescale of elementary nuclear motions visualizing the onset of structural dynamics of matter at nanoscales. Also, we have uniquely combined cathodoluminescence (CL) with UEM. A synergistic use of the two methodologies is essential because CL and UEM are the highly sought-after approaches with required spectral and spatiotemporal sensitivity, respectively. For nitrogen-related color centers in nanodiamonds, we demonstrate the measurement of CL lifetime with the local sensitivity of 50 nm and the time resolution of 100 ps. It is revealed that the emitting state of the H3 color center (N2V) can be populated by hole transfer from the NV center excited by free carrier transfer diffusing across diamond lattices upon electron beam excitation. The technical advance achieved in this study will deliver new concepts for specific control over energy conversion relevant to quantum dots and single photon sources.