To investigate the initial process of Joule heating in semiconductors microscopically and quantitatively, we developed a theoretical framework for the ab initio evaluation of the carrier energy relaxation in semiconductors under a high electric field using a combination of the two-temperature model and the Boltzmann equation. We employed the method for bulk silicon as a typical example. Consequently, we found a remarkable difference in the energy relaxation processes of the electron and hole carriers. The longitudinal acoustic and optical phonons at the zone boundary contribute to the energy relaxation of electron carriers, whereas they contribute negligibly to that of the hole carriers. In addition, at the band edge, the energy relaxation rate is maximized for the electron carriers, whereas it is suppressed for the hole carriers. These differences stem from the presence/absence of intervalley scattering processes and isotropic/anisotropic band structures in electrons and holes. Our results lay the foundation for controlling the thermal generation in semiconductors by material design.