The existence and stability of atoms rely on the fact that neutrons are more massive than protons. The measured mass difference is only 0.14\% of the average of the two masses. A slightly smaller or larger value would have led to a dramatically different universe. Here, we show that this difference results from the competition between electromagnetic and mass isospin breaking effects. We performed lattice quantum-chromodynamics and quantum-electrodynamics computations with four nondegenerate Wilson fermion flavors and computed the neutron-proton mass-splitting with an accuracy of $300$ kilo-electron volts, which is greater than $0$ by $5$ standard deviations. We also determine the splittings in the $\Sigma$, $\Xi$, $D$ and $\Xi_{cc}$ isospin multiplets, exceeding in some cases the precision of experimental measurements.