We demonstrate that the near horizon symmetries of black holes in Einstein--Yang-Mills (EYM) theory are generated by an infinite-dimensional algebra that contains, in addition to supertranslations and superrotations, a non-Abelian loop algebra. This means that the Virasoro--Kac-Moody structure of EYM in asymptotically flat spacetimes has an exact analog in the near horizon region.