We present a search for "hyper-compact" star clusters in the Milky Way using a combination of Gaia and the Dark Energy Camera Legacy Survey (DECaLS). Such putative clusters, with sizes of ~1 pc and containing 500-5000 stars, are expected to remain bound to intermediate-mass black holes (Mbh~10^3-10^5 M-sun) that may be accreted into the Milky Way halo within dwarf satellites. Using the semi-analytic model SatGen we find an expected ~100 wandering intermediate-mass black holes with if every infalling satellite hosts a black hole. We do not find any such clusters in our search. Our upper limits rule out 100% occupancy, but do not put stringent constraints on the occupation fraction. Of course, we need stronger constraints on the properties of the putative star clusters, including their assumed sizes as well as the fraction of stars that would be compact remnants.