The absolute sets of local systems on a smooth complex algebraic variety are the subject of a conjecture of N. Budur and B. Wang based on an analogy with special subvarieties of Shimura varieties. An absolute set should be the higher-dimensional generalization of a local system of geometric origin. We show that the conjecture for absolute sets of simple cohomologically rigid local systems reduces to the zero-dimensional case, that is, to Simpson's conjecture that every such local system with quasi-unipotent monodromy at infinity and determinant is of geometric origin. In particular, the conjecture holds for this type of absolute sets if the variety is a curve or if the rank is two.