The kagome superconductor AV$_3$Sb$_5$ (A=K, Rb, Cs) is a rare platform to explore the interplay between topology, geometrical frustration and symmetry-breaking orders. In addition to the charge density wave below $T_{CDW}\sim94$~K and superconductivity below $T_{SC}\sim3$~K, recent surface sensitive studies of CsV$_3$Sb$_5$ find evidence of 1$\times$4 superlattices below $T^{*}\sim60$~K. Interestingly, this unidirectional $4a_0$ phase may intertwine with charge density wave and superconductivity and possibly responsible for spectroscopic and transport anomalies below $T^{*}$. Here, combining high-resolution X-ray diffraction and scanning tunneling microscopy, we demonstrate that the 1$\times$4 superstructure emerges uniquely on the surface and hence exclude the $4a_0$ phase as the origin of $T^{*}$ anomaly in the bulk CsV$_3$Sb$_5$.