The ultimate scaling limit in ferroelectric switching has been attracting broad attention in the fields of materials science and nanoelectronics. Despite immense efforts to scale down ferroelectric features, however, only few materials have been shown to exhibit ferroelectricity at the unit-cell level. Here we report a controllable unit-cell-scale domain in brownmillerite oxides consisting of alternating octahedral/tetrahedral layers. By combining atomic-scale imaging and in situ transmission electron microscopy, we directly probed sub-unit-cell-segmented ferroelectricity and investigated their switching characteristics. First-principles calculations confirm that the phonon modes related to oxygen octahedra are decoupled from those of the oxygen tetrahedra in brownmillerite oxides, and such localized oxygen tetrahedral phonons stabilize the sub-unit-cell-segmented ferroelectric domain. The unit-cell-wide ferroelectricity observed in our study could provide opportunities to design high-density memory devices using phonon decoupling.