Wind-fed accretion onto magnetised neutron stars drives long-term spin variability that can be used to trace angular-momentum transfer in structured stellar outflows. We analyse five years of TESS optical timing to measure the spin evolution of the wind-fed X-ray pulsar 4U 1954+319, which hosts the slowest known accretion-powered pulsar. Coherent hour-scale pulsations are detected in all eight TESS sectors. The optical spin period evolves non-monotonically from $5.51\pm0.06$ to $5.72\pm0.06$ h and then returns to $5.55\pm0.05$ h, consistent with a mid-2023 break and a torque reversal from spin-down to spin-up ($\kern1.5pt\dot{\!P}_1=+0.68\pm0.08$ s d$^{-1}$ to $\kern1.5pt\dot{\!P}_2=-2.16\pm0.35$ s d$^{-1}$). The observed $\kern1.5pt\dot{\!P}$ corresponds to torques of $N_1=(\!-\!1.2\pm0.1)\times10^{32}$ dyn cm and $N_2=(\!+\!3.6\pm0.6)\times10^{32}$ dyn cm for a canonical moment of inertia $I=10^{45}$ g cm$^2$, while the associated spin-down power remains negligible compared with the X-ray luminosity. Within the quasi-spherical settling accretion regime, a change in mass accretion rate by a factor of ${\sim}3.6$ is sufficient, at the order-of-magnitude level, to account for the measured torque ratio. The inferred torque scale is compatible with a surface dipole field $B\approx(0.5-5)\times10^{12}$ G, with the magnetospheric radius remaining well inside co-rotation and no requirement for a propeller transition at the inferred luminosities. No clear contemporaneous hard-X-ray brightening is apparent in the BAT monitoring, which is consistent with an origin of the optical pulsations in reprocessing of the pulsed high-energy emission. Overall, the data favour a picture in which 4U 1954+319 is a long-period, near-equilibrium wind-fed pulsar whose torque reversals can arise from moderate changes in settling accretion, without invoking magnetar-strength fields or persistent propeller transitions. They also show that long-baseline optical timing can provide useful constraints on torque balance in faint wind-fed pulsars.