The “speech envelope” is often used as an acoustic proxy for neural rhythm. The problem is its assumption that the unfiltered, broadband signal can satisfactorily model neural modulation in the auditory pathway (and beyond). However, the auditory system does not function as a passive transducer but rather decomposes and segregates the signal into an array of tonotopically organized frequency channels. This modulation filtering results in a partitioning of slow (3–20 Hz) neural modulation patterns across the tonotopic axis that bear only a passing resemblance to the broadband speech envelope. Such polychromatic diversity (in frequency, magnitude, and phase) of auditory modulation patterns is critical for decoding the speech signal, as it highlights critical linguistic properties such as articulatory-acoustic and prosodic features important for decoding and understanding spoken language. The low-frequency modulation patterns associated with high-frequency (>2 kHz) auditory channels are especially important for prosodic processing and consonant discrimination, both key for speech intelligibility, especially in adverse listening conditions and among the hard of hearing.