Behavior varies from trial to trial even when the stimulus is maintained as constant as possible. One of the most important questions in neuroscience concerns the origin of this variability. In many models, it is attributed to noise within the brain, often in the form of independent Poisson variability in spike trains, or variations thereof. Here, we show that suboptimal inference caused by the deterministic approximations of the statistical structure of the sensory inputs provides another major cause of variability. Importantly, we argue that in most tasks of interest, and particularly complex ones, this cause of variability is likely to be the dominant component of behavioral variability. This perspective explains a variety of intriguing observations, including why variability appears to be larger on the sensory than on the motor side, and why our sensors are sometimes surprisingly unreliable.