A long line of research shows that voters frequently evaluate objective conditions through a perceptual screen, seeing a stronger economy and more peaceful world when their party is in power. We know less about how and why these partisan perceptual differences have changed over recent history, however. This paper combines ANES measures of retrospective evaluations from 1956 to 2016 and shows that partisan differences (1) have increased significantly over the past few decades across all types of assessments; (2) are greatest, and have changed the most, amongst the most politically aware; and (3) closely track changes in elite polarization over this time period. The extent of partisan disagreement in retrospective evaluations is thus not constant, but rather contingent on attributes of the voter and the political context. Greater political awareness and more polarized politicians result in larger partisan perceptual differences, as the most engaged citizens are the most likely to receive and internalize cues about the state of the world from their party’s elites.