Despite prolonged interest in comparing brain size and behavioral proxies of “intelligence” across taxa, the adaptive and cognitive significance of brain size variation remains elusive. Central to this problem is the continued focus on hominid cognition as a benchmark and the assumption that behavioral complexity has a simple relationship with brain size. Although comparative studies of brain size have been criticized for not reflecting how evolution actually operates, and for producing spurious, inconsistent results, the causes of these limitations have received little discussion. We show how these issues arise from implicit assumptions about what brain size measures and how it correlates with behavioral and cognitive traits. We explore how inconsistencies can arise through heterogeneity in evolutionary trajectories and selection pressures on neuroanatomy or neurophysiology across taxa. We examine how interference from ecological and life history variables complicates interpretations of brain–behavior correlations and point out how this problem is exacerbated by the limitations of brain and cognitive measures. These considerations, and the diversity of brain morphologies and behavioral capacities, suggest that comparative brain–behavior research can make greater progress by focusing on specific neuroanatomical and behavioral traits within relevant ecological and evolutionary contexts. We suggest that a synergistic combination of the “bottom-up” approach of classical neuroethology and the “top-down” approach of comparative biology/psychology within closely related but behaviorally diverse clades can limit the effects of heterogeneity, interference, and noise. We argue that this shift away from broad-scale analyses of superficial phenotypes will provide deeper, more robust insights into brain evolution.