Sleep is essential for life, including daily cognitive processes, yet the amount of sleep required for optimal brain health as we grow older is unclear. Poor memory and increased risk of dementia is associated with the extremes of sleep quantity and disruption of other sleep characteristics. We examined sleep and cognitive data from the UK Biobank (N = 479,420) in middle-to-late life healthy individuals (age 38-73 years) and the relationship with brain structure in a sub-group (N = 37,553). Seven hours of sleep per day was associated with the highest cognitive performance which decreased for every hour below and above this sleep duration. This quadratic relationship remained present in older individuals (>60 years, N = 212,006). Individuals who sleep between six-to-eight hours had significantly greater grey matter volume in 46 of 139 different brain regions including the orbitofrontal cortex, hippocampi, precentral gyrus, right frontal pole and cerebellar subfields. Several brain regions showed a quadratic relationship between sleep duration and volume while other regions were smaller only in individuals who slept longer. These findings highlight the important relationship between the modifiable lifestyle factor of sleep duration and cognition as well as a widespread association between sleep and structural brain health.