There is strong evidence for brain-related pathologies in COVID-19, some of which could be a consequence of viral neurotropism. The vast majority of brain imaging studies so far have focused on qualitative, gross pathology of moderate to severe cases, often carried out on hospitalised patients. It remains unknown however whether the impact of COVID-19 can be detected in milder cases, in a quantitative and automated manner, and whether this can reveal a possible mechanism for the spread of the disease. UK Biobank scanned over 40,000 participants before the start of the COVID-19 pandemic, making it possible to invite back in 2021 hundreds of previously-imaged participants for a second imaging visit. Here, we studied the effects of the disease in the brain using multimodal data from 782 participants from the UK Biobank COVID-19 re-imaging study, with 394 participants having tested positive for SARS-CoV-2 infection between their two scans. We used structural and functional brain scans from before and after infection, to compare longitudinal brain changes between these 394 COVID-19 patients and 388 controls who were matched for age, sex, ethnicity and interval between scans. We identified significant effects of COVID-19 in the brain with a loss of grey matter in the left parahippocampal gyrus, the left lateral orbitofrontal cortex and the left insula. When looking over the entire cortical surface, these results extended to the anterior cingulate cortex, supramarginal gyrus and temporal pole. We further compared COVID-19 patients who had been hospitalised (n=15) with those who had not (n=379), and while results were not significant, we found comparatively similar findings to the COVID-19 vs control group comparison, with, in addition, a greater loss of grey matter in the cingulate cortex, central nucleus of the amygdala and hippocampal cornu ammonis (all |Z|>3). Our findings thus consistently relate to loss of grey matter in limbic cortical areas directly linked to the primary olfactory and gustatory system. Unlike in post hoc disease studies, the availability of pre-infection imaging data helps avoid the danger of pre-existing risk factors or clinical conditions being mis-interpreted as disease effects. Since a possible entry point of the virus to the central nervous system might be via the olfactory mucosa and the olfactory bulb, these brain imaging results might be the in vivo hallmark of the spread of the disease (or the virus itself) via olfactory and gustatory pathways.