Mammalian herbivores can have pronounced effects on plant diversity but are currently declining in many productive ecosystems through direct extirpation, habitat loss and fragmentation, while being simultaneously introduced as livestock in other, often unproductive, ecosystems that lacked such species during recent evolutionary times. The biodiversity consequences of these changes are still poorly understood. We experimentally separated the effects of primary productivity and herbivores of different body size on plant species richness across a 10-fold productivity gradient using a 7-year field experiment at seven grassland sites in North America and Europe. We show that assemblages including large herbivores increased plant diversity at higher productivity but decreased diversity at low productivity, while small herbivores did not have consistent effects along the productivity gradient. The recognition of these large-scale, cross-site patterns in herbivore effects is important for the development of appropriate biodiversity conservation strategies.