Agriculture.
Once you get villages, you get towns. Once you get towns, you get cities. Once you get cities, you're going to get technological development, whcih leads to bigger cities. It's that constant process of accretion. Admittedly, I've simplified it way too much, but the basic principle is there.
So, I don't know, wipe out some domesticable crops or something? Perhaps something like Australia, where the climate is so random and so inimical to life that agriculture never got a chance to develop (and not having the crops needed helped too). So perhaps the forebears of maize and wheat never develop.
You could still have villages, even small towns, based on fishing and other crops, but without the staple foodstuffs there wouldn't be any way to feed a large city.