Like everything else, the answer is complicated.
- Politics. BR had requested £1.2 billion in the 1950s to modernise the network, and subsequently got it. That's roughly equal to approx £30 billion today - and look at the amount of people complaining (rightly or wrongly) about the price tag for HS2. BR needed to show results for the investment quickly; part of the reason we ended up with Beeching in the 1960s is because BR couldn't deliver on the investment. Politics is short term, and the Government wanted results from the money to showcase.
- Coal Mines. The coal industry came under attack from cheap oil imports by the late 1950s. Coal (pits) were subsidised as they were not particularly efficient initially; the lack of subsidies caused pits to close.
- Environmental. From the creation of the London Green Belt in the 1940s heralded the start of environmental awareness actually in political action. Steam engines, much as enthusiasts love them, are great bit dirty things. Imagine living next to a railway, with several trains per hour belching out smoke, and then think about the major British cities. People wanted steam engines gone.
- Image & cost. Diesel & electric trains were already present in places. People could see higher speeds and faster acceleration without any help. BR could see the results from better efficiency, easier refuelling, no runrounds (when multiple unit used), and could see they were much cheaper to operate. Diesel and electric trains basically revolutionised train travel with a massive step change. When Beeching came along, for the rail routes which had significant public outcry, there were often investigations in to potential cost savings, which almost always rotated around "have diesel railbuses been tested?". BR rarely listened and usually closed it anyway; but it was known that electric in particular was a draw of passengers to the network. Diesel & electric traction was also substantially cheaper to run then steam.
- Time. The investment in such large numbers of diesel trains came just as the requirements for the national rail operator were changing. Containerisation was just starting to kick in (which was one of the things Beeching would later get right), which meant that BR's existing freight model (of local station yards and little local freight trains trundling through the countryside) was about to become irrelevant. Growing car ownership and motorways would completely change passenger transport. To be fair, you'd have needed a crystal ball to predict what was about to happen, but the large investment in locomotives were often targeted in the wrong area with hindsight.
- And finally, Politics once more. While there was experience of building electric trains in the UK, the '55 Modernisation Plan didn't call for massive electrification - it was more targeted to specific areas of the country, and left diesel traction to fill in the rest. However, UK industry had little experience of diesel traction manufacture; the centre of knowledge was in Germany. However, politics of the time predictably required a nationalised industry to buy British. Consequently, large numbers of ill-suited locomotives were produced, sometimes with large problems that would surface later in their lives. Tallied to this, BR was almost a de facto holding company for the pre-nationalisation "Big Four" companies, which continued to operate as they had, and basically acted completely independently of each other. Consequently, each region stipulated their own requirements, and made no efforts to create common designs for better economies of scales (and potentially allowing workshops more design time per locomotive). The clearest examples is that all but WR (formerly GWR) opted for diesel-electric traction; WR opted for diesel-hydraulic.
