Keir Starmer did not accept £5m from a foreign-based crypto billionaire like Nigel Farage, or hold parties in Downing Street in breach of lockdown rules like Boris Johnson, or crash the economy like Liz Truss. The outgoing prime minister is neither corrupt, compromised nor crazy. The deep public hostility to him is hard to explain – and in many ways unfair – but leadership is like an X-ray of the soul and Starmer’s political fractures were exposed from the start. 

Some point to the winter fuel allowance cut as the “original sin”; others highlight the botched welfare reforms or the decision to accept free glasses. MPs criticise Starmer’s failure to “tell a story”, highlight his lack of charisma, or say he has been a terrible judge of character in making appointments. His supporters suggest that he fell foul of the anti-incumbency mood that is sweeping the world. 

I think the real problem was more profound. Starmer could never properly explain why he wanted to be prime minister. In fact it was always unclear whether he really knew himself. There was no “irreducible core” to his premiership, no governing project around which everyone could unite. That meant Whitehall did not know what it was supposed to deliver and the voters could not see the point of his government. It left the prime minister swerving all over the place, nicknamed the “ditherer-in-chief” by exasperated officials.

Despite a belated radicalism that saw the government announcing a ban on social media for under-16s and promising to get closer to Europe, Starmer too often seemed to have his foot on the brake when the electorate wanted the country to accelerate towards change. His government was defined by incrementalism because it did not dare to press ahead with transformatory change in so many areas, from welfare reform to social care, education or digital ID. There was plenty of analysis about the past but no prospectus for the future. The prime minister failed to provide the optimism people crave.

One explanation for these flaws is political. Starmer is a liberal north London lawyer, on the soft left of the Labour Party, but he suppressed those instincts because he thought he had to woo rightwing Reform voters. He hated politics so he subcontracted it all to strategists like Morgan McSweeney who gave him a harder edge on immigration, the environment and Europe. As a result, Starmer found himself pretending to be something he was not. He read out a speech suggesting that Britain was turning into an “island of strangers”. He never believed it but he thought it was what he had to say to win. It meant he came across as inauthentic – he was inauthentic – and there is nothing the voters hate more.

At the same time, the prime minister was nervous of championing the two things that did drive him on in politics – the desire to ensure that every child fulfils their potential and a support for human rights and international law. His advisers told him they were not popular or relevant. When I asked him directly last year why he wanted to be prime minister, he said it was to turn around the lives of young people who had been “collateral damage” for the failure of past governments. But within days, Downing Street was declaring that the cost of living was his only priority. Starmer struggled to make decisions because he did not trust his gut and his advisers were only interested in political strategy rather than policy ideas. 

McSweeney once patronisingly compared the prime minister – his boss – to the driver of the driverless Docklands Light Railway. Starmer might have done better if he had had the courage to follow his own convictions. He appointed Peter Mandelson as ambassador to Washington, against his better judgement, because he was told it was the politically smart thing to do. It turned out to be a disaster. Starmer then blamed everyone around him when things went wrong, brutally throwing good people under the bus, instead of taking responsibility.  

There may also be a more psychological explanation for the prime minister’s failure to cut through. Politics is an emotional as much as an intellectual endeavour and Starmer was unable to form that connection with the voters. His own personal story is one of triumphing against the odds. He was the first in his family to go to university and as a child he spent many days sitting beside his mother’s hospital bed. 

It is a powerful narrative but those who know him best believe he struggled to tap into the emotional potential of that personal experience because he felt guilty that he had “succeeded” in the eyes of society while his siblings had a difficult time. His father always told him he was no better than them and he learned early on to suppress his emotions. His tearful promise at the end of his resignation speech to be the best husband and father was a rare glimpse of the sort of humanity the electorate now demands from its leaders. But it was too late.

As he stood behind the podium outside the door of No 10, Starmer listed his achievements. He had turned around a Labour Party that was “politically, financially and morally bankrupt” then won a landslide general election victory. He had ushered in a stronger economy and improvements on NHS waiting lists, infrastructure, workers rights and immigration. He accepted “with good grace” that his time was up but he insisted that his successor “will inherit a Britain that is far stronger and fairer than the one I inherited two years ago.” 

Andy Burnham will walk up Downing Street next month promising a new era of “hope”. The voters in Makerfield warmed to his human touch and his positivity but he will be facing all the same dilemmas and trade offs as Starmer. One former cabinet minister says: “We can’t have a repeat of a Labour prime minister walking in through the door of No 10 without an agenda or a plan, that’s what happened last time with Keir and it would be fatal for the Labour Party and frankly a dreadful outcome for the country.” 

The last few months have shown that politics is more febrile than ever. Events seem to be playing out at double speed at Westminster and beyond. Burnham will not have long to prove himself. The moment he moves into No 10 he will be transformed from insurgent to incumbent. Hope will rapidly turn to disappointment if he does not deliver genuine and tangible change.