It’s a scene in which Khan is at his most convincing, at home in a youthful, multicultural London. Today, he’s unveiling £30m for youth services. Khan’s relaxed demeanour may seem unsurprising given opinion polls that give him leads over Tory rival Susan Hall north of 20 percentage points. Yet if you listen to Khan, or the experts taking a close interest, there are some strange political currents tugging below the surface in the capital.

Given Labour’s strong lead both in London and nationally, why has Khan been claiming to be in the hardest fight of his political life? Why have internal critics questioned his tactics? Why do his Tory opponents accuse him of underestimating a backlash against him? And despite such a commanding lead, why do some analysts believe this is the last London contest he can win?

Let’s start with a spoiler. Excluding a calamity, Khan will almost certainly win in May. Though it would rather be a difficult Win.

Specific pressures he faces this May, Khan leads a city that has faced a tumult of change since he took office in May 2016. His honeymoon lasted just a month before the Brexit vote upended the capital city – it led to an exodus of EU citizens, fewer European arrivals and so far the London economy is estimated to be £30bn worse off by some calculations. With those effects still playing out, Covid hit, providing another blow as Londoners worked increasingly from home and others took the chance to move out. The collapse in travel also blew a hole in the city’s finances and left Khan battling a hostile government for funds.

In more recent times, events in Gaza have heaped further pressures on community relations in London, including protests, which Khan has had to navigate. Meanwhile, the generational challenges of affordability continue, most strikingly apparent in the falls in the number of children in the city. Beyond that, Khan and London have been the targets of culture-war clashes.

That’s all on top of the notoriously difficult challenge of securing a third term. And all while staying broadly in line with Keir Starmer’s new-look Labour regime that has been – sometimes publicly, sometimes privately – uneasy about Khan’s tactics. He is more passionately anti-Brexit, pro-congestion charge and unabashedly metropolitan than the national Labour leadership, which is desperately trying to woo disgruntled Tory voters.

Khan is the first to argue this election is his toughest – unfavourable changes to how the election will work are top of his anxieties. “There’s a reason why the Conservative party is the most successful party in the western world,” he says. “It’s not just that they know how to win elections, they are even willing to change the rules to make it easier for them to win. If we were speaking before the last election, the Labour party was also, inverted commas, miles ahead. On election day though, in the first round, we only won by 5%. In the second round, I then won by a lot because many Lib Dems and Greens gave me their second preference, and I’m very grateful. Those Lib Dems and Greens haven’t got that insurance policy [this time].”

That’s because the voting system has been changed to first past the post. Those second preferences that previously helped Khan are no more. Meanwhile, he says new voter ID rules could affect 900,000 Londoners – disproportionately his supporters. So that 5% margin from 2021 could, he claims, be eroded.

Simon also points out how hard the contest has been to predict in the past. “That said, our recent poll from February showed that Khan had a 25 percentage point lead overall, and other kinds of similar polls have come up with similar levels of lead in the last few weeks,” she says. “So things like voter ID effects and the size of this underestimation of other candidates would have to be quite big to completely erode Khan’s lead, but it’s not impossible.”

With Labour polling so high nationally and the Tories continuing to nosedive, why isn’t Khan polling even higher than he did before the 2021 contest? One answer is incumbency. Winning a third term is hard and his team have sought advice from those involved in Michael Bloomberg’s successful third term bid in New York in 2009, Tony Blair’s historic 2005 victory – and even those involved in Ken Livingstone’s failed attempt in 2008. Stopping the contest turning into a referendum on Khan was the main lesson – achievements on free schools meals and fare freezes are being heralded.

There’s no doubt that expanding the Ultra Low Emission Zone (Ulez) has solidified the view of a decent chunk of Londoners. The Tories managed to win the Uxbridge and South Ruislip byelection last year by opposing the expansion. It’s no surprise Hall is pledging to ditch it. Perhaps more significantly, the policy earned a rebuke from Starmer, who warned there was “something very wrong” when a party policy ended up on a Tory leaflet.

Some of the jobs that cities need in order to stay lively, vibrant and diverse don’t pay enough to live there.

Relations between the London mayor’s team and the Labour leader’s office are, by all accounts, much improved now. But there continues to be a residual criticism by some in Labour of the type of campaign Khan wages – some regard it as too focused on a core vote, bolstered by those Green and Lib Dem tactical voters, at odds with the Starmer drive to reassure swing voters.

It is an accusation put even more bluntly by Hall’s allies. Hall herself was said to be too busy to speak to the Observer, but those close to her accused Khan of complacency. “He seems to be ignoring quite a sizeable chunk in London,” said one Tory. “He’s focusing on core Labour turnout areas and passion projects. If you go on the doorstep, the minute you say ‘Sadiq Khan’ the conversation changes really fast. There’s a big anti-Sadiq vote out there.”

The latest polling does show Khan polling slightly more strongly in inner London than the outer “donut” that delivered Boris Johnson his victories, but it is not particularly pronounced. Other analysts say that an abrupt change in approach from Khan would make little sense. “He’s probably not the person to be a reassurance figure in the outer boroughs,” says one pollster who has carried out private work in London. “That time has been and gone. It’s not how he won last time, either.”

Khan simply dismisses such analy­ses as plain wrong. “I’m everywhere,” he says. “I was in Kingston over the weekend, last time I checked Kingston wasn’t in central or inner London. Not only am I here for all Londoners, but I’m campaigning everywhere as well. There are no ‘no-go’ areas for me.” Merton, Harrow, Brent and Barnet have all been visited in recent days.

Then there’s crime, which tops the list of concerns for Londoners. “Across London crime is what comes up all the time,” says a Tory campaigner. “The real worry, if you take Bexley as an example, is that burglaries have increased. In inner London, businesses talk about shoplifting. Knife crime comes up pretty much everywhere.” Hall’s flagship offering is an extra £200m for policing. Khan is also pledging more police on the streets.

There are also, however, some almost dystopian trends that have continued under Khan that feel too big for any mayor to tackle with their limited powers. “The trend line that has been there for a while is still there,” says writer John Lanchester, whose eloquent works have portrayed both London’s diversity and its inequalities. “It’s that sense of London being slightly hollowed out by money and that some of the jobs that cities need in order to stay lively, vibrant and diverse don’t pay enough to live there.

“Falling school registrations isn’t a warning sign, it’s an actual emergency happening right now. It’s a disaster. It should be a genuine source of panic in the political class and for the people who run London. That means there’s a whole demographic which you’re driving away. If your city doesn’t appeal to aspirational young people, it’s got a real problem. A huge part of it is housing.”

Both Khan and Hall have included big housing pledges in their campaigns. Khan promises the “greatest council homebuilding drive in a generation” and the completion of 40,000 new homes by 2030. Hall pledges to make it “easier to build the family homes that Londoners want”. But given the alarming trends, smaller parties are trying to tap into general disenchantment.

“People can see primary schools closing and families moving out of the city, people know that they’re finding it really hard to just participate in day-to-day life,” says Zoë Garbett, the Green party’s candidate. “I hear that people aren’t sure what Labour’s offering.” Any evidence that smaller, progressive parties are picking up protest votes is a problem for the Khan campaign. Another uncertainty is the impact of Labour’s stance on Gaza. It has angered many in the Muslim community, as well as those on the left whose support Khan has previously enjoyed. Khan broke ranks to call for an immediate ceasefire months ago.

Susan Hall

Tory challenger Susan Hall. Photograph: Stefan Rousseau/PA

All these political challenges are hard for an incumbent of eight years, however propitious the circumstances. Pollster opinions of Khan’s personal approval ratings vary from not great to “very bad”. Some have even concluded this contest should be the last outing for Khan in London. “I cannot think that he’ll stand for a fourth term, because he definitely won’t win,” said one pollster, speaking anonymously. “I imagine that they will be lining up a new candidate. The performance for mayor of London will be much more disappointing than it is elsewhere in the country in terms of swing.”

Peter Kellner, the veteran pollster, said that while a Khan defeat would be “pretty catastrophic for Labour”, he struggled to see that outcome. He added that Starmer’s team would be trying to learn any lessons should Khan underperform. “Labour becomes vulnerable when people feel their lives are threatened by what Labour is proposing – and that was certainly the case in Uxbridge,” he says. “Labour doesn’t need to actually have such a plan. But if people think that the Labour government might, it’s the sort of thing that an unscrupulous Tory campaign might propose.”

There are signs that Khan is coming up with formulations that fit in with Starmer’s drive to reassure voters, without departing from the proud metropolitan that is his authentic self. He even credits a previous brand of Tories and hits the culture warriors where it hurts – patriotism.

“I don’t think David Cameron and George Osborne were anti-London,” he says. “The prime ministers that followed have been very anti-London. There’s a tactic and a strategy to this. The easiest way in their view to win votes in the north-west, north-east, east and west Midlands and so forth, is to bash London. It used to be the EU, and now it’s London. They’ve realised that the lessons from overseas are that cultural wars do work in relation to winning elections.

“Park for a second the economic benefits for our capital city that come from diversity. Do we really think it’s patriotic, slagging off the capital city? I think the Conservative party will look back in years to come with regret, embarrassment and shame that they didn’t stand up more to those of the Tory party who’ve thrown fuel on the fire of this culture war. And it’s very dangerous by the way, it polarises communities.”

Even as the mayoral campaign kicks into full swing, chroniclers of the city are fearful of the social and economic forces that will loom over any holder of the office. “A very dark future would be if the whole of London became a version of Mayfair – a kind of stage set that’s empty most of the time,” says Lanchester. “You have rich people who own the properties but mainly don’t live there, and then poor people who come in from miles away to service them – and no one in the middle.

“It’s a sort of museumification. It’s hard to imagine that happening in a vibrant, diverse, energetic, astonishing global capital. But if you’re driving away people in their 20s and 30s, how do you fix that short of a huge campaign to create places for them to live? Maybe some of these issues around polarisation and having a target painted on London’s back are an early sign [of the political ramifications].”