Keir Starmer has become exasperated about in-fighting across government involving Sue Gray, with the prime minister under pressure from senior aides and cabinet ministers to resolve the row. Sources said the situation would come to a head after he returned from his trip to the UN in New York, where he was joined by his chief of staff. Starmer is said to be dismayed at tensions inside the No 10 machine. Senior Labour figures believe the prime minister knows he needs to “get a grip” on the situation, which he is understood to regard as a frustrating distraction from the getting on with changing the country. Gray has become a divisive figure in recent weeks. While several senior ministers have rallied around her, saying the “appalling” rows must stop or risk undermining the government, she has alienated some of her political colleagues, who have accused her of “control freakery” and creating a “bottle neck” in No 10 that has delayed policy decisions and appointments. She provoked further anger when it emerged that she was paid £170,000 – more than the prime minister – while at the same time political special advisers had their pay cut and their contracts stalled. On his trip to the US, Starmer was asked by broadcasters about unhappiness in his government towards Gray. The prime minister did not comment on the row but admitted that it was his responsibility to “deal with” unhelpful briefing. “I acknowledge that briefings to you are not helpful to the government. My focus is on what we need to do as a government to change the country for the better,” he told the BBC. “It’s my job to deal with briefings etc and I take responsibility for that.” However, government insiders believe the dysfunction in Downing Street extends beyond tensions over Gray to the role played by the outgoing cabinet secretary, Simon Case, who they say lacks the trust of senior No 10 figures. The head of the civil service has been privately accused of failing to get a handle on leaks about the chief of staff, with several cabinet ministers telling the Guardian this week they believe him to be responsible for some of the briefing against Gray himself. A Cabinet Office spokesperson said: “These accusations are categorically untrue.” Case had been expected to stand down from his role in January but is now thought likely to go before the end of the year, with Starmer understood to agree that he has become part of the problem. At the heart of the row around Gray has been alleged tensions between her and Starmer’s top political adviser, Morgan McSweeney, as well as some of their allies. Some believe the dysfunctional situation has gone too far, and the prime minister will have to take dramatic action, with one cabinet minister telling the Guardian: “One or both of them will have to go. It’s not going to be Morgan.” Others suggested that Starmer was more likely to tweak his top team – including expediting the process to replace Case and finally appointing a principal private secretary – and then give it some time to settle down. “It’s difficult to know which way Keir will go,” one insider said. “He can be ruthless as hell if he needs to, but he also won’t want to act rashly and look weak. But he knows he has to act.” One Labour figure said that Starmer may also consider “reading the riot act” to disgruntled special advisers who are believed to have briefed against Gray, who opted not to attend the Labour party conference in Liverpool this week. Over the course of the conference, numerous political advisers told the Guardian that she controls access to Starmer and papers that go in his ministerial red box. One of her allies, however, said she was simply doing her job. She was also accused of cutting special adviser numbers so that the civil service could run the show, leaving No 10 undermanned, with no legislative affairs special adviser, for example, to tackle the large number of bills the government wants to introduce. Other sources, however, have defended Gray, saying that she was very personable, a problem fixer and understood how the civil service worked, which was a huge help to ministers who were entering government for the first time. One cabinet minister said: “For some who are used to wandering in to speak to Keir when they like, it’s a really difficult adjustment because she now controls the access. “But for some of us who could never get him to respond to us, she’s amazing because she is responsive, and she mediates and gets people together when there are clashes. Disputes are resolved far quicker, from my perspective.”