Conservative grandees are urging Boris Johnson to quit after a historic double byelection defeat, as rebellious MPs began plotting new ways to oust him. The former Conservative leader Michael Howard was among those who demanded the prime minister stand down after the losses in Tiverton and Honiton and Wakefield which prompted the immediate resignation of the party’s co-chair, Oliver Dowden. In his pointed resignation letter, widely regarded as a call to others to act, Dowden told the prime minister: “We cannot carry on with business as usual. Someone must take responsibility.” Johnson’s critics are hoping to secure a majority on the executive of the influential 1922 Committee of Tory backbenchers, in the hope they can change the party’s rules to allow a fresh confidence vote without waiting for a year. As the scale of the twin defeats sunk in – both with worse-than-expected swings against the Conservatives – a string of senior Tories added their voices to those calling for Johnson to go. Asked whether the prime minister should quit, Howard said: “I do”. “The party and more importantly the country would be better off under new leadership,” he added. “Members of the cabinet should very carefully consider their positions. It may be necessary for the executive of the 1922 Committee to meet and to decide to change the rules so another leadership election could take place.” Former foreign secretary Malcolm Rifkind suggested discontented ministers could act together to persuade Johnson to step aside. “I think it is hugely in the public interest that preferably the prime minister seeks their views. If he’s not willing to see their views, because you might be rather worried of what they might say to him, then they must, at least in some number, come together and go and see him,” he said. Johnson struck a defensive tone at a press conference in the Rwandan capital of Kigali, where he is attending the Commonwealth Heads of Government summit, refusing to admit anything about his own behaviour was to blame for the byelection calamities. “I genuinely, genuinely don’t think the way forward in British politics is to focus on issues of personalities whether they are mine or others,” he said. “No doubt people will continue to beat me up and say this or that and to attack me.” He added: “In the end, voters, journalists, they have no one else to make their complaints to. I have to take that. But I also have to get on with the job of delivering for the people of this country and that’s what I was elected to do.” The prime minister is not due to return to the UK until Thursday evening, after attending a G7 summit in Germany and a Nato meeting in Spain. Keir Starmer claimed the Tory party was “absolutely imploding” after Labour won Wakefield with a 12.7% swing – enough to secure a majority government if replicated nationwide. In Tiverton and Honiton, the 24,239 Tory majority the Liberal Democrats swept aside was the largest ever overturned in a byelection. “Boris Johnson has deceived the British people and taken them for granted for far too long,” said the Lib Dem leader, Ed Davey, at a victory rally. “He has lost the confidence of his own party. He has lost the confidence of the people of Tiverton and Honiton – a seat his party has held for more than 100 years. And he has lost the confidence of the country,” he added. Andrew RT Davies, the Conservative leader in Wales, also broke ranks to criticise the prime minister for the first time, telling BBC Radio Wales: “Each and every day the prime minister gets up, like any leader, they have to look in the mirror and ask themselves can they continue to deliver for their country and for the people who have put them into office?” Andrew Bridgen, one of Johnson’s most persistent Tory critics, told the Guardian he would be putting himself up for election to the 1922 executive on a specific platform of changing the rules to allow another vote of confidence. More than 40% of Johnson’s MPs voted against him earlier this month, but another challenge is usually not allowed for a year. Bridgen said the 1922 election could be regarded as a “stalking horse” bid to change the prime minister. “The 1922 is a vote of the party. If the places are filled with people who are pro-rule change, a sensible person in No 10 might think the game is up,” he said. Former Brexit minister Steve Baker echoed the call for Johnson’s cabinet to act. “Like so many backbench MPs, I am looking to the cabinet for leadership, especially from those who aspire to be seen to provide it,” he said. Another senior party figure who has until now supported Johnson told the Guardian: “It wouldn’t do him any harm if he wanted to look in the mirror. He needs to ask himself, ‘have I got the stomach for this, and am I going to be able to do this. Is it me?’” Most of the cabinet remained silent through Friday. The chancellor, Rishi Sunak, tweeted: “We all take responsibility for the results and I’m determined to continue working to tackle the cost of living.” A Conservative party source dismissed claims that cabinet members had been slow to offer support as “barrel-scraping” – and that Johnson had spoken to his health secretary, Sajid Javid, and others. “He has spoken to Saj, [Stephen] Barclay and others. [Dominic] Raab and Priti [Patel] were on the media before nine,” the source said.