Keir Starmer has accused Boris Johnson of “parroting the conspiracy theories of violent fascists” for political gain after the prime minister doubled down on false claims about the Labour leader’s blame for the failure to prosecute Jimmy Savile. Johnson has been condemned by victims of the disgraced broadcaster, as well by as some MPs and the Commons Speaker, Lindsay Hoyle, for making the claim in parliament on Monday. Michael Gove, the levelling-up and communities secretary, dismissed the concerns earlier on Wednesday, saying there was no need for the prime minister to apologise or withdraw the claim, which is strongly associated with far-right conspiracies. Beginning this week’s prime minister’s questions, Starmer appealed directly to Conservative MPs over Johnson’s tactics. “I just want to say to the members opposite: theirs is the party of Winston Churchill. Our parties stood together as we defeated fascism in Europe,” Starmer said. “Now their leader stands in the House of Commons parroting the conspiracy theories of violent fascists to try and score cheap political points. He knows exactly what he is doing. It’s time to restore some dignity.” Johnson, however, declined to apologise, saying: “On his first point, I don’t want to make heavy weather of this, but I am informed that in 2013 the right honourable gentleman apologised and took full responsibility for what had happened on his watch. I think that was the right thing to do.” It is correct that as director of public prosecutions Starmer apologised on behalf of the service for the fact that Savile, a prolific sexual offender for decades, was not prosecuted when he was alive. However, the idea, as raised by Johnson on Monday, that this was due to Starmer’s actions has been debunked, and has largely been propagated on far-right Facebook pages. In the exchange on Monday, Johnson claimed Starmer “used his time prosecuting journalists and failing to prosecute Jimmy Savile”. On Tuesday Hoyle rebuked Johnson, saying he was “far from satisfied that the comments in question were appropriate on this occasion”. Earlier that day, Julian Smith, the Conservative former Northern Ireland secretary and chief whip, said Johnson’s comments had been unacceptable. He said: “The smear made against Keir Starmer relating to Jimmy Savile yesterday is wrong and cannot be defended. It should be withdrawn.” Despite the criticism, Gove said Johnson had nothing to apologise for. He told Sky News: “I think this is a uniquely sensitive issue and it does need to be handled with care, and I listen with enormous respect to those who act for victims of the actions of a terrible, terrible criminal. “But – and it’s not a subject that I want to dwell on because it is uniquely sensitive – it is the case that the CPS apologised for the handling of this case and what happened in 2009, and I think we should acknowledge that an apology was given at the time and respect that.” He added: “Keir Starmer acknowledged that mistakes had been made by the organisation of which he was head, to his credit. He was very clear about those mistakes.” Richard Scorer, the head of abuse and public inquiries at the law firm Slater and Gordon, said: “Victims of Savile I have spoken with today have told me that they want Johnson to withdraw these comments and apologise, and I call upon him to do that right away.”