A former Downing Street race adviser has warned the government not to use its diversity as a “get-out-of-jail-free card” on tackling racism after a senior minister denied that the Conservatives had a problem after the Frank Hester row. Samuel Kasumu, a former No 10 adviser on civil society and communities, said he was “frustrated and disappointed” by the party’s response to the scandal, adding: “I know that we can do better.” Rishi Sunak has been under pressure to hand back at least £10m of donations from Hester after the Guardian reported that the businessman had said looking at the MP Diane Abbott makes you “want to hate all black women” and that she “should be shot”. Hester has apologised for the remarks, but denied they were motivated by race or gender. Kasumu, who quit his role in 2021, told the BBC’s Sunday with Laura Kuennsberg programme: “It’s very frustrating for me when ministers say we have the most diverse cabinet in history like that is somehow a get-out-of-jail-free card. “To paraphrase Martin Luther King, it is not the colour of your skin that matters when it comes to tackling racism and discrimination … It has to be the content of your character and your willingness to lead. That is not what we’re seeing today.” Kasumu said he had been frustrated and disappointed. “Yes, because I know that we can do better,” he said. “I know that there are people who … come into public life because they want to try to find a way to bring this country together. “But that’s not what we’re seeing … The problem is not just the prime minister, it’s also his advisers, people who were my former colleagues. I just do not know what they’re playing at. We have to do better.” He added that politicians were failing to rise to the challenge on tackling racism: “The facts do present a really damning picture, and so, clearly, there is a challenge that needs to be dealt with.” The businessman and Dragons’ Den star Theo Paphitis, who also appeared on the BBC programme, said he believed Hester’s remarks had been “criminal” and suggested senior politicians had caused division. “I very rarely agree with anything Diane Abbott says. But for Mr Hester to say what he’s said – I’ve heard it described as abhorrent and racist, but I actually think it’s criminal. It’s appalling. If he is a Tory party member, he should be thrown out by the party. “A fish rots from the top. If we’re going to have this divisive politics that we’re actually living through at the moment, unfortunately, that’s the outcome.” The transport secretary, Mark Harper, denied that the Conservative party had a problem with race, saying he was part of “the most ethnically diverse cabinet there has ever been”. “As the prime minister said this week, we are a party proudly led by the first British Asian prime minister. We are a party that welcomes people from across the UK whatever their background, whatever their race, if they share our values and approach to politics.” Harper refused to comment on reports that the Conservatives had accepted another £5m donation from the donor or would take more from him in future, adding that “if in the future there’s a donation, that will be declared in the usual way”. The senior Tory defended Sunak over the 24-hour delay in condemning Hester’s comments as racist. “Although you said it took an age in political terms, it didn’t really take very long at all,” he said. “Having an argument about how many hours it took to say something, I don’t really think is the issue here.”