David Lammy, the shadow justice secretary, has defended Angela Rayner for calling the Tory government “scum”, saying the Conservatives should get their house in order before criticising Labour for “fruity” language. Speaking to the Guardian in Brighton where he will address Labour’s annual conference on Tuesday, Lammy attacked the government’s apparent outrage over Rayner’s comments at a late-night conference rally. “I too have been known for fruity remarks over the years. I’m not that keen to take lectures from senior Tories who have a leader who described people like me as being piccaninnies with watermelon smiles, who describes gay men as tank topped bumboys or Muslim women as bank robbers,” he said. Rayner declined to apologise for telling activists on Saturday evening that “we cannot get any worse than a bunch of scum, homophobic, racist, misogynistic, absolute vile … banana republic, vile, nasty, Etonian … piece of scum”. Transport secretary Grant Shapps described the comments as appalling. Labour leader Keir Starmer appeared uncomfortable when asked about the comments, saying he would not use that language himself. But Lammy said: “I’ve heard a lot of noise from senior Conservatives. I think they should get their own house in order – and with so much going on, it’s slightly absurd that people are rushing to focus on a form of words, when there is some serious substance to talk about.” Lammy is regarded as a strong performer in the House of Commons. He said he was keen to take on his new opposite number Dominic Raab, who was moved to become justice secretary in Boris Johnson’s reshuffle, and is expected to take aim at the Human Rights Act. “I relish the opportunity to go toe-to-toe with Dominic Raab in the coming weeks and months,” he said. Raab is widely viewed at Westminster as more hardline on issues including human rights than his predecessor, Robert Buckland. “I hope they’re not seriously trying to abolish human rights, which I think is a huge cornerstone of the world order that was set up after the second world war,” Lammy said. “There is no clamour from the public to get rid of human rights: it’s extraordinary.” In his speech on Tuesday, Lammy is set to announce that law firms will be prevented from taking on lucrative government contracts unless they are also doing some pro bono work (representing clients for free). “The government is handing out huge procurement contracts – has been during the pandemic and certainly was during Brexit – and what we’re saying is that there will be a requirement to do pro bono work to get those contracts,” he said. He warned about the challenge of what he called “advice deserts” in which low-income people are left without any affordable local lawyers to consult. Lammy also criticised the government for failing to support the judiciary in recent months, with home secretary Priti Patel bemoaning the activities of “activist lawyers”. “It’s been unsavoury and unbelievable to listen to people like PP disparaging lawyers: ‘do-gooders,’ ‘activist lawyers’. This has been a very tough period,” he said. After Labour delegates spent the weekend wrangling over complex rule changes aimed at making it harder for a far-left MP to win the leadership again, Lammy said it was time for the party to look outwards. “We’ve won the argument, it’s over, it’s one day, we’ll get on with conference,” he said. “I’m afraid we’ve been too good at navel-gazing, and having what should be private rows in public. It’s time to bring the washing in from the line.” Lammy said Labour would act to repair the legal aid system, which has been hit by significant government cuts in recent years. “When we talk about the welfare state, we talk about that safety net, that we’re very proud of in this party having set up in that Attlee government, after 1945. A significant part of that was actually legal aid as well,” he said. “That concept of equity, and justice: that sense that whatever your background you can get justice. It’s fallen away and we want to restore it.”