“I don’t care for these new Nazis. And you may quote me on that.” – John Mulaney

The genius of this line, from John Mulaney’s best comedy special, lies in the faux bravery of its delivery; this sense that he thinks he’s out on a limb, being political, taking a stand. Condemning Nazis is easy as long as their numbers and influence remain a rounding error on the sum of political life. Perhaps that’s one reason politicians love to do it.

We’ve seen lots of this in the aftermath of last weekend’s anti-immigration protests from politicians across the spectrum. The neo-Nazi presence was alarming, especially in Melbourne, where a leader of the movement – since arrested over a violent attack on an Indigenous protest camp – was given the stage. But then come the difficulties. Once you’re done condemning Nazis, how far and how hard do you want to go?

That largely depends on how far you want the taint of neo-Nazi involvement to fall on the protesters as a whole. And it’s on this question that the Labor Party now finds itself at odds.

The prime minister chose to distinguish the far right from the mass when – in condemning the protests – he told the ABC he was sure there “would have been good people who went along”. Labor MP Ed Husic, freed from cabinet solidarity, and reflecting the views of several colleagues, took issue, responding: “I’m not in the business of doing the ‘good people on both sides’ argument … I haven’t seen a good fascist yet”.

I well understand Husic’s objection. These were ugly scenes, especially in Melbourne where there were moments of violence, some accompanied by racist insults.

It’s not quite right to say, as the opposition did, that the protests were “hijacked” by neo-Nazis. Hijacking implies they were a going concern in their own right: that far-right actors jumped on a bandwagon and commandeered it against the wishes of its rightful owner. The truth on this occasion is much the opposite: that these protests were a far-right vehicle, designed to pick up thousands of ordinary people once it got going.

We know this because an ABC investigation tracking the origins of the protest revealed the original March for Australia website specifically called for “remigration”: a far-right term for the mass deportation of non-white immigrants with the aim of retrieving an ethnically homogenous society. Organisers scrubbed this term from the website days after the launch, but those same organisers have repeatedly promoted far-right ideas on social media. One, organising the Sydney event, confessed the march’s goal was “protecting white heritage”, but said that it was “more appealing to the public” to refer instead to “Australian heritage” even though it “means the same thing”. Elsewhere, she called for a repeat of the Cronulla riots because “we need violence” if “anything significant is going to change”.

Husic is right to say that “a lot of people were warned” that these “rallies were whipped up by far-right extremists and neo-Nazis”. He therefore assumes attendees turned up knowing this, and must be ideologically aligned with them: fascists one and all. It’s here that I think Husic’s formulation falters. What it gains in moral clarity, it loses in social accuracy.

Reporting from the ground paints an enormously varied picture, much as you’d get from any protest. Some openly admitted to being neo-Nazis and racists. Others lamented the far-right presence and insisted “most of us don’t stand with them”. Lots were focused on housing affordability, a problem for which they blame high immigration. Some specifically said they don’t oppose immigration, as long as it doesn’t outpace infrastructure. One held up a sign saying “Prosecute Dan Andrews for Treason”. You don’t make sense of that by calling the whole thing a fascist rally.

Albanese clearly sees something quite different to Husic. Not so much a rally of true believers, but a conveyer belt carrying people from an anxiety about immigration levels to extremist politics. I think that’s a more accurate view, not least because the far right seems to acknowledge this themselves. To listen to that neo-Nazi speaker in Melbourne was to see a man who knew the crowd was not quite his. He asked them to overlook their differences “on historical events or versions of ideology” – a colossal euphemism for something like the Holocaust – in support of a common cause. He implored attendees “to learn to make friends” with those with whom they disagree. Amid the cheers, there were also boos. This was a man speaking not to the converted, but to the convertible.

In that context, Albanese’s remarks are worth more detailed attention: “I’d just say to people – and I have no doubt that there would have been good people who went along, heard about a rally, concerned about [social problems] … have a look at who you were with on Sunday. The motivation that they have … isn’t actually about housing or our economy or anything else. It’s about sowing division. Neo-Nazis have no role.”

Later, when challenged in caucus, he explained his approach: “We have to make sure we give people space to move away and to not push them further down that rabbit hole.” Albanese is speaking to the convertible too.

That makes no sense if you think they don’t exist. But if they do, it’s hard to see how calling them fascists will achieve much other than encouraging them to become exactly that. We’re watching a process of radicalisation on the margins of our society gaining momentum since at least the anti-lockdown protests of COVID. The far right was active then, too, and it’s now pretty clear that yelling at all who attended while a conspiratorial fringe offered them a home wasn’t a winning strategy. That Dan Andrews sign wasn’t out of nowhere.

Radicalisation relies on a cycle of alienation and rejection. That isn’t broken by condemnation. It’s broken by giving people an off-ramp; a way of stepping back from the brink without feeling humiliated.

The point isn’t whether or not the protesters really are “good people” – whatever that means. It’s to appeal to them in the hope they might live up to the billing. Perhaps then they might have no need to care for these new Nazis either.

Waleed Aly is a broadcaster, author and academic.

Get a weekly wrap of views that will challenge, champion and inform your own. Sign up for our Opinion newsletter.