As Liberal party fractures over the voice, Peter Dutton turns to Alice Springs - and a dog-whistle | Paul Karp

"I don't think that you should go off and do this every day. This is not a decision I took lightly," Julian Leeser remarked to Guardian Australia after pulling the pin over the Liberals' stance on the Indigenous voice.

Leeser's principled position came hot on the heels of the former minister for Indigenous Australians Ken Wyatt quitting the party - two decisions that front-loaded the pain for Peter Dutton for his call to push the Liberals to oppose the voice.

Leeser has opted for the high road, to put his belief in the voice into action and promising to campaign for yes even if his preferred amendments to the constitutional alteration bill are not made.

While the member for Berowra wanted to make clear he was "not playing a game here", the opposition leader travelled to Alice Springs to make unsubstantiated claims about alleged rampant child sexual abuse and to assert that locals did not support the voice, despite several Indigenous groups contradicting him.

When reporters noted that the Central Land Council and Lhere Artepe, which represents local Arrernte traditional owners, support the voice, Dutton claimed authority from conversations with "people on the ground, shoppers in shopping centres, people who are having their businesses boarded up, people who are actually living in the community and aren't afraid to speak out".

Dutton has used this logic before to defend unpopular positions on everything from marriage equality to the safeguard mechanism: to claim that those who agree with you are really the majority, while others are somehow cowed into silence or disagreement.

The tour produced terrible optics for Dutton, standing next to the founder of the Action for Alice Facebook group which has faced suspensions for "bullying and harassment", explaining away ignoring local Indigenous groups in favour of listening to mystery shoppers.

On Thursday Dutton said "you can't have a situation where the rule of law only applies to some, but not others in a community", a dog-whistle which was pretty easy to hear in a town where the perception is that it is white business owners and residents who are victims, boarding up shops and living under "self-imposed curfew".

When an ABC journalist challenged Dutton for evidence of "rampant child sexual abuse" and noted the SNAICC, which advocates for Indigenous children, had rejected his call for a royal commission, Dutton snapped back "do you live locally".

Yes, the journalist did live locally. But Dutton ploughed on with his logic that only those who agree with his assessment are local enough to be listened to.

Dutton has lost the moral and political cover of having Leeser, a prominent conservative, advocate for an Indigenous voice for a decade, in the role of shadow minister for Indigenous Australians.

But he still had the Country Liberal senator Jacinta Nampijinpa Price along for the Alice Springs tour, on hand to agree with his assessment about what needed fixing and that a constitutionally entrenched body would somehow harm these efforts.

Either Price or the Liberal senator Kerrynne Liddle, the only two Indigenous members of the Coalition party room, is expected to get some portfolio responsibility for Indigenous Australians, although this may be as shadow assistant minister rather than stepping straight into shadow cabinet.

Liddle is also an opponent of the voice in the constitution, and defended Dutton's position on alleged child sexual abuse on Friday by reversing the burden of proof.

"I say prove it's not happening," Liddle told Radio National.

Despite at least three senior Liberal moderates opposing locking in against the voice in shadow cabinet, there was no stampede out the door to join Leeser on the backbench.

But the opposition Senate leader, Simon Birmingham, said he won't be campaigning against the voice and gave a reassurance to Dutton - that bordered on a threat - that he won't be quitting the frontbench "at this stage".

Birmingham and the manager of opposition business, Paul Fletcher, can't bring themselves to say how they will vote in the referendum secret ballot, leaving the distinct impression they will be voting yes.

While nobody is in the "unique" position of Leeser, with a decade of advocacy on the voice, things are still getting very ropey for Dutton.

The Liberal MP Bridget Archer helpfully retweeted a Guardian Australia story noting the Northern Territory police minister had labelled Dutton's child sexual abuse claims a "dog act".

Meanwhile, the opposition's deputy leader, Sussan Ley, went on a pre-budget blitz of 16 marginal or target seats, signalling that the Liberals have not given up on the inner city, with visits to four seats held by teal independents and a few Labor gains in traditional Liberal heartland.

This sort of activity is perfect for an ambitious deputy because it's consistent with doing right by the Coalition team to get cost of living messages up in the media, but also boosts her profile as focused on bread-and-butter issues, which will serve her well if Dutton stumbles and his leadership becomes untenable.

The voice referendum is not the only electoral test Dutton is likely to face this year, with colleagues anticipating Scott Morrison will quit, triggering a byelection in Cook.

Perhaps Dutton should stay away from Alice Springs and get back to standing in front of new housing projects, declaring that higher interest rates affect black and white Australians equally.

Dutton claimed, after losing the Aston byelection, that his leadership had kept the Coalition together.

But it may be proven that only the state of the economy and cost-of-living pressures are capable of doing that.