It's not that simple. Each time that an agency was dismantled, it was always replaced by something else. If we were to look at the history:

Aboriginal and Torres Strait Islander Commission - established by Labor, dismantled by Liberals
Ministerial Taskforce on Indigenous Affairs - established by Liberals, dismantled by Labor
Office of Indigenous Policy Coordination - established by Liberals, dismantled by Labor
National Indigenous Council - established by Liberals, dismantled by Labor
Department of Families, Housing, Community Services and Indigenous Affairs - established by Labor, dismantled by Liberals
Indigenous Advisory Council - established by Liberals and still exists
National Indigenous Australians Agency - established by Liberals and still exists

Looking back through the history, it could be argued that Abbott was responsible for the heaviest dismantling, but it wasn't really connected to election cycles.

The current structure under the NIAA seems to be the most detailed, transparent, and accountable body that we have had so far. The Corporate Plan and Reconciliation Action Plan are worth a read. It definitely makes you wonder why we need a Voice when the plans, structure, and hierarchy is already in place.
https://www.niaa.gov.au/sites/default/files/publications/niaa-rap-2022-25.pdf
https://www.niaa.gov.au/sites/default/files/publications/niaa-corporate-plan-2022-23_0.pdf

Are you a lawyer? Have you read the actual wording of the Constitution Alteration (Aboriginal and Torres Strait Islander Voice) 2023?

The proposed amendment says:

In recognition of Aboriginal and Torres Strait Islander peoples as the First Peoples of Australia:

i. there shall be a body, to be called the Aboriginal and Torres Strait Islander Voice;

ii. the Aboriginal and Torres Strait Islander Voice may make representations to the Parliament and the Executive Government of the Commonwealth on matters relating to Aboriginal and Torres Strait Islander peoples;

iii. the Parliament shall, subject to this Constitution, have power to make laws with respect to matters relating to the Aboriginal and Torres Strait Islander Voice, including its composition, functions, powers and procedures.

That last paragraph means that the government of the day can still functionaly gut the Voice by altering its "composition, functions, powers and procedures" and then ignoring its representations anyway.

To me the only real value I see is the first paragraph, which formally acknowledges the Aboriginal and Torres Strait Islanders as First Peoples.

Edit: typo, no one will be recognised as "Dirst Peoples"

This constitutional amendment doesn’t do anything to prevent the Coalition dismantling it. There’s zero detail of its makeup, other than the existence of something called “The Voice”. If he had control in both houses, Dutton could simply redefine “The Voice” as being the Minister for Indigenous Affairs, and disband everything else.

You can’t say “it’s so important that it can’t be left up to the government of the day to legislate it”, but when people ask “where’s the detail?” the answer is “the detail isn’t in the amendment because the government of the day will legislate it”.