No signs nor visible tracks show the way to and over the dunes to where tributes to some of the township's earliest pioneers are now covered with thick overgrown vegetation.

About half a dozen graves are visible at the site but historians estimate between 50 and 200 burials took place there between 1850 and 1860.

A Department of Environment, Land, Water and Planning spokeswoman said access to the cemetery, on crown land, was better kept limited to ensure its "heritage values" were protected.

An interpretation plaque in the dunes provides an overview of the site and highlights its history of neglect.

The department has left community groups to maintain the remote and isolated site, which is bordered with coastline and to the north by private land.

But local historian Marten Syme said its location meant keeping up maintenance had proven unfeasible for volunteers and governments.

"The difficulty for the government is they are responsible, yet they have no capacity to manage it," Mr Syme said.

"It's not feasible to expect the community to maintain it or even the community to pay money to the government for it to be maintained.

"But these are pioneers of Port Fairy and they ought to be remembered."

The DELWP spokeswoman said the department would work with other stakeholders to consider future management options.

Mr Syme last worked on the site with Lions Club volunteers in 2017 when they used machinery to replace the marble capstone of pioneer Michael Connolly's grave after it was believed vandals removed it decades earlier.

"Michael Connolly is one of the founders of Port Fairy and his grave was totally disregarded," Mr Syme said.

He said a number of early settlers were likely buried at the site, including captain Lewis Grant who was known for a business that ferried goods from ships to the shore.

Another quirk of the site is despite appearances it's known as the 'new cemetery' because it is predated by the township's 'old cemetery' still used today.

Mr Syme said the government of the day was likely forced to build a public cemetery on land outside of the then privately owned township, known as 'Belfast' and owned by solicitor James Atkinson.

"You have this isolated cemetery surrounded by land the government had sold because they had basically painted themselves into a corner," Mr Syme said.

"This is another aspect of this very unusual situation in Port Fairy, where the special survey allowing Atkinson to own acres and lease it to people created a number of problems for the government."

He said the cemetery's isolation had always been the cause of issues, with government reports from as early as 1866 stating the site was in "unacceptable condition".

"By 1881 there were few burials, no maintenance, sand incursion, and the government allowed the interment of coffins to the 'old cemetery' area," Mr Syme said.

He said it could be unpleasant for descendants of the deceased buried at the site to see it overgrown, but added there was unlikely "a sensible solution to the problem".

"It ought to just be known about but allowed to slowly remain. It's going to get progressively worse, but the only way it will be preserved practically is by individuals going out and doing something if they wish to."

Have you signed up to The Standard's daily newsletter and breaking news emails? You can register below and make sure you are up to date with everything that's happening in the south-west.