Chair of the Friends of Griffiths Island group Marten Syme believes the landmark location may have a new public enemy number one.
Mr Syme has a theory that ravens may be a major contributor to the killing of shearwater birds that call the island home.
The shearwaters spend each September to April on the island as part of their annual migration from the northern hemisphere.
While Port Fairy provides an idyllic location for visitors, the stay for these birds can also be one of great danger.
Some of the birds don't make it out of Port Fairy, with foxes blamed for most of fatalities that take place.
But through his observations of life on the island, Mr Syme has came up with his raven theory.
He said each April, the adult shearwaters took off and left their young on the island for seven to 10 days.
In this time, the young birds mature enough to learn how to fly and take off.
But Mr Syme said this task was fraught with danger.
"For the juvenile birds that haven't flown before, a lot needs to go right for them to successfully get into the air," Mr Syme said.
"If there is no wind during that time to lift them that can make it very difficult, or if the birds aren't overly strong and become too tired trying.
"If they can't fly they then become vulnerable on the ground.
"There were a lot of dead young birds on the island this April and last.
"Because of COVID last year and high seas this year, by the time someone has been able to get out onto the island, the carcasses haven't been in a state to assess what was the cause of death.
"They have been too decomposed to see if there were fox teeth marks or if it was just from stress or in fact whether they had been pecked by ravens.
"I suspect that could be a very real situation, there are a lot of ravens on the island and they are big birds who are good at sourcing their food.
"If it is the ravens that are taking the fatigued young shearwaters then there isn't a lot we can do about that, the ravens are native birds.
"They are growing in numbers and they are aware there is food on the island.
"Hopefully the situation arises next April where the dead birds can be examined in a more timely matter to get a better indication of what is happening to them."
While controlling the potential damage created by the ravens is not clear cut, the battle to rid the island of foxes continues.
This includes an annual baiting program to reduce the fox population.
Moyne Shire Council reported its 2020/21 fox baiting program included the laying of 94 poison-laden meat baits and seven ejector deployments.
These deployments are designed for the foxes to bite into and when they do, poison is ejected down their throats.
The other method used to control foxes is two shoots by members of the Warrnambool Field and Game Club, during which the island is closed to members of the public.
Mr Syme said these shoots were hit and miss.
"Because the island has to be closed to the public during these shoots, the dates have to be set well in advance," Mr Syme said.
"That means that when these days arrive, the conditions may not be ideal. either for shooting or for the foxes who may be taking cover if it is a poor weather day."
Mr Syme said it was hard to know the impact the baiting was having as many of the foxes that took the bait would most likely go into their dens to die.
These dens may be on the island or in the town itself.
"The baiting not only has an impact on the island but it does reduce the fox population in the town," Mr Syme said.
"It's known that the foxes cross over the causeway or come across at low tide from the island to the town and the other way around."
Mr Syme said indications were the number of shearwaters coming to Port Fairy had reduced in recent years.
He said foliage on the island was yet another challenge facing the birds.
"There is a huge amount of growth on the island, particularly of New Zealand spinach," he said.
"It is a native plant that creeps along and covers the ground, making it difficult for birds to create new burrows and uncover existing ones.
"It is overtaking bare areas so there may have to be some consideration of restricting its growth."
The Friends of Griffiths Island group works closely with Moyne Shire Council on the upkeep of the Island, with both flora and fauna, and visitors, under that model of care.
A recent upgrade made on the island was an upgrade of 500 metres of walking path.
The section of the track received a new layer of limestone.
This path and the rest of the island was off limits for a period last weekend.
Rough weather and high seas led to 400 metres of the path on the island being under water.
The water took about an hour to subside and be safe for visitors to return to the island.