Neighbours want the plan scaled back to half the amount of patrons and outdoor alcohol consumption hidden from the street.

But the site's owner said the plan was already smaller than he envisaged and margins could become "paper thin" if patron numbers were reduced further.

Former US-chef Tim Sutherland has lodged a plan with Moyne Shire Council for a $1.5 million development of the old National Australia Bank site, on the corner of Sackville and Cox streets, which he bought as part of a retirement vision in 2019.

If approved he plans to renovate the ground level of the two-storey structure, building a restaurant, commercial kitchen, outdoor beer garden, and two retail spaces available for tenants facing Cox Street.

"The concept is that it's a restaurant with a bar, that happens to have a number of spaces that can be utilised," Mr Sutherland said.

But up to 15 Port Fairy residents opposed to the development met this week to demand patron numbers be kept at 100 overall, including only 50 outdoors.

They also want drinking kept to limited areas and screened from street view, music to be unamplified and a buffer for neighbouring properties.

Dr Ian Sutherland, a resident in the town for 30 years, owns a neighbouring property and says the proposal is not sympathetic to the values of the area.

"That building has always been a bank and it has always been supremely quiet," he said.

"It was a commercial hub, but never a nightclub drinking hub. If you wanted that you would go down the other end of Sackville Street."

Dr Sutherland took issue with the capacity for people to consume alcohol in view of the street.

"We historically have had a lot of problems with public drinking during the folk festival," he said. "Alcohol is our number one drug problem in Australia, and our number one drug problem in this community.

"I don't have any problems with it being like the Oak and Anchor, or the Merrijig, they do it very well. It's all within the building and very much contained."

Residents fear the proposal will decrease available street parking, with no on-site parking planned for staff.

The residents also hold concerns with the council advertising the planning proposal during a holiday period.

"Whilst those in the immediate vicinity have received direct notification from council, the rest of the community must have the opportunity to be consulted," they said in a statement.

The owner, Mr Sutherland, no relation of the neighbouring Sutherlands, said he had made neighbours aware of his plan before lodging it with the council.

He said plans were originally for a venue capable of hosting 600 people, but COVID-19 forced him to reduce the size to remain flexible.

"Based on the plans that we have drawn, we have struck a balance. We brought in retail space, it's a grand old building that has been underutilised for many years on that corner, it is bringing life into that part of town," Mr Sutherland said.

"We have had to do a total re-think if the COVID-style restrictions stay in place what does that mean for the development of the venue?"

He said alcohol consumption would be "predominately back of house" but he planned to serve patrons in areas visible from the street during busy periods, such as New Year's Eve.

Mr Sutherland said reducing the numbers could make the business unviable and difficult to manage for staff.

"If we reduce the numbers below 200, the numbers begin to get paper thin whether it's viable," he said.

"There will only be a couple of times a year we are approaching that number, Christmas to New Year's, during the folk festival. The day-to-day numbers being more in that 60-to-80 person range."

More information about the planning application is online.

Have you signed up to The Standard's daily newsletter and breaking news emails? You can register below and make sure you are up to date with everything that's happening in the south-west.