The 2021 census shows the median age of residents in Camperdown, Cobden and Terang - three primary locations identified as council priorities for future housing and development - increased significantly.

While the median age of those in Camperdown (50) remained steady since 2016, Cobden's increased by three years (51) and Terang by two (49).

Charles Stewart real estate agent Brian Rowbottom said that trend was bad news for first home buyers.

"You've got people in that 50-plus age group and hardly a block of land available in Camperdown at the moment," he said.

"What you're finding is people are subdividing properties but because it's so hard to get a block in a decent area, the blocks are selling really high for $250,000-plus. For a young person trying to get into the market it makes it really hard. What we really need is more land so younger people can get in and not just those over 50s who have the capital behind them."

He said while many older residents had chosen to keep their land, those wanting to subdivide it were facing a myriad of challenges.

"There's a bloke I know, he's got 40 acres and he'd be three kilometres from town," he said.

"He's not allowed to subdivide his block, but he could potentially have eight, five-acre blocks which is huge. You could move eight people out of town to that area so there'd be more houses for sale in town."

Council manager of planning and building Aaron Moyne said "unlocking" land was a strategic priority but was difficult.

"Council's priority is to continue working in each of those towns and other townships across the shire to unlock and seek to release land for residential land supply," he said.

"What our previous work and review has found is there is some land supply in Camperdown but it does have constraints for development whether it be servicing and infrastructure limitations, a lack of land owner interest or other financial costs for the development land is quite high.

"The financial viability for residential development in rural townships was marginal compared to what it had been in other areas previously.

"The potential available land in Camperdown needs to be unlocked through development - some properties are locked out at the moment given they can't get the services through to their land and there would be exorbitant costs.

"... We're starting to see some changes which is good, what I'd say broadly in terms of the median age is that whilst there has been a slight increase we know many of our towns do have an ageing population but that's also being offset by new families and younger people buying into the market and building houses at the moment.

"I think that correlates with the high building and planning activity we have in the construction of new dwellings, particularly in Camperdown at the moment."

The number of new dwellings in the township grew by 97 from 2016 to 2021 while the number of families grew by 26 during the same period.

Mr Moyne said younger families were increasingly looking to move to the shire.

"If you drive through Camperdown at the moment there's houses being built all over the place and there's a range of people moving back to the area," he said.

"There are also people looking to move from Geelong and Melbourne to Terang and Timboon. Timboon has been a town that's seen quite strong interest from families and people outside of the shire who are buying land and building on that land to move either straight away or later on down the track."

While Mr Moyne would like to see more young people moving to the shire, he said council was working to ensure amenities were up-to-date to retain its older population.

"Cobden's a good example where with the structure plan we're looking to identify the important need of aged care housing to be provided in the future to ensure the ageing residents stay within the town," he said.

"We don't have that population leakage to Warrnambool or Colac or those regional centres because that will in the long term help with township sustainability and liability."

He said that was particularly important given the shire's small population increase of 0.3 per cent.

"Our increases will normally be small, it's important we pursue and focus on that trajectory going up," he said.

"Our residential population growth will never be in a large scale, that's why smaller-scale residential development providing housing is important in each of our towns rather than the growth you'll see in Warrnambool or other areas."

All but south-west ward councillor Kate Makin voted on Tuesday night to refuse a planning permit application for a two-storey dwelling at 33 Pitcher Street, Port Campbell.

The refusal was made partly on the grounds the proposal did not fit the existing low-scale residential character of the town.

Mr Moyne said the tourist destination was in a period of transition.

"It is a town in transition," he said.

"(The refusal shows) where there's a divide between what happens on the ground sometimes and what planning schemes say.

"Council at some point will need to look at Port Campbell and do some further long-term strategic work for the town to look at what the next 15-20 years will look like and that will guide planning decisions as well.

"Port Campbell will see some continued growth in the coming years - there is a planned and approved 50-lot subdivision that will go ahead there in the short term so there will be a slight population increase."