Investors in Banksia Securities who live in the non-bank lender's heartland say the collapse has left them badly shaken.
The company was founded in Kyabram, in the Goulburn Valley, and today the farming town was haunted by news of the $660 million investment freeze.
It is believed some Goulburn Valley locals have investments of almost $1 million in Banksia.
John Zobec said he had invested $80,000 in Banksia for his four children's education.
He said Banksia was trusted in the Goulburn Valley because that is where it had been established. He first invested more than 20 years ago.
"It was virtually a family business, it was all in town, there was no reason to go elsewhere," Mr Zobec said.
"I might be wrong. I hope I'm wrong. But I think the whole area will really hurt because of this."
Receivers McGrathNicol took charge of Banksia last night and froze the $660 million in investments from about 3000 investors. All interest payments were stopped as it began an urgent review of the company's accounts.
As a non-bank lender, Banksia offered investors high interest on debentures and then lent these funds out as mortgages or commercial property loans. But as Banksia does not hold a banking licence, the funds in the debentures are not backed by a deposit guarantee.
Pastor Robert Arnold from the Kyabram Baptist Church said an investment worth tens of thousands of dollars that had been earmarked for an extension of the church had been frozen.
Speaking inside the church, less than 100 metres from the Banksia office, Mr Arnold said plans had been approved for a carport, veranda and beautification works, but that would have to be put on hold.
"The money had all been donated over many years, so if we don't get it back we'll just have to start again," he said.
"There's other people who are probably worse off than us, I think most people wanted to support a local business.
"Nearly every organisation around the town would have some money with them."
A retired Echuca dairy farmer, who did not wish to be named, said he a joint investment with his wife worth $60,000 and his son had invested $50,000.
The 76-year-old said he did not expect to see a dollar of it back.
"We'd hope to get the lot, but it's ridiculous to think that," he said.
A woman leaving the Banksia office in Kyabram today said she had about $20,000 invested and had been "sickened" to hear of the collapse.
"It's frightening really. It's not as if I had millions in there, but I had enough for a new car if I needed it."