A team of nine groundworkers were left unpaid on the firm’s site in Leighton Buzzard.
The gang are all from Portsmouth and had come up following the promise of years of work.
Construction worker Peter Taylor told the Leighton Buzzard Observer: “Since that happened, no-one is speaking to us.
“We’ve asked over and over when we’re getting paid and we’ve heard nothing.
“We were promised a couple of years’ work here as groundworkers and came up from Portsmouth.
“Some of the guys gave up their homes to come here and they’ve been left homeless because of how Vantage have treated us.”
The men remained in the site huts while some found odd jobs at nearby sites.
The site is now in the possession of Catalyst Housing Association and a security firm hired by CHA arrived and ordered the men off site last Friday.
Taylor said: “We’ve been living here and now they’ve put up extra security refusing us access to get our gear. It’s getting worse. They are stamping all over us.”
It is understood the men have since been allowed back to collect their belongings.
A spokesman from CHA said: “Since taking possession of the site we have employed a security company to keep it safe and ensure it remains in a condition ready for the resumption of building.
“As part of this process we have discovered that some contractors were staying on the site overnight without our permission, and in response we have installed new locks to prevent this happening.
“These security measures are an important part of fulfilling our legal responsibilities as the owner of the site.”