5,500 people have turned down social housing since 2016.