It is understood that some people had gathered there to stop works on a building that they believe is earmarked for asylum seekers.