Campaigners said some of the public’s valid concerns about hotels being used for asylum seekers are ‘being hijacked by a violent minority’.