Majority in British poll say Brown unfit to lead