Sunni speaker’s party urges him to quit