Canada summit protest dwindles to a hardy few