People flee Sri Lanka port town