As tens of thousands flee their homes in border region, many say government downplayed threat of invasion