Some travelers are avoiding the U.S. altogether, and of those who are coming, many are rethinking their budgets.