The historical Jesus was not a child refugee. He was from Nazareth, period. The stories of the family traveling to Bethlehem are not in the oldest gospel (Mark) and almost certainly got added in to explain why the messiah was not from Bethlehem when prophecy said he would be, the same home town as King David.