At various times, and in certain circumstances, the idea of the Holy Family being refugees is claimed. This status is only speculation. The story is found only in Matthew 2:11-23.
The story has 3 parts: 1. The departure in to Egypt; 2. Death of the baby boys; 3. Taking the family to Israel.

by Giotto di Bondone in Capella Dei Scrovegni, Padua – Wikipedia

Parts 1 and 2 are in the same form:
1. Joseph’s dream about Herod (verses 11 and 19)
2. Instruction to Joseph to “arise” and “take” the family to Egypt and Israel (verses 11 and 20)
3. Joseph’s obedience, who “arose” and “took the young child and his Mother” (verses 14 and 21)
4. The journey as a fulfilment of prophecy (verses 15 [Hosea 11:1] and 23 [with no Biblical reference]
Note: The story of the entry into Israel contains an extra paragraph on Joseph’s fear of Archelaus (Herod’s son) and another dream directing him to go to Galilee, to Nazareth (a town of fewer than 2,000 people).
Notice the journey to Egypt and the journey to Israel (Nazareth) are described in the same way. The story points out that both journeys are the same. So if the journey to Egypt was as refugees, the same would apply to the journey to Israel.
Historically we have to note that there was a significant and vibrant Jewish community in Egypt at this time. The Island of Elephantine was one. Another was in the large city of Alexandria, where the Hebrew Bible was translated into Greek, a translation called the Septuagint (referring to the tradition that 70 scholars worked together on the project).
The possibility is just as great that Joseph took the family into a Jewish community or area, and that Jesus was never taken out of His spiritual culture, even if the address changed.
So should we see The Holy Family as refugees or immigrants or migrants?