Minimally restricted immigration to the U.S.

[Phalamus]

So, I came across this (rather ideologically motivated and not particularly plausible but still rather interesting) AH write-up from a libertarian website that imagines a timeline where the US Supreme Court rules against immigration restrictions in the late 19th century:

The Chinese Exclusion Act was ultimately challenged; the Supreme Court famously overturned it in the case of Chae Chan Ping v. United States in 1889. The Court found that the U.S. Constitution contained no broad grant of power that permitted Congress to restrict immigration—unless it was necessarily and properly related to another enumerated power, such as restricting the movement of foreign spies and criminals, or in accordance with the Law of Nations.

Now, IOTL the Chae Chan Ping was a unanimous one, so that obviously isn't a plausible POD.

However, by going back further in time and doing whatever tweaks necessary, would it be possible to end up with a scenario where the United States finds it constitutionally difficult to restrict immigration and attempts to pass an amendment to address that fail?

If so, what would be the consequences to US demographics, economics and culture? Would the population be much larger and more diverse? What would be the political implications?

Alternatively, if it's not possible in this way or anything resembling it, what would be some other ways of minimizing immigration restrictions?