Well, the goals of the American Fenians was not to set up an independent state in the New World. Their stated goal was to take control of Canada and to hold it hostage, ransoming the colony off for the promise of an independent Ireland. Possibly not the msot realistic plan in history, but you have to give them some major props for creativity and gumption!
Now, if you want the Fenians to actually set up a state, you are going to need a few things. The first thing being, of course, that the United States doesn't back away from their initial covert support (this may mean a different President than Grant). Secondly, maybe you get some more realistic leaders in the organization figure that the Maritime territories would be easier to take and hold, for ethnic and strategic reasons, and put their effort into conquering that (after all, they could blockage the St. Lawrence from this position, and significantly damage connections between Britain and Canada without having to hold ALL of Canada).
So, lets say this somehow works out. The Irish Fenians take control of Nova Scotia and the rest of the Maritimes, and get dug in. They are given covert US support in the form of weapons, and turning an eye to any Irish volunteer who wants to go join the fight. They also manage to NOT piss off the local population (difficult, I would imagine, but not impossible. Perhaps they could play up, in addition to ethnic ties, the economic strife between the Maritime colonies and Canada/Britain.)
Assuming Britain's response isn't "lets just roll over them with everything we got," there might be some negotiations. Great Britain is NOT going to trade Ireland for a far flung colony, but the rebel presence is annoying, and it does put Canada at some real risk. Since the Fenians have US support, come the negotiations, I could see a situation where the Maritimes become independent as the Republic of Nova Scotia (Nova Hibernia, maybe?), with the Irish Tricolor floating gentle in the breeze over Halifax. This would be ironic as it is not what _ANY_ of the parties want
Not sure how such a Republic would turn out, but it would be interesting.