I'm guessing you meant ancestor, rather than descendant. Otherwise there is something really strange going on here. =)

I cannot identify the exact unit, but I can offer an educated guess, based on some of the elements of the uniform.

He is wearing a ceremonial military style uniform of Scottish origin, with doublet of a cut and type common from the mid-19th century up until after the second world war. The cut of the dress, quality of the photo and style of hair and mustache makes me guess the photo is taken around 1900, which would fit well with the man appearing to be in his thirties.

The trews (tartan trousers) and glengarry cap marks it out as a Scottish style dress. I say Scottish style, rather than Scottish outright because they were common among Scottish expatriates in Canada too, and remain so to this day. Wearing the trews is also commonly a hallmark of lowland Scots units, rather than those of the highlands (who use kilts instead).

Secondly he is wearing swallow's nests on his shoulders (the white, stripy things were his sleeves meet the jacket torso), marking him out as a musician.

Thirdly, it could be a police or other civilian unit uniform, but the collar dogs (collar badges) appear to be standard regimental grenades, suggesting he is indeed a member of a military unit. The grenade itself would have an inscription identifying the exact unit, but the picture is too grainy to tell. The same grenade is also found as a cap badge on the glengarry, lying next to him.

Provided it is indeed the uniform of a military regimental band it would probably help to know where he lived in either Canada or Scotland, as regiments in these countries were (and are) often highly territorial, with strong local connections.

You are helped by the fact that the grenade is a fairly uncommon identifier badge for Scottish and Scottish-Canadian regiments, many of which do instead use a variation of the St.Andrew's cross. The only Scottish regiment I know of using a grenade cap badge at the turn of the century is the Royal Scots Fusiliers (but there might well be more).

The tartan pattern on the trews should also help you identify the unit, but there again the picture makes it difficult to pick out details of the pattern.

All of this boils down to - He's likely wearing an army musician's uniform. That is not to say he was a full time or professional soldier. He could have been, but he might just as likely have been a part time musician member of his local militia or reserve force band. On a hunch, and since he ended up settling in Canada around 1900, I'd suggest looking up the cap badges and uniforms of Canadian-Scottish militia units active before the Great War.