The simple answer is because the border was set along the Bitterroot Range of mountains (a subset of the Rocky Mountains).

The longer answer is that originally, the "Idaho Territory" encompassed present day Idaho, Montana and Wyoming when it was established in 1863. Montana became the "Montana Territory" in 1864, and most of Wyoming became the "Dakota Territory"... but the boundary between the Idaho and Dakota Territories was not the straight north-south line we have today, but instead continued the line of the Bitterroots south. This was changed in 1868 when the Wyoming Territory was established, setting the borders we know today.

So as to the rationale, there's a short answer and a murkier one. The short answer is that Congress set the boundaries of the Montana territory in the 38th Congress as follows:

“following the crest of the Rocky Mountains northward till its intersection with the Bitter Root Mountains; thence northward along the crest of said Bitter Root Mountains to its intersection with the thirty-ninth degree of longitude west of Washington; thence along said thirty-ninth degree of longitude northward to the boundary line of the British possessions.” - Source - Library of Congress

The murky part was that first this is pretty inaccessible territory at the time (the railroad didn't even cut through the Bitterroots until 1857) and there was a huge amount of "local interest" and new wealth from gold and other mining. Add to this the fact that the Civil War was occupying the governments in Washington and Richmond, and it's easy to see how the expediency of setting the border along a mountain range rather than trying to survey and cut through the Rockies to make it square ended up creating what looks strange, but was actually quite a sensible boundary.

(edit: formatting and a misspelled word)