I'm wondering, is that true?

Not exactly.

Very early in the history of the country, there was no standing army to speak of (technically it existed, but it was very, very small) and the bulk of the defense was entrusted to the militia. By law, the militia included (white) men of military age, and the Militia Acts of 1792 defined the structure of the militia, and the duties of its members. To provide assurance that the militia would be able to provide for the defense of the country in an age when the standing army was almost non-existent, there is a relevant passage:

[Each militia member] provide himself with a good musket or firelock, […] or with a good rifle, [and appropriate kit]; and shall appear so armed, accoutered and provided, when called out to exercise or into service [….] [and that] arms, ammunition and accouterments, required as aforesaid, shall hold the same exempted from all suits, distresses, executions or sales, for debt or for the payment of taxes.

So there was imposed a rather explicit duty to own these items, and you could be fined for failing to comply (Nothing I have read, however, seems to indicate strong level of enforcement in too many regions though, but if someone else weighs in on that, awesome!). The Militia Act was overturned in 1903 with the Dick Act, which did away with the requirement to be armed, and created the basis for the National Guard system that we now have in place of a militia.