Buttons (and, for that matter, lacing) are only necessary on most garments if your clothing fits so tightly that it must have a slit cut into it in order for you to put it on or take it off. This is a development that did not occur in Europe until later in the Middle Ages - on a general scale, I'd put it at the fourteenth century, but the French bliaut was a tight-fitting, probably or possibly laced gown/tunic worn from the late tenth century. I discuss the shift to more fitted clothing in this previous answer.

The toga is actually a Roman garment, worn over the tunic, which was the basic unit of clothing in Ancient Rome. Tunics were made with minimal cutting and fitting in order to preserve the integrity of the cloth, as I understand it, which meant no tailoring that would require buttoning. The only place I have come across buttons in this context is potentially along the top of the arm and shoulder in some women's tunics which have no continuity of fabric at the top and are fastened at intervals with something - but these are only speculated to be buttons, and may instead be brooches or sewn beads of some sort.

The answer to this is: technically (depending on how you define the Roman period).

Buttons don't appear in Europe until the 7th century AD, and the late 7th Century at that, first appearing on Kaftans of the Late Avars, Caucasian Alans, etc. after being transmitted out of Central Asia and ultimately from Mongolia/China. This is what we call the "Early Byzantine" period usually although it was literally just the Roman Empire.

At this point the Toga had been out of use for about 2 centuries. It had been replaced by the Pallium (a relatively similar garment) in the Theodosian code in the late 4th century AD. The Romans started using Kaftans of the Gothic, Alan, and Hunnic styles in the early 5th Century AD, probably. We're not sure but Gothic Kaftans appear in Roman art in the late 5th century at the earliest. Hunnic Kaftans we think were of the type found at Loulan in Xinjian, while Alan/Iranian nomad style Kaftans we think were of the type found at Antinoe in Egypt, of which two surviving examples exist. The Gothic style (on the Bargello Ivory and Halberdstadt Cathedral Diptychs) was called a Zostarion, while the Iranian nomad style was named the Kandys. The Hunnic style is unnamed but was probably also called Zostarion. There was also the Gounoberkonion which was a really baggy Kaftan worn over military equipment to shield it from rain.

Buttons are first used on what are called Persomanikia or tightly cuffed sleeves, basically, on Tunics. You can see them in this image of Justinian II's entry into Thessalonike where they have the tight cuffs but baggy, loose sleeves (contrary to the typically tight sleeves of late Roman Tunics called the Tunica Manicata). These first appear during the reign of Anastasius I who is the first emperor to use a Tunic with Persomanikia on his coronation. Buttons are introduced to these in the late 7th and 8th centuries. The Romans also started using various kaftans and some long tunics with buttons in this period, often called the Kavadion, Skaramangion, etc. Many of these garments (particularly the Skaramangion) had formal court dress functions.

Before the late Roman/"early Byzantine" period? We know some wooden toggles existed. Elaborate bronze toggles have been found for 1st-2nd century Paenulae but these weren't the same as buttons. Many also used hooks and loops as well (Also bronze).

Sources:

Dawson, Timothy. By the Emperor's Hand: Military Dress and Court Regalia in the Later Romano-Byzantine Empire.

D'Amato, Raffaele. Roman Military Clothing (3).

Sumner, Graham. Roman Military Dress.