Olive oil was not neccessarily a luxury good in ancient Rome, and would've been an extremely important part of even the poorest roman's diet. Olive oil is a more calorie dense food than bread, and would've supplied up to ⅓ of the annual calorie consumption for a roman peasant (bread and grains would make ⅔,). One estimate puts average olive oil consumption at 20 liters per year per person in Rome-- an imperfect, but significant estimate that really shows just how important olive oil was. Olive oil, grains, and wine-- all of which could be stored effectively through the winter months--made up the basis of the roman diet, and were supplemented by other seasonally available products. One poem, attributed to Virgil, details a poor farmer's production of a meal, which involves supplementing his bread with a spread made from seasonally available herbs, some hard cheese, garlic, and-- you guessed it! Olive oil. For this presumably stereotypical poor farmer, olive oil and grain is a given, the point of the poem is understanding the ways those would've been supplemented with what was available.

Certainly, there were expensive highly refined olive oil made from the best, youngest olives, but there were also cheaper olive oils of low quality that would've cost very little. By the 3rd century AD, the grain ration given to around ¼ of the population expanded to include olive oil, which was seen a staple product. Even though the official addition of olive oil to the grain dole didn't occur until the 3rd century, an inscription from the reign of Hadrian to the praefectus annonae, the imperial Roman official in charge of the grain and food supply, was erected by the grain and olive oil merchants of Africa, so there's some reason to believe that the praefectus was actively involved in the distribution of olive oil.

In terms of what would've been inaccessible to most peasants, meat would top the list. Meat consumption was not common outside of the upper class (but within the upper class, boy was it absolutely crazy-- Pliny's Natural Histories offers an account of a guy who ate three wild boars in one sitting, presumably just to show he could?). Cheeses, vegetables, and other seasonable ingredients were more or less common depending on location, occupation, season, so on and so forth.

If you want to learn more about olive oil consumption in ancient Rome, see "Olive Production and the Roman Economy: The Case for Intensive Growth in the Roman Empire" and "The Feeding of Imperial Rome: The Mechanics of the Food Supply System" from Ancient Rome: The Archeology of the Eternal City.