Or were there just smaller cats, like our modern house cat that we domesticated?
The current speculation is that the domestic cat is the descendant of the Near Eastern wildcat, Felis silvestris lybica. It is similar in size to the domestic cat. However, this is not a history question, as this is not based on historical records.
http://www.nytimes.com/2007/06/29/science/29cat.html?_r=0
This is probably a better question for /r/askanthropology. Here are a couple of threads they have had in the past on the issue.
Early history of animal domestication
In the thread above they suggest that we might not have domesticated dogs but they may have domesticated themselves.
Here is a thread on felines.