The current genetic research suggests that wild populations of horses began spreading through the Eurasian steppes from a more restricted environment sometime around 160kya (thousand years ago).

It also appears that horse domestication most probably (I stress the "probable" there) began on the western edge of the Eurasian steppes (modern day Ukraine or perhaps further east in Kazakhstan) and spread east from there with considerable continued interbreeding with wild horse population. What literature I am familiar with mostly supports the idea that domestication occurred once and spread east, but there is the possibility of several staggered domestications of horses throughout the Eurasian steppe. The genetic evidence seems to support the general west to east trajectory of domestic horses, but the specific locales in Ukraine and Kazakhstan are based on two archaeological sites with very early evidence of horse domestication. The more conservative argument would be to say that these sites might represent early domestication, but don't necessarily mean the earliest domestication is in either locale.

As for when, it looks like at the earliest horse domestication began no earlier than 3000BC, with full domestication perhaps coming a bit later (around 2000BC).

As for why the horse specifically, hopefully someone can give a more satisfactory answer, but the majority of animals we have domesticated have a combination of desirable uses for human populations (as labor or food) as well as traits which make them easily domesticated. For instance, herd or pack mentality (like cows, horses, and dogs) make breeding and controlling large numbers of animals much easier and so make the domestication process viable. Trying to domesticate loner species, like big cats, is going to be excessively difficult because of difficulties in corralling them and then breeding them. A generally friendly or non-skittish disposition towards humans is also a big plus for domestication. There has been some suggestion that horses were originally domesticate for milk production, but there really isn't any solid evidence of that so it is speculatory whether or not they were originally domesticated for riding or serving as pack animals, or for some other use (milk, leather/hides, meat, etc...).

Edit: Fixed all the cardinal directions so they actually make sense.