Fruit growing is associated with long-term residence in a particular place, and can be argued to be the distinguishing characteristic between nomadic agriculture (where a group might settle for long enough to grow a crop and then move on) and settlement (where they plant fruits and wait years for them to actually produce).

As such, it's deep in pre-history, and the best anyone can currently do is to pin it to "somewhere between 6000 and 3000 BC". There are two place that could currently be pointed to as the first locations with definitive fruit cultivation (as opposed to gathering): Mesopotamia and the Nile Valley in Eygpt, both around the 3000 BC mark. There are illustrations of fruit cultivation from both.

Olives, dates, grapes, and figs seem to have been the earliest fruits cultivated - not necessarily because they were the most useful, but because they were the easiest to propagate.

It's very hard to tell what scale these were cultivated on, though. What we think of now as large scale fruit cultivation depends on 20th century technologies - machinery, pesticides, pollination tools, and so on, and a few from more recent centuries, of which an understanding of the effect of rootstocks and grafting on tree shapes is probably the most important. So before that, the largest scale were big orchards and vineyards - a few hundred trees or vines, say.

Sources:

Janick, J, 'The Origins of Fruits, Fruit Growing and Fruit Breeding', in Plant Breeding (2005), Vol. 25, pp 255-320.

Zohary D, Spiegel-Roy P., 'Beginnings of fruit growing in the old world' in Science (1975) Vol. 187, pp 319-27.