Why were streets such as The Shambles in York ( https://en.wikipedia.org/wiki/The_Shambles ) built with the streets so narrow and the houses so close together when there was presumably so much space available?
I can not specifically comment on the structure of medieval cities, and the shambles, but I think your question about the structure of narrow city streets extends beyond the medieval era. As a result, I can at least offer some evidence that explains the reason for 19th century cities building narrow streets and closely knit houses.
First it is important to consider that the primary mode of transportation for goods and people for a large part of history occurred through the labor of animals like ox and horses. These animals transported goods to the markets, supplies necessary to construct the buildings, and people. So long as the horses were the primary source of transportation cities were forced to accommodate them. Since horses and ox were the only form of transportation there was not much justification for larger roads.
In the article by Sherry Olson (in Animal Metropolis) labelled, The Urban Horse and the Shaping of Montreal, 1840–1914 she demonstrates the importance of horses in Montreal and how the large population of horses shaped the streets of the city to become narrow and compressed. In the 1840s the horses began to use narrow streets (similar as above) to travel in between buildings and different parts of Montreal to reach the interior blocks of neighborhoods. By the 1870s the city continued to evolve to meet the growing population and the subsequent increase in the number of horses in the city. New narrow lanes were built and property owners created tunnels through buildings to provide better access to the back for the horses. Larger roads were not yet necessary as horses were the dominate form of transporting goods. Even with the introduction of the rail road horses were the only from of transporting goods from the railroad station to all of the stores and shops hidden in the narrow alleys of Montreal. Similarly, cities across North America that were founded on a reliance on the labor of horses had their streets shaped to accommodate this mode of transportation.
Although I was not able to provide a specific answer as to the Shambles in York, you can certainly attribute some of the narrow streets as a result of people simply accommodating their primary source of transportation.
Are these street narrow? In the linked picture one can see that the street allows seven or eight people to easily stand abreast. It may be not much for modern standards (modern roads are build with safety of motorized transport in mind), but was more than enough for the late Middle Ages (if memory serves, the oldest buildings in the Shambles and Small Shambles come from mid-14th century), when the traffic in predominantly residential areas was limited to pedestrians with an occasional horse, mule or hand-cart. Roads leading from gates and towards the markets or other important location in the cities were usually wider, to facilitate use of larger vehicles, such as wagons transporting goods for sale and to accommodate for denser pedestrian traffic.
Now, the space in the city was not that abundant, quite the contrary. Given that the extent of the construction area was limited by the city walls (that could have been expanded, but this was costly and time-consuming process), the housing pattern became denser and the houses themselves became taller to accommodate the needs of the rising population of the cities. Archaeological evidence found in the Bavarian cities suggests, that in late 13th/early 14th century the construction lots were quite loosely located, sometimes several meters apart and the buildings themselves usually had only two floors with an optional attic. In late 15th century, houses were more often than not either adjacent to each other or built so close that there was as little as 3-4 feet of space between them Additionally, buildings themselves tended to be taller, with three or four floors becoming the norm. The streets may also seem optically more narrow than they really are due to the fact that in the timber-framed buildings, each subsequent floor is often somewhat wider than the one below thus additionally limiting the amount of natural illumination, especially in case of tall buildings This problem is usually not present in case of houses made of bricks or stone.