Was there as mathmatical reasonuing behind it (e.g. at 3/5's, number of northern and southern delegates were equal) or was it arbitrary? Is there a non-mathematical reason?
If anyone is more intimately familiar with the intricacies of US constitutional or political history, I would be grateful if you could step in and illuminate any details I may have missed (or which perhaps have been eclipsed by more recent movements in the scholarship on the origins of the US Constitution), but I have otherwise attempted to answer to the best of my ability from a slavery studies perspective.
Essentially, it was a political calculation and compromise through and through, without much basis in any kind of social reality or mathematical calculation. Population had a role to play in determining both the size of a State's representation in the federal electoral process - how many seats it had in the House of Representatives and how many votes in the electoral college - and how much tax it paid. I should emphasise that this was very much post-colonial America; the United States were very much thoroughly engaged in a nation-building exercise independent of Britain at this point.
In the drafting of the Constitution, representatives of southern States wanted their slaves to be counted as full persons for the purposes of representation. Their rationale behind this demand was that it would, in practice, enhance the power of white southerners in the federal government. Slaves obviously could not participate in the democratic processes of government in any way, but by counting them as people, they would still be entitled to represent in the House and Electoral College. Thus in practice, there would be many more representatives of southern white voters than there would be if only free people were counted for population. Some northern leaders found this notion abhorrent; one even argued that if slaves were property in the south, then northerners should be entitled to include the animals they own as property for purposes of representation, too.
In the end, a compromise was reached whereby slaves would only count as three-fifths of a free person for the purposes of representation, but they would also only count as three-fifths of a free person for the purpose of direct taxation too. Thus, in exchange for paying a lower share of tax than they arguably ought to have, the southern elite accepted a lower share of representation. It ultimately still represented a victory for the south in many ways - counting slaves as persons for the purpose of representation at all, by any adjusted value, exaggerated the political influence of the southern planter class. Whilst they did not get quite the measure of influence they wanted, the boost to their representation was substantial, and they were able to enjoy reduced taxation. The North on the other hand had to be content with under-representation for the voting white population compared to the South and a higher tax burden. The South's reluctance to compromise was also tempered by a belief that, in the long run, the expansion of slavery would give them a majority in the House even under the compromise.
The specific choice of three-fifths stemmed from an earlier debate preceding the drafting of the Constitution.
The Articles of Confederation had empowered the government to raise revenue on the basis of land value from each member state, rather than population, which was increasingly unpopular method of raising revenue. In 1783, there had been a debate over changing this system to a population-based one; it was there that, after much discussion, the three-fifths compromise had been reached but never adopted. As far as I am aware, three-fifths seems to have been the most agreeable middle point between other proposals ranging from one-half to three-fourths and the one that was ultimately able to enjoy enough combined Northern and Southern support to stand a chance at passing, though the amendment never actually succeeded due to rules requiring unanimous approval. Thus when the Constitution was being drafted, the three-fifths compromise was harking back to an earlier agreement over the principles of taxation, which was now being tied directly to representation.
It has also been characterised as part of a wider constitutional compromise that transcended slavery and from which it can't be easily separated, which included equal representation to the Senate for the benefit of the smaller States who were fearful of being dominated in the House by larger ones. The argument has been made that sympathy from these small states, who had a vested interest in not reaching an agreement that all States should have representation equitable to their white populations, played a role in helping the South get a favourable deal.
As it happens, in the Convention, the compromise as we know it began its life as a proposal simply that direct taxation should be tied to representation; the rationale of this proposal being that no State would try to increase its representation unfairly because doing so would mean a punitive tax burden. The original intention was also that Congress would decide as necessary, in line with the census, what the formulae should be. By this point, there had already been a general agreement that slaves should count in some way, but not how much (though the free-fifths proposal had already been mooted several times over). James Wilson essentially distorted this plan and amended the three-fifths compromise to it as a means to make it palatable to the North and South alike. Southern delegates were content because they had enhanced representation with reduced taxation; Northern delegates were willing to acquiesce because at the time, there was no clear understanding of what direct taxation implied, and few could have envisaged how significant such taxes could ever become. The three-fifths compromise could also be interpreted as an incentive to free slaves - making them whole persons in representation - rather than acquire new ones.
In essence, the compromise was a necessary part of finding a working constitutional settlement. If the USA was to be a united nation, some recognition of slavery had to exist, but it had to balance the interests of both parties. Three-fifths was the awkward compromise agreed upon in 1783; I confess I am not sure what the practical impact of the compromise was on the precise balance of power, not being a scholar of American political history, though it certainly didn't work nearly as much to the South's advantage as it might have seemed it should. It also was by no means without controversy, and divided the Convention deeply, but was nonetheless just workable enough for the framers of the Constitution.
Source material
First, this wasn't really "Colonial" America; the "Three-Fifths Compromise" was conceived during the 1787 Constitutional Convention and included in the ratified document itself. It counted slaves as 3/5 of a person for the purposes of both taxation and representation in the House of Representatives. This gave slave states more relative power than they would have had, yes, though I don't believe the North/South ideological divide was quite that clear yet, IIRC.
But to your larger question of the reason that specific number was chosen, I'm not sure there is an answer. It's always been presented to me as what a compromise usually is: An agreement in which each party gives some ground to the other. I know James Wilson is credited with proposing the compromise, but I don't believe I've ever read any of his actual writings.
I'm interested to see if anyone knows anything more specific about that decision. Wilson was supposedly highly regarded by the other delegates, at least intellectually; one would think he'd have discussed it on paper either then or at some point after.