Context: In the diaries of one of my ancestors he meets one such person and i am really curious about how his life would have been like. I know he spoke English and some French and travelled from Italy to France on a ship, the journey had been paid for him, he didnt have any money on him and he would take up work as a servant in France. Would his parents likely have been slaves? How much and what kind of racism would he have had to face? Would he have grown up with an education? I unfortunately only find info on well-off black Americans travelling through Europe at the time or people like Alexandre Dumas who were mixed and well connected.
In order to help provide an answer to your question, could you perhaps indulge us by letting us see the diary entry of your ancestor?
The reason for why I ask is because information like the one provided by the encounter your ancestor had with a man of African descent in the mid-19th century is usually all that we have to go by in reconstructing the worlds of black Europeans during this time period. As you point out, most information will point you towards those who could and did leave their own accounts behind, men and women who were well-connected, who were wealthy or educated. These are also the individuals that are usually focused on in many narratives on pre-20th century African presence in Europe, individuals who are often noted as "exceptional" because their contemporaries thought of them as being such. For the ordinary individual of African descent, we are often left with nothing but small traces: Church records, census records (if we know the name of the individual -- unlike the United States, race was not something that was included in censuses), or other fragmentary sources.
It is for this reason that it is difficult to give you a straight answer without knowing more context. You mention that the man spoke English, which makes me believe that he was either from Great Britain or its overseas colonies (Canada, West Indies, etc.), or that he was from the United States. This opens up a realm of possibilities as to a possible background. Slavery did not always define the experience of Africans in Europe. While there were those who did arrive enslaved or whose parents were enslaved, there were those who had been living in Europe for generations.
But if there's anything more you can add to your post, I'd love to know and see if I can try to paint you a picture.