It depends on the time period; I know that's a bit of a cop out answer but there it is :P You'd have to specify down a bit more to get some more detail. However there are some generalized trends.

Contrary to popular belief those conducting the siege are at the disadvantage, nor the advantage. Think about it for a moment; those inside the castle have warm beds, solid walls around them, months if not years of food, maybe even a small underground (metaphorical or literal) route that brings more supplies in. Those on the outside though? They're in hostile territory in general, they are without reliable sources of food (they got supply chains but again, hostile territory -- those can be fickle and only come in sporadic deliveries) with a hostile populace and, most importantly, no permanent shelter.

So what would a sieging army do? Go hunt for food and subdue the local populace while also building up their siege camp. They need beds, they need tents and buildings; some more permanent than others. Many would be devoted to sapping duties (digging trenches closer to the walls to sabotage them) or just shooting artillery at the wall persistently. However your average rank and file would be out and about subduing the local populace, dealing with bandits and raids, raiding local farms and villages for supplies, and generally just lounging around otherwise yeah; there's a reason attrition was so high for those conducting a siege, soldiers were generally forced into service in the Early-Modern period especially and with that much stagnation would just...drift away.