Assuming that we are speaking about high to late medieval period in Europe (portcullis gained popularity somewhere in 12th century), they in most cases, raising or lowering portcullis or drawbridge single-handedly would be by all means possible.

Contrary to common portrayal, especially in fantasy, the portcullis was most often than not made of sturdy wooden beams, sometimes reinforced with sheet metal or other metal elements. This means, that although quite heavy in its own right, especially in case of larger gates, a strong person would be able to raise it from below.[1] Even the heavier specimens could have been easily opened by a one or two guards thanks to commonly used counterweight system that usually comprised of weights that would make raising the portcullis easier and in case of rapid closing also protected it from impact damage [2]. Earlier specimens were simple stone blocks cut to size, later a cast iron or brass weights were used.[3,4]

In the period we're speaking about, portcullis or drawbridge was usually raised and lowered by a windlass attached to a crank, more often than not a simple round log equipped with a ratchet and pawl mechanism and operated by wooden spikes turned simultaneously on both sides. In theory, this could have been done by one strong person, especially in case of a smaller gate. The Monk Bar, one of the gates leading to the city of York still houses an original (although partially refurbished later) 14th century mechanism.[5]

Of course, portcullis was only one of the defensive measures used in castles. Most often than not, a gatehouse, being a single point of entry, usually housed at least two portcullises (but more was always an option, as is the case of Beaumaris, Harlech or Caernarfon castles in Wales), one gate and a drawbridge if the castle was separated from the approach by a moat or trench. Lowering the bridge, raising both portcullises and opening the gate would take a lot of time, so although a single person could do that, a narrator of the story should better have a sound explanation why no one notices the protagonist doing all of these. Also, the gatehouse was usually a guardhouse, meaning that besides people responsible for operating the machinery, there were usually a lot of guards resting, socializing or doing everyday chores, especially if the enemy was known to lurk outside. Taking upon all of them would be far more difficult than opening the gates single-handedly.

References:

[1] Guide at Warwick Castle demonstrates how the reconstructed wooden portcullis could have been raised by hand.
https://youtu.be/4K0b3Ym1DmA?t=154

[2] Portcullis inside the gatehouse leading to the Lower Castle at Marienburg Castle in Malbork, Poland. Note the weights near the lower edge and the two solid beams in the second row from the top securing the portcullis from falling when raised and preventing it from being lifted from below when closed.
https://imgur.com/a/6FjYAQC

[3] Drawbridge leading to the Upper Castle, also at Marienburg. Of note are two separate bridges for wagons and people.
https://imgur.com/a/gMRDIS1

[4] Counterweights used for raising the drawbridge from the picture above (on the left).
https://imgur.com/a/zbw7zpN

[5] Mechanism for operating the portcullis at Monk Bar city gate, York, England (14th century original with some 17th century replacements).
https://www.british-history.ac.uk/rchme/york/vol2/plate-38