In Full Metal Jacket (1987) the Marine cadets seem to alternate using "Sir, yes, sir." and "Sir, aye aye, sir." in their responses to the drill sergeant. What's the rules for using one vs. the other. Is/was there a convention during the Vietnam era or otherwise?

The convention continues, at least among the Marines I've known, but I am Army myself.

"Yes sir" is used when answering a direct question. "Aye/aye aye" is used to acknowledge you have heard a command.

For example:

Lieutenant Butter Bar: Private, did you submit your leave form?

Private Snuffy: Yes, sir.

Vs.

Lieutenant Butter Bar: Private, go to the motor pool and check the trucks.

Private Snuffy: Aye, sir.

"Yes sir" can also be used to acknowledge you've heard orders just like aye, but aye doesn't fit in the reverse. Also, once marine recruits are out of boot camp, they cease calling non-commissioned officers "sir", and instead refer to them by their full rank, ie "Aye, Staff Sergeant" as in the military only officers and warrant officers are called sir. Marine recruits call their NCO instructors "sir" because they are not yet considered Marines and have not yet earned the right to refer to Marine NCOs as such. Once they finish boot camp, they are recognized as Marines.