Certainly. This is a big question, so I'll just look at East Asia, mostly China, but also your sub questions have easier answers so I'll address those first then give some examples of regulation.

Which language system was the earliest?

I'm not sure one person here can answer that unless we all answer it then compare dates. For all we know there were Sumerian education ministries that regulated how the language was used, but that's not something I'd know about, assuming anyone knows about it, assuming there are records of that.

Also, what do you mean by "pre-modern"? There are a lot of ways to define that.

Where was the line drawn between colloquial speech and regulated language?

There always is. In linguistics this is part of what is covered when we talk about "prestige". Think of the way newscasters in your country speak. Think of the way stereotypical poor/rural/uneducated people speak. I say stereotypical because we all know that people with X accent aren't actually stupid for having that accent, but the fact that people might characterise them that was reflects the prestige of their way of speaking. Prestige is not objective, nor is it stable, but it's always something communities feel toward different varieties. We can also talk about register here. Often the "standard" "high prestige" way of speaking matches up with a more formal register. Basically everyone is multi-dialectal. You change the way you speak based on who you're speaking to. For some people the differences are big. For others not so much. Point being, there's always a line drawn between colloquial and regulated, even if "regulated" just means regulated by your social circles.

Under whose authority was it to make these decisions?

So now we can get into the main part of the question. Whose authority is usually if not always the government, be it the emperor/king, or organisations set up by them. In modern China there's SAPPRFT, formerly SARFT, the state broadcasting administration, combined with the Ministry of Education which sets the curriculum. Prior to broadcasting, in more modern times, equivalents to the MOE would handle this stuff, since the way you get people to use the standard is to teach it to them.

Anyway,

Was there ever a pre-modern system of regulating languages?

Yep. If we go back to the start of the Qing, for example, the original policy was that all government officials had to learn and use Manchu. This was later relaxed, but is a pretty striking case of the distinction between formal and informal you mention, as it required use of an entirely different language that people were forced to learn.

Before (and after) that, there was the difference of prestige between written vernacular Chinese and Classical Chinese. The vernacular was basically not to be used in any formal documents, even though that's what people spoke, and instead everything had to be done in an archaic form as represented in the Classics.

But lets go back as far as we can in Chinese history. The earliest mention of standardisation comes from the Rituals of Zhou (周禮), which describes one role of the External Affairs minister as spreading the character forms throughout the land (掌達書名於四方). In other words, it was his duty to propagate standards. There's a lot of commentary during later periods that glorifies the state of things during the Zhou as a sort of linguistic Golden Age. These should be taken with a grain of salt. Still, we at least have record from the period itself of the handling of standards in writing. Additionally, during the Zhou there were state-established schools which taught (among other things) writing.

Later, during the Qin (秦朝), there was an effort by the Court to propagate the standards developed in part by Li Si (李斯). It's presented as this great success even today, but as I mentioned to /u/iwaka, it's a myth. They didn't do what they're given credit for, namely standardising the characters across the land. It was a long process that happened over a period of time before during and after the emperor's rule. Standardisation was not successful under his rule, and even well into the Han we see it hadn't yet really happened. See Imre Galambos' The Myth of the Qin Unification of Writing in Han Sources and Orthography of early Chinese writing for a lot of discussion on that topic.

It was during the Han that things really started to congeal. Use of non-standard characters was punished. This is also the time that we see some real consistency in character forms, something not present during the Qin.

Later, during the Sui (隋) and then Tang (唐), the Civil Service Exam was developed and put into use. It was primarily based on knowlege of the Classics such as the writings of Confucius, and this became the primary place where education and thus knowledge of the standards was taught. The Tang period also saw a change in which script was to be the standard, moving away from what's known as Clerical Script.

Answering about spoken standards is a lot more difficult. I've focused mostly on written standards because that's what we have records of going that far back. For spoken standards, they still existed, but there's not as much to say on that. The most significant example, however, would be the system of fanqie, where character pronunciations, particularly in Buddhist texts, were represented through the use of two other characters with well-known reading. A great many characters would be easy to shift in pronunciation (as "Peru" 秘魯 has from bilu to milu because 秘 is also pronounced mi, just not in 秘魯) because of confusion of the character forms, of things indicating pronunciation of an earlier period but not the present (a lot of characters are very conservative in what pronunciation they reflect). The fanqie system solves this to a large extent.

Rime tables, as were used in dictionaries and most notably the similar system in texts like the Qieyun / Guangyun, were a more developed form of the fanqie system, standardised and organised to also more explicitly reflect tone categories. What we don't have a lot of are records of prescriptive rules for syntax. However, in written texts, syntax was constantly evolving along with semantics and pronunciation, so this is not too surprising.

My Chrome keeps crashing every time I change input methods, so I'm going to stop here, but I think that answers your question. Let me know if you have follow ups and hopefully things will be less crashy then.

/u/keyilan, would you say that the writing system reforms under Qin Shihuang fall under language regulation? It seems so to me. I think China would definitely be on the list of the earliest language regulating societies, with Qie-yun being more of a standardised pronunciation guide, amirite?

Anyhow, this is a shout out to my man /u/keyilan, who can provide an answer with regard to language regulation in China. Do tag me in your post, I'd like to read it, too!