In retrospect, it seems like MSDOS is a fairly simple OS in terms of functionality. Compared to Windows, I find it hard to understand what made it proprietary. What made it the distinct option it was during the time period for IBM?
IBM's PC program wasn't a prestigious project. IBM was heavily invested in their mainframe business, and the PC was expected to be, at best, a sideline, a small project to show that they were still the dominant force in computing.
There were plenty of people who had the skills to write a brand new proprietary OS, but they had jobs to do programming mainframes. It was easier, cheaper, and quicker to buy one from somebody else.
What was special about Microsoft's DOS? Nothing. At the time the contracts were signed, Microsoft's DOS didn't exist. At the time, Microsoft's signature product, the product that brought IBM to them in the first place, was not DOS but BASIC, a simple programming language that could be used to program the new PC. IBM wanted to buy an OS and a language from the same people, but Microsoft didn't have an OS to sell. They sent the IBM folks to their colleagues at Digital Research, who were the OS people (well...person, anyway).
For reasons that have never been entirely clear, the negotiations between IBM and DR did not work out. The legend is that the IBM folks arrived at the home of DR's owner when he was out, and met his wife instead. They asked her to sign a nondisclosure agreement - their standard procedure - and for whatever reason, she refused.
IBM went back to Microsoft, and asked again for both an OS and a language. Microsoft weren't stupid; Bill Gates agreed to provide both products. Again for uncertain reasons, they also failed to actually buy the OS outright; instead, they licenced it, which left MS free to licence it to anyone else who wanted to build an "IBM-compatible" computer. Gates then purchased an OS called QDOS (for Quick and Dirty Operating System) from a friend, made a few minor changes to turn it into MS-DOS, and passed it along to IBM. The rest is history.
Microsoft's domination of the early PC market was almost entirely due to a series of chance events, and IBM's misunderstanding of the potential of the market they were entering.
IBM did make a version of DOS themselves. PC DOS was the IBM brand for their PC operating system. Up through DOS 5 it was basically a rebranded MS DOS. In 1993 MS released a version 6.0 and IBM released a separately developed "PC DOS 6.1", which would go on to have several subsequent major revisions. The IBM DOS's impact was limited; two years later MS released Windows 95, which destroyed most interest in any DOS.
IBM at this time had OS/2 as well (which was also originally an MS partnership). And there were other competing operating systems for PCs. CP/M (which became DR DOS) and Novell Netware were also options and mostly compatible with DOS programs.