The best way to think about it is to go back to the original purposes of each organization. The EU originated from organizations focused on economic integration and free movement of people and goods (the European Coal and Steel Community, established in 1951, and its successor, the European Economic Community, which was established in 1957) and the member countries joined based on the economic benefits of their participation. NATO is military alliance based on the principle of collective security, which was established as a Western defensive bulwark against the Soviet Union in 1949 (or Russia since 1991). So, essentially, the decision to join the EU was/is based primarily on economic interests, while the decision to join NATO was/is based on military and national security concerns. When you juxtapose those two things, you can see how membership in one could benefit a country while membership in the other wouldn't.

Norway, for example, doesn't really need to be integrated with the EU economically because its main source of income is its abundant natural resources (especially oil), but it does face the security risk of sharing a border with Russia, and saw value in the defensive protection offered by NATO membership. Sweden is the exact opposite: its military neutrality obviated the need for NATO membership (until recently), but economic integration into the EU was highly beneficial because its economy was and is reliant on exporting manufactured goods and raw materials.

Then there are other countries that benefit from membership in both or don't benefit from membership in either one. Countries like Poland and Romania, for example, benefit economically from participating in the EU because of the free movement of labor and goods and also see value in the defensive security offered by NATO due to the perceived threat from Russia, a historic rival of both countries. Switzerland, on the other hand, doesn't need economic integration with the rest of Europe because its economy is mainly based on finance and tourism, nor does it need the protection of NATO due to its military neutrality.

The states of the former Soviet Union and Eastern Bloc are in a unique position, since neither EU nor NATO membership was available to them until the 1990s, and their ability to buck Russian pressure and join either one was dictated by their level of political and economic dependence on Russia (as well as their political relationship with Russia). Countries like the Baltic states were quick to integrate with the west, since they aren't reliant on Russia economically and their politics have largely been controlled by liberal and pro-European parties. Meanwhile, other countries like Belarus, Ukraine, and Moldova, whose post-1991 politics have either been dominated by pro-Russian governments (Belarus) or split between pro- and anti-Russian factions (Ukraine and Moldova), and which remain dependent on Russia economically, either haven't wanted to or haven't been able to join either one.

Finally, there are cases like Turkey, where the benefits of NATO membership are obvious but the option of EU membership isn't available (yet), or Finland, which benefits from EU membership but wasn't part of NATO, originally due to Finlandization during the Soviet era, and since then due to both a perceived lack of need for NATO membership and public opinion being opposed to it (until recently).

There are lots of different sets of circumstances throughout Europe that dictate the political and economic calculus that go into these decisions. It's a good case study for the realist theory of international relations, i.e. that all states are competitive and self-interested and their decisions are based on those interests. Obviously the fact that we're discussing international cooperation shows that it's not a zero-sum game, but participating in that cooperation or not is still determined by self-interest.

tl;dr: EU membership is based on economic interests, while NATO membership is based on national security interests, and each country may have use for one, both, or neither of those.