We know Jews were present in many parts of the Roman Empire basically since the destruction of Jerusalem, and it would therefor make sense to me if Europe was full of Jewish communities who spoke their own version of the surrounding non-Jews' language, but I can't find any information of other European Jewish ethnic sub-divisions other than the Sephardim and Ashkenazim. Why is this?
Smaller, different groups definitely existed in Europe throughout history. For example the Romagnotes (in Greece) or the Italkim (in Italy).
Over time these smaller groups were mostly absorbed by the larger Ashkenazim or Sephardim who moved to the areas after major expulsions, though both of these examples still exist as (small) communities.
You should note that the main difference between the different categorizations of Jews (apart from geography) is a slightly different way of phrasing certain prayers, as such, it's never been that much of a problem for one such category to be absorbed into a larger one that recently moved in.
What happened to all the other European Jewish communities?
Greek Jews or Romaniotes were murdered in the Holocaust leaving behind only a few thousand survivors and their descendants in Greece. The Krymchaks, Turkic Jews who lived in and around the Crimean region, were also murdered in the same time frame with very few survivors.
Other Jewish groups, like the Georgian and Mountain Jews of the Caucasus, lived in relative isolation from Ashkenazim and Sephardim because of geographic reasons. Hence, their language wasn't influenced by the German or Spanish languages in the way that Yiddish and Ladino were.