That's more of a linguistics question really. But the premise of your question is wrong. We learn both new words and idioms by inferring it from context, both in historical linguistics as well as ordinary language acquisition. You don't need to know the etymology (origin) of a word to know what the word means.

The name 'Alfonse' is a slang word for 'pimp' in a number of language (Polish and Danish for instance), based off a character in a Dumas novel most people using the term certainly haven't read.

When you've eaten and seen stew a number of times, or read descriptions of it, you start understanding what's meant by the word 'stew'. You don't need to know that the dish 'stew' is from the name of what it's cooked in (i.e. it used to be another word for cauldron) and that word, in turn, is derived from an analogy to a bath-house or brothel, which is what 'stew' meant long ago, and that in turn comes from the French verb estuver 'to bathe'.

This is no less true with idioms. On the contrary, virtually all languages have a lot of so-called fossil words, which are words that are no longer used except for in some idiom. People know the idiom and what it means, but can't necessarily tell you what those words mean out of context. For instance, "hoisted by one's own petard" is known to many as meaning analogous to "falling for one's own trap", but many of them couldn't tell you what a 'petard' is or even what 'hoisted' means in this context. (The literal meaning is 'blown up by one's own bomb') You probably know the expressions 'much ado about nothing' or 'without further ado', but are unlikely to find 'ado' anywhere else in modern English. Another is your own example - you don't really know what a 'kettle of fish' is, and in fact nobody's quite sure what the original sense was.

Just to further underline the point, there are even idioms that have never made any sense in the first place, because they were based off mistranslations or such circumstances. German "Unkraut vergeht nicht" ('weeds don't die', that is, bad things seem to be persistent) was misinterpreted into Swedish as "Ont krut förgås inte". Where 'unkraut' was turned into the phonetically-similar 'ont krut', changing the meaning to the nonsense "Evil gunpowder doesn't die". Yet that didn't stop it from becoming a commonly used idiom understood by most people. Another example from the same language is "to sense owls in the bog" (figurative meaning 'something strange is going on'), which comes from a mistranslated Danish idiom.

Now in the historic context, even knowing the literal meaning and the origin of an idiom won't necessarily help you, because meanings of idioms can shift over time just as with words. (for instance 'per se' is often used today in ways that don't make sense with its original Latin meaning) If a word or expression is a hapax in a particular context, all you can ever do is make an educated guess.