The past is littered with musical instruments that have gone out of fashion, in different places, and for different reasons...and come back into popularity. Ukuleles were big in the 1920's, were really unfashionable by 1950, and in the past several years they've become popular again. Their early music cousin, the Renaissance Guitar, was popular only in the mid 1500's, in France, then sank away. So, the answer to your question could be, very simply, tastes change.

However, lutes are part of a large cohort of instruments, played for hundreds of years, which became more rare at about the same time. The recorders , viola da gamba and the whole viol family also became obscure. Most of these were a bit quiet, like the lute, or had a limited range, and the later 18th century saw bigger orchestras, bigger halls. Louder instruments that could play for big audiences could of course also play for smaller ones, but lutes and recorders that worked best in intimate gatherings couldn't hold their own in big orchestras, so musicians who wanted to work took up something else. Even instruments that survived into the 19th c., like the violin, were altered to become louder. Lutes did not disappear immediately- they became bigger, were used for other jobs.. theorbos were playing continuo in Italy well into the 18th c. ( they do have a very lovely bass sound which is hard to replace with anything else) . But eventually it was hard to justify the time to learn lute, if you could learn piano or something trendy.

A little more hard to quantify is also the matter of tone...Baroque music has a pretty architectural quality, and harpischords and lutes do it pretty well. Music by the later 18th c. was trending more sensual, with influence of the Romantics, and lush tones and wider dynamic range became more important. Harpsichords aren't as sensual or dynamic as pianos, and anyone needed something lute-like increasingly went for a guitar.