This seems oddly specific. I have not read any accounts of agriculture being abandoned in Sweden before, especially in such a narrow timeframe.

What we do know now, due to extensive archeological and more recently DNA testing of grave sites etc is that Sweden was populated in several waves. Recently resarchers have identified three distinctive groups that are the ancestors of the modern Swedes (but not the Sami, who migrated to northern Sweden from the east). In general, there was a group with prevalance of dark-skinned and blue-eyed individuals who migrated to Sweden after the ice age and lived as hunterer-gatherers and a second group that came from Ukraine and generally had lighter skin and darker eyes and lived off fishing and seal hunting. Both these groups more or less merged into one, and were then assimilated by a third group that came from Anatolia, bringing agriculture and domesticated crops with them.

The merged first and second groups and the third group lived side by side for hundreds of years, having different burial traditions and many other things. The hunter-gatherers seem to have learned pottery from the farming group at this time, as distinct pottery shows up at their settlement sites and in grave gifts. However, the farming group seem to have taken in and assimilated the hunter-gatherer group, as they have some of their DNA, while the opposite is not true - the hunter-gatherers do not have DNA from the farming group.

As far as I know, both groups were fairly fluid in migration, and would move back to better grounds in case of famine or failed crops - while it is hard to know for certain, the general consensus in Sweden is that the farming culture came and might have went back for a while, and eventually assimilated the hunter-gatherer culture as evident by DNA, due to being much larger in numbers. The hunter-gatherers did not adapt and then abandon agriculture and then re-adapt it - while some of them joined agricultural communities and eventually had children with the farming group as evident by DNA testing of skeletal remains at burial sites.

You can read about Project Atlas, a large scale project identifying migration paths, cultures and ethnic groups and the research they have done recently.

The "evidence" is the Pitted Ware culture, a hunter-gather culture that appeared in the area after farming. The arrival of this culture coincided with the disappearance of some of the earlier, farming settlements, so the theory that this culture developed from the already-existing farming culture (the Funnel Beaker culture) through the abandonment of farming was plausible. This theory was commonly, but not universally accepted; this is probably the basis of Diamond's statement.

However, even at the time when this was a common theory, it wasn't the only one - the other main theory being that a separate hunter-gather people lived alongside the farmers (and a 3rd view being that the Pitted Ware culture was just the fishing/sealing people of the farming culture). Genetics has shown that the 2nd theory is basically correct, and u/vonadler gives the outline of the genetic picture. It should also be noted that farming didn't disappear from southern Sweden - some farming settlements disappeared, but far from all.

The Pitted Ware culture was coastal, and a large part of their hunting and gathering was fishing and sealing. They appear to have depended heavily on trade, shipping seal oil in ceramic jars across the Baltic, and trading with their inland farming neighbours as well.

A good overview of the Pitted Ware culture, including a brief discussion of the competing theories for their origin is given by Marek Zvelebil, "Pitted Ware and related cultures of Neolithic Northern Europe", pp 431-435 in P. Bogucki and P.J. Crabtree (eds.), Ancient Europe 8000 BC–AD 1000: Encyclopaedia of the Barbarian World, Vol. I The Mesolithic to Copper Age (c. 8000-2000 B.C.), Charles Scribner's Sons (2004).