To start, New Jersey isn’t exactly “fragmented” - it is, as you stated, the most densely populated state in the union, and forms an integral part of two of the nation’s largest metropolitan areas, New York City and Philadelphia. City limits in the U.S. are often a poor way to judge size, anyways, considering that Mesa, Arizona is larger than Miami, Atlanta, or Kansas City. Once you consider metropolitan areas, New Jersey seems a lot more prominent - just within the top ten, Newark, Jersey City, and Camden all appear as integral parts of some of the country’s largest metro areas. So really, city limits are a poor way to judge a city’s size, and in general it’s probably more accurate to look at MSAs or CSAs, combined statistical areas.

Historically, New Jersey was one of the first states to industrialize, and Newark, in particular, prospered tremendously from the growth of industry in the state. By 1930, its population had grown to over 440,000, making it the 18th largest city in the country, arguably a “traditional big city”. From that point forward, however, its population consistently declined until the 2010 census. So what happened? A combination of factors, the most obvious being deindustrialization and suburbanization, as Newark’s white population fled the inner city to more distant suburbs. Here is an excellent series of maps that show this process in action. Of course, this was happening across the nation, but Newark’s heavily industrial economy made it even more susceptible to decline, and high racial tensions “accelerated [changes in ethnic composition] after the city was wracked by rioting in 1967.” (https://www.britannica.com/place/Newark-New-Jersey) A similar shift happened in the Philadelphia suburb of Camden, as it shifted from a predominantly white, industrial city to an impoverished, minority-majority community. There’s plenty of existing literature on practices such as redlining which disadvantaged minority residents and led to poor, minority-majority urban centers, and I won’t spend time outlining it all here, but it was far from a New Jersey-specific phenomenon - however, as a heavily industrial state, deindustrialization hit New Jersey harder than most, and so the urban decline was greater than most.

Aside from that - and this is just a hypothesis which someone more versed in the subject can affirm or refute - I’d imagine New Jersey’s status as a state that is in some ways essentially a New York/Philadelphia suburb has contributed to its relatively “fragmented” population, since the wealth and population is going to concentrate just across the state border, while smaller supporting communities spring up across the Hudson and the Delaware, respectively. Suburbs are rarely “huge cities” population-wise without space to sprawl, which New Jersey lacks, and even in those cases they rarely surpass or equal their “central city” in the public imagination. (The only that comes to mind are places like Fort Worth or Oakland, and even those are still rarely held to the same level as Dallas or San Francisco.) I mentioned Mesa earlier, because despite being the 35th largest city in the nation, it’s not the type of city that would come to mind as a “big city”, because it’s just a Phoenix suburb. City limits are misleading.

The answer to your question is simple. Jersey's cities are small because they are geographically small. Go on google maps and click on somewhere like "Guttenberg", and see that the city limits aren't much bigger than a few blocks. Even somewhere like Jersey City has city limits that are maybe a third the size of Manhattan Island.

The more complicated question is: why are Jersey's cities so small? It's not that Jersey is a rural place, it's the most densely populated state in the USA, just as you've noted. The entire Northeast corner of the state is a big urban area with a population of around 4 million, and it would easily be one of the largest cities in America if it were a single city. Even if it were divided into 2-3 major cities, they would still be very large.

But this is New Jersey, and the NE is divided into more than a hundred municipalities. They have strange borders and aren't really distinct from each other, but they are independent municipalities nonetheless.

For example, the borders of Guttenberg, encompass a city composed 100% of midrise buildings. Surroundnig these borders are identical mid-rise city blocks its neighbours of North Bergen and West New York (yeah). The City of East Orange is entirely covered by suburbia, and it is surrounded by identical suburbia surrounding Newark. In size, locality, and function, these cities are more like neighbourhoods of a larger city. Unlike a neighbourhood however, they all have their own mayors, councils, and related administration.

In New Jersey, there are cities, towns, townships, boroughs, and villages. They are all politically equivalent when it comes down to legal standing, services, and powers, despite featuring slightly different forms of government. Collectively, they are referred to as municipalities, and there are 556(!) of them in total. From afar, it's hard to see any particular reason why North Bergen is a township, Guttenberg is a town, and Fairview is a Borough, when all three seem to be small neighbourhoods of Jersey City (Or NYC if you're feeling cynical).

However, each municipality of Jersey represents the culmination of some political movement. From 1834 to 1958, some 400 municipalities were created from localized secession movements. The New Jersey legislature made it possible for secessionsits to self-execute bindings referendums fairly easily, and it sort of became a common move for Jerseyites to rally behind secession for all sorts of things.

The former New Jersey lawmaker Alan J. Karcher lists five main reasons for municipal secession in New Jersey

1. "Street Fights". Parts of an existing municipality would elect road superintendents that were supposed to raise funding in from the municipal government for road funding. When they failed, the irate residents could use their powers of secession to form a new municipality, create their own laws and possibly setting up a toll road or using some other way of raising funds. Sometimes, a crafty road superintendent would just rile people up this way and set themselves up as a mayor or councilor for a new municipality.
2. Commuters vs. Farmers. The railroad allowed for the economical creation of commuter towns in Northeastern New Jersey for workers in NYC. Tensions thus rose between these workers that enjoyed urban amenities and farmers who didn't. Political spats between the populations sometimes resulted in the spontaneous secession of a new municipality out of an older township.
3. School districts. This was huge. An 1894 law in the state legislature basically mandated any township to consolidate its school districts into one. Previous school districts followed no boundaries. However, it also mandated that the collective finances of the old school districts be passed onto the new one. Most importantly, it also stated that any town, city, or borough HAD to have its own independent school district. This all combined into a massive explosion of boroughs seceding from townships, mostly along lines of the former school districts trying to escape from debt.
4. Dry Towns vs Wet Towns. As the Prohibition movement grew, New Jersey reacted predictably. Municipalities going dry would see secession movements from people wanting to keep their bars. Muncipalities failing to go dry would see secession movements from people wanting to ban alcohol. Much confusion and certain unenforcability ensued.
5. Exclusion. In a word, people seceding from larger municipalities specifically in order exclude people of other groups. Racism directed at African-Americans was a part of this, but this would also be directed against Catholics, or specific denominations of Protestants.

These factors existed throughout the ~100 years when most of New Jersey's municipalities were created, fading in and out as local politics shifted. From 1958 to 2019, only 2 of the 568 municipalities have agreed to consolidate with another, meaning that when you drive through Bergen county and whip by the dozens of signs telling you you're in "so-and-so" borough, you're passing through a physical boundary created by political spats that occured at least 50 years ago.

Now I'm not a resident of New Jersey, so I have no idea how this functions, how it's maintained, nor if this represents the average New Jerseyite's vision of an ideal government. I will say that it's confusing, weird, and interesting.