Let's be clear "castle" is really a convenience term. Exactly what we mean by this varies, e.g. between languages and time and place, and a lot of the categorisation is done long after the constructs themselves lost relevance of original purpose.

The word derives from Latin castrum (camp, fort, citadel, stronghold), and it's diminutive castellum (small camp, fort). This is relevant as the people building these structures would in writing, Latin to start with, use either word but not necessarily in particular preference (or using the same distinction as Romans would have e.g.. Something that started as a castellum would often end up being further developed into a castrum.

As is often the case it is sometimes easier to define what something is by looking at what it isn't.

Iron-age hilltop forts are usually called "forts", not castles (by us), because we see them as distinct from castles, both in design and timeperiod and use. A hill-top fort is a natural feature, like crags and hills, augmented with earthen banks, stone and wood, in various configurations (either one, two or all three as local construction allows or has been possible) forming an enclosed protected space you can defend more easily than before. I'm putting this in the context of how we get to castles, obviously forts were continued in use in various forms.

The other defensive structure is the stand alone tower or manor, often just referred to as "house" in some languages where it denotes a (fortified)stonebuilding. Basically a solitary "keep" or fortified manor. These were actually often called castles, e.g in old Swedish "hus" i.e. modern house, specifically meant a stonestructure of this type and the term remains in the name of many castles (often built as expansion to such an original structure). The keep of course combines living and defence into one imposing package for the aspiring lord.

There is a reason I brought these two up. Because if you combine the "central" (not necessary literally) keep with the protective wall of forts you get what I would argue is the central distinction of a "castle". A minimally (two-)layered defence where the two parts exist to support each other. Arguably almost anything we recognize as a castle will adhere to this.

A third point could be that the defensive structure is inhabited "permanently". It has a garrison that lives there, but the same can be said of many forts, or if we so wish, fortresses. So I sort of see this as a weaker denominator.

A walled/fortified town/city of course also contains permanent habitation, but it lacks the central keep that is the mainstay of the wall+keep=castle combination and therefore lacks the defensive in-depth concept. I.e. the town within is in itself the reason for the walls.

One of the earliest castles as we would here define it is the motte & bailey, a (often) artificially made hill with a keep (motte) and a walled courtyard (bailey) popularised by the Norman conquest of England when they used this technique to stamp their authority on the country.

I'll add that while I try to categorise things here for ease that doesn't mean reality is always so neat. If a noble builds a manor at the end of a peninsula that he walls off with moat and walls, does that necessarily constitute a castle? And as mentioned earlier nobles would refer to their stonehouses as castrum even though they might not be as extensive that they had walls. Some later castles (eg renaissance palaces and those of the later 1600s and 1700s) wouldn't necessarily contain all the features but still be, if nothing else colloquially, know as castles. Some of course being built on former sites of mediaeval castles.

I'm also materials agnostic. Many early castles used mainly wood and were perfectly viable for a time before siegecraft improved. Especially the Baltic and Scandinavia saw many. Later on stone would replace the most strategically sited and important ones, or brick in the later medieval age (a prestige material in the aforementioned regions) and more easily to transport through the basically roadless wilderness.

TL:DR I would say the basic feature that makes a castle is a combination of wall+keep that's mostly permanently habited for some kind of defensive purpose.