The purpose of a castle (or, really, any fortification) is not limited to defending those areas within arrow shot of its walls. Castles of this sort are best thought of as fortified bases that project power. Yes, it would be possible to land a mile or two away, but that doesn't remove the essential problem. If the invading force moves inland, its (very tenuous) supply lines are left open to the castle's garrison, who can ride out, raid it, and retire within the safety of the castle's walls. If the invading force leaves troops to protect its supply lines or besieges the castle, that allows the defenders (England) time to gather troops and meet the invasion.