The Pilgrims were religious and political dissidents who thought English society was going to hell(literally). Virginia had already been settled, and the Puritans wanted to separate themselves from English society, so they didn't want to settle there. Further south than Virginia they would have been hassled by the Spanish and pirates.

New England itself had been "advertised" as being similar to the English climate, with plenty of 'unclaimed' land and tons of 'untouched' wilderness just begging for English 'civilization'. This attracted the first Puritan leaders, who saw the harsh winters as a blessing from God because it allowed them to work hard.