Reports in the 18th century of such impromptu boxing matches are reasonably common, but the reports themselves are pretty vague. There are sometimes references to rings being put up etc. so there does seem to have been rules atleast in some instances.

—

Caledonian Mercury - Monday 13 November 1721

Some time ago very high Words arose, which begat Wagering, which begat Boxing, which begat a Challenge to a Duel, and when like much of the South Sea Contracts, remains as yet impersorm’d, between an Englishman and a Dutch Jewler, at a publick House in Clare market, which was occasioned by the formers reflecting upon all the [Producing] of the latters Country, viz. Holland, saying that a Dutchman is [thick] a Dutchman is squab; saying a Dutchman is round, a Dutch dog [is] short, a Dutch sSip is broad-bottom’d, and in a Word, that we would think that all the Commodities of the Country were cast in the same Mould with their Cheeses.

—

Southampton Hampshire Chronicle - October 11, 1773

Last Tuesday two servant men, belonging to a farmer at Chalfont, in this county, having some words about a young women, which each of them paid his addresses to, agreed to decide the matter by a boxing round; one of them struck at his antagonist with great force, but missing his aim, fell with great violence; his head picching against a stone, fractured his skull, and expired before any assistance could be had.

—

London Evening Post - January 13, 1774

A severe boxing match was fought yesterday morning in a field near Pancras, between the son of a gentleman of fortune at the West end of the town, and a schoolmaster, wherein the latter received an unlucky blow on the temple, which killed him on the spot. It is said the young gentleman held him a grudge on account of some cruel treatment he received from him when he was his pupil, and meeting him accidentally a few days since at a coffee house in the city, took him by the nose, which was the occasion of the battle, The deceased, it seems, bot understanding the sword, chose to decide the quarrel with their fists, rather than fight with pistols.

—

I think such encounters were probably not the beginings of modern boxing however. They came from around the same time or later than reports of boxing as a sport. The early history of modern boxing is far too poorly recorded to come to much strong conclusions about it's origin. There was awareness of ancient boxing. My general impression is that the increase in the reports of such encounters seem to come after reports of the more formal contests, with many being in the 1740s and 1770s, both of which seem to have had an increase in the popularity of boxing.

The earliest report known of modern boxing in Engalnd is from the late 17th century, like many of the reports, it's extremely vague.

—

Protestant Mercury December 28, 1681 (page 4) Yesterdaya Match of Boxing was performed, before his Grace Duke of Albemarl, between the Dukes Foot-man and a Butcher, the latter won the Prize, as he hath done many before, being accounted (though but a little Man) the best at the Exercise in England​

—

The common locations for the early 1700s boxing matches included the Bear Garden (named for Bear baiting), as well as ampitheatres, such as those owen by Stokes and James Figg (who likely wasn't a boxer at all BTW, despite often being listed as the first Champion of England). There is quite a lot of association with things like Cock fighting, as well as the fencing contests and cudgel fighting.

For example here's an advery for a contest between Thomas Allen (known as Pipes) and Jack Broughton (often called the father of boxing)

—

The Daily Post- 3 May 1731 (page 1)

At Mr. STOKES’s AMPHITHEATRE,

In Islington Road, this present Monday,

Being the 3d Instant, will be a great

Entertainment, Viz.

Variety of Bull-Baiting, Bear-Baiting, an Ass dress’d with Fire-works, and a Bull dress’d in the like Manner.

Two Dogs fight a Match first at the famous Hertford hire Bulls fairest and nearest his Mark for one Guinea.

A Pair of Cock fights for Ten Shillings.

Attendance will be given at Two and the Diversion begin at Five. And To-morrow, being Tuesday, will be fought the great Boxing Match between the famous Thomas Allen, Pipemaker, and John Broughton, for 100 Guineas. He that does not mount the stage exactly at Six o’Clock forfeits ten Pounds.

—