The Nelson Arms Tavern was originally three 14th Century farm cottages that belonged to Castle Farm, part of the Bishop of Winchester's estate and residence at Farnham Castle.

The frameworks of these cottages form the rear section of the pub where the ceilings are at their lowest. The cellar beneath the pub also follows this pattern and disappears into a sealed tunnel that is rumoured to run to the Castle.

In 1604, on the north (uphill) part of the building was a Public House, and it was known as the 'Hand and Pen', named for the Scriveners that worked in Castle Street duplicating important legal documents for the Bishop and Law Courts of Winchester. Most of the interior structures you can still see in this part of the pub were in-situ at this time, with the forward supporting beams and uprights believed to have been recycled from a Tudor warship.

Farnham was an important focal point during the English Civil War (1642-1651) but the pub remained structurally unchanged throughout the 1600's. At this time, the large lanterned fireplace was an even larger Inglenook with built in seating and the bar was hatch in the opposite rear corner, where, although now partially built over with a staircase, the tile floor remains from a time this was refurbished as a Jug & Bottle area (an early type of off-licence). There have been several sightings of a ghost in the Civil War period dress beside the larger fireplace.

From the Nelson's rear courtyard you can see the three pitched roofs on the Nelson that cap the footprint of the original buildings. The fourth pitched roof (on the corner of Park Row) caps an extension built by a carpenter named Robert Knight after he purchased the pub in 1729. The extension he built contained a large oven/fireplace (now the bar section containing the larger pumps, the ceiling of which is at the bottom of the chimney) and by 1739 the pub was known as The Bakers Arms.

The Bakers Arms pub continued to thrive throughout the 18th Century and eventually occupied the Bakery toward Park Row as the pub does today. Eleven hand-carved ivory fish used in the drinking games in the 18th Century Anglers were discovered in the Park Row Corner during renovations in 1926.

The present pub name is a result of the local support for Admiral Horatio Nelson. His visits to the town to see his paramour, Lady Hamilton, who lived on Firgrove Hill (a large house since replaced by Trafalgar Court), probably played a part in this. Their last meeting in Farnham was believed to have been in the grounds of Farnham Castle and there are many rumours of their visits to the pub.

The Arms displayed by the pub are of a design that predates the Battle of Trafalgar and are unusual in that they are hand carved.

The pub once held a famous 'Nelsons' glass eye lodged in one of the wooden beams, but alas, it was liberated long ago.