The Tory House on Central Square facing east and the town common is a two-and-one-half story center-chimney house decorated in the Federal style. The main block is augmented by a two-story ell on the northeast corner. The ell was moved to the site and attached to the right rear of the main block in the nineteenth century. The entrance doorway with its three-light transom and fluted column entablature exhibits Federal era details. Nineteenth century double-hung windows, all of similar dimensions, fenestrate all of the elevations. Earlier, possibly eighteenth century, windows with twelve-over-twelve double-hung sash remain in the upper level of both side elevations.

The original First-Period house was a single-cell, end chimney structure now the front, southeast corner of the house. This original portion of the house has a decorated binding summer beam which runs from the front girt to the rear girt. The ten-and-one-half inch planed summer is decorated with a one inch flat chamfered edge, a lamb's tongue stop, and small pip. All other interior architectural features in the room, such as the fireplace, indicate Federal period renovations. Framing evidence in the cellar of the house, such as foundation wall and wood materials, seems to indicate simultaneous construction of the southwest bays and second pile rooms. These rear rooms include a staircase in the northwest corner and a middle room with seven-foot wide fireplace. Another small room is located behind the southeast room. This first addition appears to have been built in the first half of the eighteenth century There are no exposed framing members now visible in the southwest room, the rear first-floor rooms or in any of the second-story rooms. The southeast exterior wall on second floor is, however, noticeably deeper than the walls in the center bay and southwest portion of the house.

It is likely that a third addition made at the end of the eighteenth century raised the roof to its present gable form. The Federal style mantels and entrance door entablature may also have been added at this time. A curved staircase just inside the front entrance may also date from this or a fourth renovation in the middle of the nineteenth century.

Today the Tory House is one of the few residential buildings in the immediate area of the town center. Vestiges of the early town common remain and now serve as a traffic island between the divided state highway also running in front of the house. It is surrounded by a combination of nineteenth and twentieth century public and commercial structures. Set some twenty-five feet from the sidewalk, the property is bounded by an iron fence. Shade trees and a grassy area separate the house from the road immediately in front of the house. An asphalt driveway on the left-side of the house leads to a commercial building at the rear of the property and other businesses and a parking area are on the right-side of the house. Additionally, the South Parish Church is two buildings to the left of the house and the Town Hall is across the common and on the opposite side of Central Square.