The novel is set in the mid-19th century, but flashbacks to the history of the house, which was built in the late 17th century, are set in other periods. The house of the title is a gloomy New England mansion, haunted since its construction by fraudulent dealings, accusations of witchcraft, and sudden death. The current resident, the dignified but desperately poor Hepzibah Pyncheon, opens a shop in a side room to support her brother Clifford, who has completed a thirty-year sentence for murder. She refuses all assistance from her wealthy but unpleasant cousin, Judge Jaffrey Pyncheon. A distant relative, the lively and pretty young Phoebe arrives and quickly becomes invaluable, charming customers and rousing Clifford from depression. A delicate romance grows between Phoebe and the mysterious attic lodger Holgrave, who is writing a history of the Pyncheon family.

The house was built on ground wrongfully seized from its rightful owner, Matthew Maule, by Colonel Pyncheon, the founder of the Massachusetts branch of the family. Maule was accused of practicing witchcraft and was executed. According to legend, at his death Maule laid a curse upon the Pyncheon family. During the housewarming festivities, Colonel Pyncheon was found dead in his armchair; whether he actually died from the curse or from a congenital disease is unclear. His portrait remains in the house as a symbol of its dark past and the weight of the curse upon the spirit of its inhabitants.

Phoebe arranges to visit her country home, but plans to return soon. Clifford, depressed by his isolation from humanity and his lost youth spent in prison, stands at a large arched window above the stairs and has a sudden urge to jump. The departure of Phoebe, the focus of his attention, leaves him bed-ridden.

Judge Pyncheon arrives at the house hoping to find information about land in Maine, rumored to belong to the family. He threatens Clifford with an insanity hearing unless he reveals details about the land or the location of the missing deed. Clifford is unable to comply. Before Clifford can be brought before the Judge (which would destroy Clifford's fragile psyche), the Judge mysteriously dies while sitting in Colonel Pyncheon's chair. Hepzibah and Clifford flee by train. The next day, Phoebe returns and finds that Holgrave has discovered the Judge's body. The townsfolk begin to gossip about Hepzibah and Clifford's sudden disappearance. Phoebe is relieved when Hepzibah and Clifford return, having recovered their wits.

New evidence in the crime that sent Clifford to prison proves his innocence. He was framed for the death of his uncle by Jaffrey (later Judge) Pyncheon, who was even then looking for the missing deed. Holgrave is revealed as Maule's descendant, but he bears no ill will toward the remaining Pyncheons. The missing deed is discovered behind the old Colonel's portrait, but the paper is worthless: the land is already settled by others. The characters abandon the old house and start a new life in the countryside, free from the burdens of the past.