The setting is the fictional frontier town of Dawson's Landing on the banks of the Mississippi River in the first half of the 19th century. David Wilson, a young lawyer, moves to town, and a clever remark of his is misunderstood, which causes locals to brand him a "pudd'nhead" (nitwit). His hobby of collecting fingerprints does not raise his standing in the eyes of the townsfolk, who consider him to be eccentric and do not frequent his law practice.

"Pudd'nhead" Wilson is left in the background as the focus shifts to the slave Roxy, her son, and the family they serve. Roxy is one-sixteenth black and majority white, and her son Valet de Chambre (referred to as "Chambers") is 1/32 black. Roxy is principally charged with caring for her inattentive master's infant son Tom Driscoll, who is the same age as her own son. After fellow slaves are caught stealing and are nearly sold "down the river" to a master in the Deep South, Roxy fears for her son and herself. She considers killing her boy and herself, but decides to switch Chambers and Tom in their cribs to give her son a life of freedom and privilege.

The narrative moves forward two decades. Tom Driscoll (formerly Valet de Chambre), has been raised to believe that he is white and has become a spoiled aristocrat. He is a selfish and dissolute young man. Tom's father has died and granted Roxy her freedom in his will. She worked for a time on river boats, and saved money for her retirement. When she finally is able to retire, she discovers that her bank has failed and all of her savings are gone. She returns to Dawson's Landing to ask for money from Tom.

Tom responds to Roxy with derision. She tells him the truth about his ancestry and that he is her son and partially black; she blackmails him into financially supporting her.

Twin Italian noblemen visit Dawson's Landing to some fanfare, and Tom quarrels with one. Desperate for money, Tom robs and murders his wealthy uncle and the blame falls wrongly on one of the Italians. From that point, the novel proceeds as a crime novel. In a courtroom scene, the whole mystery is solved when Wilson demonstrates, through fingerprints, both that Tom is the murderer, and not the true Driscoll heir.

Although the real Tom Driscoll is restored to his rights, his life changes for the worse. Having been raised as a slave, he feels intense unease in white society. At the same time, as a white man, he is essentially excluded from the company of blacks.

In a final twist, the creditors of Tom's father's estate successfully petition the governor to have Tom's (Chambers) prison sentence overturned. Shown to be born to a slave mother, he is classified as a slave and is legally included among the property assets of the estate. He is sold "downriver", helping the creditors recoup their losses.