In Puritan Boston, Massachusetts, a crowd gathers to witness the punishment of Hester Prynne, a young woman who has given birth to a baby of unknown parentage. She is required to wear a scarlet "A" on her dress when she is in front of the townspeople to shame her. The letter "A" stands for adulteress, although this is never said explicitly in the novel. Her sentence required her to stand on the scaffold for three hours, exposed to public humiliation, and to wear the scarlet "A" for the rest of her life. As Hester approaches the scaffold, many of the women in the crowd are angered by her beauty and quiet dignity. When demanded and cajoled to name the father of her child, Hester refuses.

As Hester looks out over the crowd, she notices a small, misshapen man and recognizes him as her long-lost husband, who has been presumed lost at sea. When the husband sees Hester's shame, he asks a man in the crowd about her and is told the story of his wife's adultery. He angrily exclaims that the child's father, the partner in the adulterous act, should also be punished and vows to find the man. He chooses a new name, Roger Chillingworth, to aid him in his plan.

The Reverend John Wilson and the minister of Hester's church, Arthur Dimmesdale, question the woman, but she refuses to name her lover. After she returns to her prison cell, the jailer brings in Roger Chillingworth, a physician, to calm Hester and her child with his roots and herbs. He and Hester have an open conversation regarding their marriage and the fact that they were both in the wrong. Her lover, however, is another matter and he demands to know who it is; Hester refuses to divulge such information. He accepts this, stating that he will find out anyway, and forces her to hide that he is her husband. If she ever reveals him, he warns her, he will destroy the child's father. Hester agrees to Chillingworth's terms although she suspects she will regret it.

Following her release from prison, Hester settles in a cottage at the edge of town and earns a meager living with her needlework, which is of extraordinary quality. She lives a quiet, somber life with her daughter, Pearl, and performs acts of charity for the poor. She is troubled by her daughter's unusual fascination with Hester's scarlet "A". The shunning of Hester also extends to Pearl, who has no playmates or friends except her mother. As she grows older, Pearl becomes capricious and unruly. Her conduct starts rumors, and, not surprisingly, the church members suggest Pearl be taken away from Hester.