Frederick Douglass was born into slavery on the Eastern Shore of Maryland in 1818, and was given the name Frederick Augustus Washington Bailey (Baly), after his mother Harriet Bailey. During the course of his remarkable life he escaped from slavery, became internationally renowned for his eloquence in the cause of liberty, and went on to serve the national government in several official capacities. Through his work he came into contact with many of the leaders of his times. His early work in the cause of freedom brought him into contact with a wide array of abolitionists and social reformers, including William Lloyd Garrison, Elizabeth Cady Stanton, John Brown, Gerrit Smith and many others. As a major Stationmaster on the Underground Railroad he directly helped hundreds on their way to freedom through his adopted home city of Rochester, NY.

Renowned for his eloquence, he lectured throughout the US and England on the brutality and immorality of slavery. As a publisher, his North Star and Frederick Douglass' Paper brought news of the anti-slavery movement to thousands. Forced to leave the country to avoid arrest after John Brown's raid on Harper's Ferry, he returned to become a staunch advocate of the Union cause. He helped recruit African American troops for the Union Army, and his personal relationship with Lincoln helped persuade the President to make Emancipation a cause of the Civil War. Two of Douglass' sons served in the 54th Massachusetts Regiment, which was made up entirely of African American volunteers. The storming of Fort Wagner by this regiment was dramatically portrayed in the film Glory! A painting of this event hangs in the front hall at Cedar Hill.

All of Douglass' children were born of his first marriage, to Anna Murray. He met Murray, a free African American, in Baltimore while he was still held in slavery. They were married soon after his escape to freedom. After the death of his first wife, Douglass married his former secretary, Helen Pitts, of Rochester, New York. Douglass dismissed the controversy over his marriage to a white woman, saying that in his first marriage he had honored his mother's race, and in his second marriage, his father's.

In 1872, Douglass moved to Washington, D.C. where he initially served as publisher of the New National Era, intended to carry forward the work of elevating the position of African Americans in the post-Emancipation period. This enterprise was discontinued when the promised financial backing failed to materialize. In this period Douglass also served briefly as President of the Freedmen's National Bank, and subsequently in various national service positions, including U.S. Marshal for the District of Columbia, and diplomatic positions in Haiti and the Dominican Republic.