- Latimer was the youngest of four children born to George and Rebecca Latimer, who had escaped from slavery in Virginia six years before his birth. Captured in Boston and brought to trial as a fugitive, George Latimer was defended by abolitionists Frederick Douglass and William Lloyd Garrison. He was eventually able to purchase his freedom, with the help of a local minister, and began raising a family with Rebecca in nearby Chelsea
- Latimer married Mary Wilson in 1873, and they had two daughters together
- Latimer enjoyed other creative pastimes, including playing the flute and writing poetry and plays.
- In 1864, at the age of 16, Latimer lied about his age in order to enlist in the United States Navy during the Civil War.