To find work, Baker's father and several of his siblings moved to Norfolk. Eventually, Blake Baker settled into a job working on a steamer that ran from Norfolk to Washington, D.C.