The correct answer is C. the little boys of Baltimore who had taught him how to read.
He specifically says he was not attached to Master Hugh or his wife, as both of them had changed very much - Master Hugh because of alcohol, and his wife because of slavery. Frederick goes to live with Mr. Thomas and his new wife. He felt the strongest attachment to the boys, and was now desperate, believing he would never see them again.