Former Residence of Yoshida Shoin, Exiled