There was the possibility that people far less kind than Emma Swan (whom he realized he had not done much to keep tabs on -- knowing she was somehow from an earlier time made things feel precarious, and she had thankfully not tried to meddle with him or fuss). People who would not hesitate to "warn" others about him. At a time when he had as little magic as they would, it would have been even ground for a time.

And of course, there was another part of that. There were some people that might appear here, and his goodwill and reputation would be threatened not because of anything they could say? But because he would probably kill them and would not be subtle about it.

"To having a place to start over without others to meddle, then."