"You have a very ... rose-colored view of how this all pans out."

Because now he was catastrophizing in his head. He would never find the sorcerer's hat again. And at least one person who did not want him in Storybrooke knows where the dagger is. Plus one person dead set on getting back in there who might stop at nothing to find it.