There are parts of Dan that he loves (his brilliance with words, the way he can follow Casey's non sequiturs, his ongoing infatuation for all things New York) and parts of Dan that turn him on (the curl of his lips, the strength of his fingers, the slope of his neck) and Casey thinks that if he could somehow divide Dan in two, separate those parts from everything else, they could have love and romance and a house in the country and a happily ever after.
It's the other parts that hold him back. The parts of Dan that are covered in scars, closed over but never healed. The parts that surface every few years and wrap their tentacles around the parts Casey loves and pull Danny down to a dark place Casey doesn't understand. If he could just get rid of those parts...
But he can't. And he can't see how he could have love and happiness and the whole thing if the lurking parts of Dan, the parts that scare him, are part of the equation.
So Casey keeps part of himself, the part that wants to take Dan in his arms and not let go, to himself.