While Christians are called to love and serve the world, I don't think that means we don't have needs for our own close relationships. Jesus Himself knew of and forgave Peter's betrayal, but that doesn't mean He took any crap from him—He didn't. So I'm not convinced it's selfish for closeness to have conditions. I'm not off the hook to love and serve even someone who stands on my last nerve, but I'm not required to put on a yoke of friendship if the other person isn't going to pull, too.
Real presence (not necessarily the physical kind) is something that's legitimate for me to desire. As my seminary professor said, "Don't give him absence." This desire must be tempered with an understanding of the ways we will constantly fail one another. The grace of Christ must inform and infuse my relationships. But if even that desire isn't evident, there's the very real possibility that we aren't headed in the same direction and shouldn't share a yoke. That's OK, and probably a far more grace-filled realization than simply continuing to try to relate in ways that lead to starvation from real presence. God Himself has declared that He wants me to live in His presence—it's a quality that's worth having in my close relationships as well.