Undoubtably you have detached an Ivy Ball in the past, and had some other balls fall off because they happened to be walking on its strands. You can avoid this easily. Simply click and hold on the ball for a moment. Any other balls on its strands will move away, and will not return as long as you hold the button down.