Sounds like there might be some interference or a weak signal (also typical of Bluetooth). You'll have to keep trying. Once it pairs it should work fine from then on.
What kind of phone do you have? Newer phones will have better Bluetooth radios, protocols, and sw.