There's always a possibility, but it's very small. Sperms are agile little bastards, especially when they are not needed. I wouldn't worry, but you should be careful.