If you drink a lot of coffee you will probably feel a little gross after the initial buzz wears off, which could affect your sex drive at that moment.

As far as long term side-effects, I think it's pretty safe.