If you really love him then tell him! Find out what he is thinking....I told my 'gf' i loved her even that we never ever hung out and she was a bitch about things and blowing me off like a motha ****er....So let him know what your thinking! Who cares? If anything, he will say it back. Who knows, he might even mean it.
But dont let this build up inside of you only to destroy your thoughts. Your gunna die someday just like everyone else so i kinda live like 'who cares, as long as its right and not evil'....lol.