Getting away from your friends is a good idea. It seems like you have a good thing going already, keep flirting. Is he the shy type to you or not? If so, and you want to progress then write down your no. if not, then wait for him to ask, but make it obvious you're interested.

That's what I would do anyway.