A possible Lord John candidate?
One of the characters in "Downton Abbey" is a young man named Matthew Crawley, played by Dan Stevens. IMDB says Dan Stevens was born in 1982, which makes him exactly the age that Lord John is in BOTB. He may be a little too tall for Lord John's 5'6", but otherwise I think he fits my mental image pretty well. And his voice is very much like what I think Lord John sounds like, too.
Here are some more pictures. What do you think? The one shown above is my favorite.
UPDATE 2/15/2012 7:00 pm: Check out my blog post comparing "Downton Abbey" and OUTLANDER.