I may answer my own question.
I could do it manually, but I am looking for an "elegant" solution.
For example:
If I manually calculate it, I'm only solving this instance of the issue. Should be a way to elegantly translate into or translate out of the viewport.mouse click at 64, 180 in client window
Where does it hit a viewport from (100,100)-(700,500) scaled with glOrtho to have coordinates of (0,0) through (320,200)
[Yes, I'm matrix rusty. I bet a year ago I would have solved this one in 10 seconds flat. ]