Probably in an anthology. Judging by age of local library collections, probably well over 40 years old. Lead character is somehow recruited by aliens to live in a special dwelling disguised as an ordinary house. As long as he stays inside the house he does not age. On most days he would only age the few minutes it takes him to walk back and forth to his mailbox and possibly chat with his postman. One of the characters has the hobby of collecting samples of unusual species of wood such as ebony and mahogany. I also remember a minor plot element that may or may not have been from this story by which a sort of password would be generated by reading the time from a special clock that operated with eccentric or elliptical gears and therefore would usually display something different than the actual time. I would have been likely to browse through anthologies containing authors like Asimov, Blish, Bradbury, Clarke, Pohl, etc.