I stumbled on the Gardner one snowy day a number of years (OK, quite a few years) ago, with absolutely no idea what to expect. It was one of the most wondrous experiences of my life, stepping from cold, dirty-snowy Boston into the warm, lush interior with all that wonderful art, all much more accessible than in most museums (though the arrangement was admittedly somewhat quirky).

It's definitely worth a visit or three if you're in the area.

And I second all the lusting after the library. If I had something like that I'd never get anything else done, though.
Thinking of Boston and libraries, I've always wanted a library patterned after the Naumburg rooms in the Fogg Museum. Sigh.