Cover Thumbnailer is a small Python script which displays music album covers and basically any image that's inside a folder as a preview for that folder. It comes in a .deb or .rpm package and it currently only works with Nautilus.
You can only enable it for music or pictures folder, select the folders for which to create previews or blacklist certain folders - all from it's preferences which can be accessed via System > Preferences > Cover Thumbnailer.
Cover Thumbnailer 0.8 was just released, brining a resize method for the music folders: crop or preserve, mosaic mode when there is more than one music cover as well as an option for the pictures folders: you can now chose the maximum number of pictures to display on the thumbnail (from 1 to 4). Another new feature allows you to change the Nautilus thumbnail size from within Cover Thumbnailer.
Ubuntu (9.10 Karmic, 10.04 Lucid and 10.10 Maverick) users can install the latest Cover Thumbnailer from its PPA:
sudo add-apt-repository ppa:flozz/flozz && sudo apt-get update sudo apt-get install cover-thumbnailer
Once you install Cover Thumbnailer, you need to restart Nautilus for the thumbnails to work:
nautilus -q
If you still don't see any thumbnails (firstly make sure there are images inside the folders), click the "Clear thumbnail cache" option from the Cover Thumbnailer preferences, then restart Nautilus as stated above.
If you're not using Ubuntu, download Cover Thumbnailer .rpm and source files.