Paul Irish created and maintains a site devoted to Ampersands. The site is a collection of open source web fonts that only contain the ampersand character. He created it based off an [article from 2008](http://simplebits.com/notebook/2008/08/14/ampersands-2/) regarding the usage of ampersands and the fact that certain variants of the character in some fonts are more interesting than others. The article also points out that the ampersand character is often used only for display purposes and not in content so there is no reason not to use more interesting versions.
With Open Source Ampersands you can view all of the fonts available, choose one that you like, download it including 4 font formats and a demo.html file demonstrating how to use it. All of the fonts are 100% open source from the Google Web Fonts Directory, The League of Moveable Type, and Kernest.
You can get more info about Open Source Ampersands and how to use it here.