The Warren House is one of several regal houses on North Water street, known for well-kept former whaling captains' homes. The house, long in a state of disrepair, dates back to the late 18th century and was the home of the Osborns, an old Edgartown family with deep roots in the whaling industry. Caroline Osborn Warren inherited the home and was a benefactor of the Edgartown Public Library next door. The house has long been called the Captain Warren house, though Frederick Warren was actually an accountant.

In the 20th century the house went from a private residence to an inn. In 2004 the town of Edgartown bought the building for $3.5 million with the idea of expanding the library next door. That plan fell through and the town sold the house for $2.5 million in late 2013. The buyer was Jeffrey Wolk, who plans to turn the mansion into a home for his family. Renovations began on that project this spring.