Shearer Cottage has been welcoming guests to its secluded Oak Bluffs location for more than 100 seasons. The lodging house is still owned and run by descendants of Charles Shearer, a former Virginia slave who with his wife Henrietta, migrated to Massachusetts in the 1880s and began summering in Oak Bluffs.

Built in 1912 to host travelers of color who were not accepted at the Island’s established hotels, the couple’s Shearer Cottage became the first landmark on the Martha’s Vineyard African American History Trail in 1997. At the Smithsonian’s National Museum of African American History and Culture, on the National Mall in Washington, D.C., a permanent exhibition called Power of Place includes furniture, a sign and a guest register from Shearer Cottage.

For the past 40 years, the rambling, red-sided building in the East Chop highlands also has become the annual destination for a group of outdoorsmen from Vermont. Over the decades, these guests have forged long-lasting relationships with Charles and Henrietta Shearer’s granddaughter, the late Doris Jackson Pope, and her daughter Lee Jackson Van Allen, who now runs the inn with sister in law Emily Thomas.

“We used to all gather in the kitchen in preparation for their visit — my mother Doris and my Aunt Liz and my Aunt Miriam and myself,” fixing cold food platters for the guests to share when they arrived, Ms. Van Allen said.

“We always referred to them as ‘the fishermen,’” Ms. Van Allen recalled of those early years, when fishing the derby was the group’s chief occupation. Nowadays, she calls them “the doctors,” because they’re also psychiatrists, all affiliated with the University of Vermont Medical Center in Burlington. Also, they no longer fish the derby.

“Originally, we did come to fish,” said Ragon Willmuth, who has been making the trip each year since 1979. “We used to get up at four in the morning and hit the beach. We’re just not doing that any more,” he added with a wry smile.

“We did our fishing at the Net Result this year,” said Mark Schultz, who while not a charter member has been making the trip for 37 of the 40 years.

In all, nine Vermont psychiatrists checked in late last month for a long weekend of bicycling, hiking, eating at Island restaurants and relaxing at the cottage. Ms. Van Allen and Ms. Thomas greeted them there with gifts of food, a banner commemorating their 40th anniversary and a custom Shearer Cottage ball cap.

Nine is a sizable party, but the group has been much larger. As their sons grew old enough to fish, up to 18 men and boys made the pilgrimage to Shearer Cottage for more than a decade. One year, they also rented a house next door to hold the overflow, Dr. Schultz said.

“We used to have the Bluefish Bowl, our annual football game,” in nearby Washington Park, recalled Richard Bernstein, another longtime member of the group.

The football games, like the fishing, are memories today — but other traditions survive. The friends always dine at Jimmy Seas Pan Pasta, the Ocean View and Sharky’s, they like to visit Larsen’s Fish Market in Menemsha and they play low-stakes poker in the old Shearer Cottage dining room, now a comfortable common room.

“We started at nickel-dime-quarter 40 years ago, and we now play for nickel-dime-quarter,” Dr. Willmuth said.

In the early 1980s, Ms. Van Allen said, the group prepared communal meals on the old eight-burner stainless steel stove in the Shearer kitchen. That appliance is now disconnected, so when the doctors cook in, they use one of the kitchenettes in their guest rooms.

“The main thing is, we’ve always felt very welcome,” said John King, who was part of the very first trip to Shearer Cottage in 1979. “Everybody’s been very good to us. We’ve been treated to 40 years of great hospitality.”

The Vermonters’ Vineyard trip is a tradition, not a reunion: These men see plenty of each other outside of work. “We are a group that has done many things together for many years,” Dr. Bernstein said. “We’ve had several softball teams, we’ve gone hiking and canoeing in the Adirondacks, we have a monthly poker game and a book club.”

But only on the Vineyard can they feel they’re part of an inclusive community like Oak Bluffs.

“We live in a basically all-white state,” Dr. Bernstein said. “What we enjoy about coming here is the opportunity to feel welcome, embraced and part of a history I am increasingly aware of, did not know about — feel awful about, just in national terms.

“There is something healing about it, in a very subtle way,” he continued, as several of the other doctors nodded in agreement. “It’s not mentioned, but for myself personally, it just feels good.”

“We’re very lucky guys to have found this place,” Dr. Schultz said.

The group also takes pride in Shearer’s place at the national museum. “We think that’s pretty neat,” Dr. King said.

“We feel a little bit a part of the history here,” said David Hedden.

For her part, Ms. Van Allen looks forward to the doctors’ visit every year. “It is special because they do remember the previous generation that ran Shearer Cottage,” she said.

“It’s special for me because they appreciate the work it takes. Every generation has had to work hard,” she added. “Really, Shearer Cottage is a labor of love — there’s not a great amount of money that is made.”

What makes it worthwhile, Ms. Van Allen said, is the “love and appreciation you receive from the guests.”

Ms. Van Allen and Shearer Cottage received an extra token of appreciation from the doctors after they returned to Vermont this year: a group donation to the African American Heritage Trail.

“I think that was very generous and thoughtful of them,” Ms. Van Allen said.