Vacationing on Martha’s Vineyard is a Russo fam ily tradition. “I’m trying to remember the first time we took my daughters to the Vineyard, but I know they’ve been coming every year since they were 10 or 11, maybe even earlier,” said novelist Richard Russo, who in 2002 won the Pulitzer Prize for fiction for his book Empire Falls.

“Now they’re in their early thirties. Their connection to the Vineyard has been [in place for] a while.”

But for Mr. Russo, the vacation tradition started over 50 years ago, when he was 10.

“It was where my mother chose to take me for the first time. It had that kind of magical quality to it. It symbolized all things that were possible.”

His early Vineyard vacation is detailed in Elsewhere, Mr. Russo’s memoir and first nonfiction book. In that book the author writes acutely about his life growing up in Gloversville, an industrial upstate New York town where he lived with his mother, Jean, and his grandparents. “My most positive memories of Gloversville were of my mother and father and grandparents and cousins who lived around the corner, and my aunts and uncles and neighbors,” he said. But memories of places, Mr. Russo added, are often intertwined with people.

“We tend to confuse places with people. And I have very positive memories of Gloversville linked to particular things, activities and people I loved. I have very fond memories of walking downtown with my grandfather and stopping at the drugstore and the soda fountain and stopping to chat along the way and be invited on people’s porches. And I have a very vivid memory of a community that was there when I was a boy,” he said.

“You almost have to go away and lose some of those people to realize this. I mean, I went off to college and got married and it didn’t take me long to realize this. My grandfather died the same week my wife and I got married. I had to go forward with life without him. We all go through this, right? You have to realize what it is that you’ve lost.”

While his tales of Gloversville are rich with details of the mill town, Elsewhere is more about his mother and the world she desired for both herself and her son.

From his early boyhood years to his mother’s death decades later, Elsewhere chronicles Mr. Russo’s unique relationship with his mother. He writes that his mother “presented herself as a woman seeking a mate more than husband, as a Nora Charles searching for her Nick, except instead of having a yippy little dog for a companion, she had me.”

“I wanted this book to be kind of a love story,” Mr. Russo said. “It was a book I had to write but not necessarily one I had to publish. And it was hard to write because it was revealing a secret my mother kept her entire life.”

The secret was a condition his mother suffered from. Throughout the memoir the “condition” is described as “nerves.” Later, after his mother’s death, Mr. Russo identifies it as obsessive-compulsive disorder.

“All families have secrets,” Mr. Russo said. “Almost anyone who grows up in a family realizes that their lives have been shaped by this family in ways they don’t completely understand. And I hoped that the book may make them feel less lonely or help them understand their own families better.”

“It was my daughters who convinced me I needed to publish this book. They were hopeful that publishing this memoir would give other people comfort.”

Mr. Russo continues to honor his mother’s gift of taking him to the Vineyard by visiting with his family once a year to unwind.

“We kind of recharge our batteries. I try to make sure if I’m working on a novel I’m not on a deadline. So when we come there for a couple of weeks my wife and I will bring a dozen books apiece with us. We may not read all of them but we like to have lots of material.”

While on vacation, he spends rainy days at Bunch of Grapes or Edgartown Books with his children, and during each visit they’re always sure to make at least one trip up to Menemsha for fresh fish.

“I’ve passed all of that, my early fondness for the Vineyard, along to my wife and my daughters. And my daughter Emily has given us two grandchildren and they’ll meet us on the Vineyard this September. They’ll find something magical about the place because it’s magical to their mother.”

Richard Russo will speak at 12:40 p.m. on Saturday, August 3, at the Harbor View Hotel and at noon on Sunday, August 4, on the grounds of the Chilmark Community Center.