Author Sarah Broom, the youngest of 12 children, is the baby of her family. But growing up, it didn’t quite feel that way. For Ms. Broom, the yellow house at 4121 Wilson avenue, her childhood home, claimed the title.

“The house, which I describe as the 13th and most unruly child, became something my mother was bent on taming,” said Ms. Broom.

On Thursday evening, Ms. Broom addressed a crowd of 558 virtual participants at the kick-off event to this summer’s Martha’s Vineyard Author Series. Moderated by Thelma Golden, director at the Studio Museum in Harlem, the event drew viewers from all corners of the country, who logged on to hear Ms. Broom discuss her memoir, The Yellow House.

The book, published last August, tells the story of Ms. Broom’s family through the lens of the sinking shotgun house in New Orleans East where she and her siblings grew up. It won the 2019 National Book Award for nonfiction.

“I was trying to understand this yellow house I grew up in,” said Ms. Broom. “Inscribed on the walls, within all the layers of renovation and reconstruction, are these people who have things to say, these histories. So that’s where I began, with the actual structure.”

The house came into the family in 1961, when Ms. Broom’s mother, Ivory Mae, purchased it entirely on her own at age 19. But as Ms. Broom told the audience on Thursday, her book begins in an era long before the family house, with her grandmother, Amelia, or “Lolo.”

“My grandmother was someone who always yearned for a home, she really bought into the narrative that home ownership is the thing that makes you an American citizen,” said Ms. Broom. Lolo’s ability to make a home anywhere she went became an important motif in the book. “This [was] her way of claiming a space for herself when she felt immensely unmoored in the world.”

Alongside the iconic yellow house, Ms. Broom’s hometown, New Orleans East, figures prominently.

“It’s not the place of most people’s imagination,” explained Ms. Broom, distinguishing the city of her childhood from the mythic New Orleans of pulsing jazz bands and bustling streets which so often stands for the city as a whole.

Instead, Ms. Broom casts light on the lesser-known part of the city—its sprawling eastern section, just across the highway. Once a booming neighborhood and a place of promise for the Broom family, over time New Orleans East was neglected and abandoned, left by the rest of the city to decay into a “land of no return.”

“It went from being a dream of a new frontier for the city of New Orleans to being essentially an embarrassment,” said Ms. Broom.

Growing up, the inaccessibility of the neighborhood posed persistent danger, requiring Ms. Broom to cross the mega-highway every day to get to school or the grocery store. In the book, the dilapidated neighborhood and ailing family house seem to echo one another. Both become a source of shame for the young author, but more broadly, they work as an entry point into Ms. Broom’s larger discussion of racial inequality and socio-economic disparity in the city.

“I marvel to remember that I actually survived,” she said. “I find it unjust and a great tragedy that there have to be survivors in this story.”

Discussions of New Orleans East also set the stage for Ms. Broom’s account of hurricane Katrina, a defining moment in her life, which wiped the beloved yellow house from its foundation, displacing her family, she said. But for Ms. Broom, Katrina marked yet another moment when Americans who hadn’t been listening finally woke up to inequality.

“So much of what the world saw after Katrina were the things we already knew,” she said, citing the history of underpaid and underserved Black communities in the city. “In order to understand what was lost and what was gained in this one moment you have to understand all these worlds before.”

Ms. Golden agreed: “African American memoir is a political act of telling these stories that are not going to be told or correcting the stories as they are often told,” she said.

During a question and answer period, viewers touched on a range of topics like home and the importance of family. One viewer cited a line in Ms. Broom’s book about the “psychic cost of defining oneself by the place where you are from,” using it interrogate what it means to belong to a place like the Vineyard.

“There is a cost when we allow something to stand in for who we are, to essentially speak for who we are. We rob ourselves of a certain amount of power,” said Ms. Broom in response at the close of the event.

As virtual applause streamed in on the website’s comment section, Ms. Broom concluded: “No place, no mythology, no sign or symbol can stand in for the human being.”

Next up for the Author Series is Erik Larson in discussion with fellow author Amor Towles on Sunday, August 2, followed by a panel of four experts on the Supreme Court on August 9. Visit mvbookfestival.com/2020-authors.