At a crowded public hearing Wednesday, the Chilmark zoning board of appeals heard the first public arguments in a heated dispute between neighbors over a large-house compound that is nearing completion on Nashaquitsa Pond.

The compound being built by Adam and Elizabeth Zoia on the former Harrison property has sparked widespread discussion throughout the town this winter and is now the subject of an appeal filed by Jill and Kenneth Iscol, who are abutters to the property. The Iscols have formally requested that the zoning board enforce and correct what they claim are zoning violations on the property. The Zoias say they have violated no town bylaws. Both property owners have hired attorneys to represent them.

The board of appeals, which continued the hearing after more than an hour of debate on Wednesday to allow time to consult with counsel, has made no ruling yet.

Many town residents crowded the hearing room in the Chilmark town hall, and board of appeals chairman Chris Murphy worked repeatedly to keep the debate on topic.

“This is not a hearing on big houses; this is a specific hearing on a complaint by a person who feels their neighbor is doing something that requires more oversight by the town and is not getting it,” Mr. Murphy said. “That neighbor has issued a complaint and that’s what we’re here to deal with.”

Filed in April through their attorney, the Iscols’ appeal claimed that an accessory structure on the property needs a special permit and was built without one, that a retaining wall on the property violated the setback rules and exceeded height limits, that the placement of a generator on the property constituted a violation and that outdoor lighting on the property meets the definition of light trespass.

On Wednesday Diane Tillotson, an attorney with Hemenway & Barnes in Boston representing the Iscols, told the board of appeals that the wall and generator issues had since been resolved. She said the accessory structure still needs a special permit under town rules and she requested that the Iscols be allowed to retain an independent land surveyor to confirm the measurements of the buildings on the Zoia property.

Questions surfaced about whether the Iscol appeal could even be considered since it was filed more than 30 days after a building permit was issued to the Zoias (the building permit was issued in January 2011; the appeal was filed in April of this year). In a July 24 letter to the board, Kevin O’Flaherty, an attorney with Goulston and Storrs in Boston who represents the Zoias, urged the board to dismiss the appeal “on this basis alone.”

The total square footage of the buildings on the Zoia property is in dispute; the buildings include the main house, the accessory structure and a converted barn which is being used as an exercise space and garage. The Iscols claim the three structures add up to nearly 20,000 square feet. A certified survey of the building on the property has not been completed. The main house on the property is about 8,200 square feet.

There was discussion about the definition of an accessory structure. Under the town zoning bylaw, accessory structures are defined as tool sheds or pump houses. All other structures are only allowed by special permit, including guest houses, which are restricted to 800 square feet, and home offices, which are restricted to 1,500 square feet, swimming pools, tennis courts and converted barns.

According to the Zoias’ attorney, the accessory structure is 2,320 square feet, and houses an indoor lap pool, steam room, bathroom, mini-laundry and sauna on the first floor and a bedroom, bath and sitting area on the second floor.

Ms. Tillotson said the structure qualifies as a guest house

“Our belief is that this structure is not permitted at all or that it requires a special permit as a guest house and a pool house,” she said. “I think it is a guest house or something that is not permitted by your bylaw. It is not an accessory. It is large, it exceeds the limitation in terms of square footage that is specifically called out as an accessory structure,” she said.

The zoning board took another view. After a site visit earlier this week, board member Frank LaRusso said the structure is not a guest house because it has no kitchen.

“There are numerous buildings in town that are not just a tool shed and considered accessory structures,” Mr. LaRusso said. “As far as detached bedrooms, we have dozens and dozens of them all over the place. I think the problem here is with our bylaw and not with our building inspector’s interpretation.”

Ms. Tillotson disagreed, saying there are “cumulative issues here.”

“Your detached bedroom here exceeds the square footage limitation for an entire guest house under your zoning bylaw,” she said. “I’m asking you to apply a common sense approach . . . in interpreting your bylaw as a whole because I don’t think what’s intended by your bylaw was a substantial single family residence, a barn that has no agriculture connection but has a gym and a garage, and a pool as a detached bedroom . . . all on the same single family lot. If you go the special permit route there are things you could do to make it more palatable for abutters.”

Chilmark building inspector Leonard Jason Jr., who issued the building permit, said he believed the building qualified as an accessory structure under zoning bylaws; he said the board of health uses the term detached bedroom for septic reasons. Mr. Jason said there are many examples in town of detached bedrooms and accessory structures.

“That’s where the chauffeurs and the maids and the cooks used to live, they slept there, didn’t cook, ate from the [main] kitchen and had a place they could call their own,” he said. “It’s historically been a detached bedroom, it’s not a guesthouse, there is no kitchen.”

Mr. Jason said what goes inside the structure is beyond his purview.

“State law says you’re not supposed to be involved in the interior of dwellings of single family homes,” he said. “We have them throughout town without special permits.”

There were some tense moments, including when Thomas Bena, a Chilmark resident who is filming a documentary about the large house issue, and Jim Belushi, a Chilmark resident and friend of the Zoias, exchanged heated words. Off-duty special police officer Bret Stearns stood nearby to keep the peace.

Elizabeth Zoia defended her family building project and called for an end to the divisiveness.

“As far as we are concerned we were issued the permits and built according to our permits,” Mrs. Zoia said. “We keep hearing excessive, over-the-top venomous treatment that my family has received over the past year. It is not acceptable. I have young children who are afraid to live in their house . . . there are certain laws that need to be followed and we feel we have followed them. We want to be left to live in peace and not be harassed.”

Mrs. Iscol said the problems between the Zoias and her family, previously “very good friends,” involve broken trust.

“We kept on asking them questions . . . and we accepted it as truth,” she said. “Last summer when we saw the wall hiding the generator . . . it was a trigger point, which led us to begin this process of wanting to make sure that there weren’t any other potential violations.”

David Heilbroner, a Chilmark resident and friend of the Harrisons, who sold the property to the Zoias, apologized for the “atmosphere of a witch-hunt and demonizing.” And he spoke of change.

“What we’ve learned is you’ve reached a tipping point,” Mr. Heilbroner said. “I have a 1,700-square-foot house, it was considered a big house [when I moved here 50 years ago] and it’s now considered a small house. As things change we have to keep up with it. I sense there’s been a desire to do a wholesale review of the bylaws and how discretion is applied . . . I think there’s a sense in which discretion can create holes that are too big to drive through.”

No date has been set yet for the continued hearing.