A cornerstone of the Vineyard economy for many decades, the Goodale Construction Company has long mined for sand and gravel without restrictions inside its 100.2 acres in Oak Bluffs.

But what was once an isolated, sparsely populated region with little more than a sand pit in the middle is today a patchwork of residential homes, farm and conservation land.

And amid growing complaints from neighbors, the town is now asking Goodale’s to obtain a special permit for its operation.

In a letter to owner Jeremy T. Goodale two months ago, building inspector James E. Dunn ordered the company to apply for a permit from the zoning board of appeals in the next 60 days. He wrote that the part of the Goodale property dedicated to earth removal “has been substantially extended since the use became nonconforming.”

Goodale's has appealed building inspector letter; a hearing is set for June 19 before the Oak Bluffs zoning board of appeals. — Ray Ewing

“I am exercising my discretion not to order you to cease and desist from continuing to operate at this time, but will be compelled to proceed with other remedies should you not comply with this directive,” Mr. Dunn wrote in part.

Attorneys for the Goodales have appealed the building inspector’s order, claiming a grandfathered right to continue their work on the property without a special permit. In the appeal, the Goodales dispute the claim that the use of other portions of their property qualifies as an expansion.

The Oak Bluffs zoning board of appeals will hear the matter at a public hearing on June 19.

The property, which is located in a residential district, was in use as a gravel and sand mining corporation before zoning laws were enacted in town. After exhausting mining possibilities on the southern and eastern limits of the property, Kevin O’Flaherty, an attorney for the Goodales, said this week that mining has been moved to a smaller area in the western part of the property. “The fact that the hole gets bigger doesn’t equal an expansion,” Mr. O’Flaherty said.

Mr. Dunn acknowledged that the issue of permitting never came up until a few years ago.

“No one has aggressively complained before, so it was just let go and assumed it was a grandfathered right,” the building inspector said.

That changed about three years ago when the owners of the gravel pit rerouted neighborhood traffic to a new road and began clearing more trees near the residential area. Neighbors began to complain. In 2011, in response to the concerns, the Oak Bluffs selectmen voted to refer the issue to the Martha’s Vineyard Commission for consideration as a development of regional impact.

Ultimately the commission decided to not review the project and sent it back to the town.

Meanwhile, the Goodale property has also come under monitoring by the state Natural Heritage & Endangered Species Program. In 2003 Natural Heritage declared that the then-forested portion of the property was priority habitat for five rare moth species. In 2012, when the company planned to clear two more acres of trees, the owners were directed to restore two and a half acres of habitat to make up for the loss. When the agency visited the property for a follow-up visit, they observed that even those scrub oak planted on an incline along the side slopes of the pit appeared to be taking root. Scrub oak thrives in regions known as frost pockets, which are naturally lower in elevation. The same is probably true inside the Goodale pit, said Jon Regosin, chief of conservation science at the Natural Heritage.

Nearby residents have complained of impacts on neighborhood. Goodale's was there before zoning and before residential homes were built in the area. — Ray Ewing

He said he was impressed by the success of the reforestation project there.

“They actually did a good job on the transplants,” he said. “This is really a potentially compatible activity, if you can take out the sand and gravel and restore the habitat as you go.” The entire pit could potentially be returned to scrub oak forest in the future, if the Goodales wish to do so, Mr. Regosin said.

But there is no obligation on the part of the Goodales, since the use of much of the pit predates the priority habitat designation.

Tim Simmons, a restoration ecologist with the Natural Heritage, said he is due for a follow-up visit this month.

Mr. Regosin said any further destruction of habitat would trigger additional review by Natural Heritage.

“Anything beyond the two acres that we have already worked out, they would need to inform us and we would have to work out what the next step is,” he said.

Meanwhile, neighbors of the Little Pond neighborhood to the west of the Goodale property have requested that the town enforce zoning laws by requiring the Goodales to obtain a special permit for their sand and gravel operation.

The residents claim that the Goodales closed off a former neighborhood access road and replaced it with a new road that hugs the southern edge of the pit, putting up a black chain-link fence where there had once been a wooden fence. They say the new access road was poorly engineered and that the fence allows sand and snow to blow through to the road.

“When I first moved here, I used to walk in the neighborhood with my friends and our babies,” said Melissa Harding, a Little Pond resident. “Then suddenly I couldn’t get to the bike path [on Barnes Road] because we couldn’t walk through the sand with our children.”

The primary concern among neighbors is the movement of the pit operations westward, toward their secluded, wooded neighborhood.

Until recently, a deep buffer of trees separated the sand and gravel operation from Little Pond homes. But the company has since removed many trees and is showing signs of further expansion, residents said this week.

“I am continuing to see expansion all the time, they are cutting trees, they are continuing to clear,” said Billie Burke, a Little Pond Road resident.

Mr. O’Flaherty, the attorney representing the Goodales, noted that the neighborhood was developed many years after the sand and gravel operation had begun.

“The folks came after, so there is this notion in the law of coming to a nuisance,” he said.

He also noted the importance of the operation to the construction industry on the Island. “Hopefully for the Island it continues for many years,” he said.

Neighbors acknowledge the sensitive nature of the issue. “It is like the elephant in the room, because they represent the construction industry on the Island,” said Daryl Alexander, a Little Pond resident.

“It’s the sort of thing that a lot of people think oh, it’s not-in-my-backyard, but if you thought it was simply that, then you are sort of disassociating yourself from the larger picture of the Vineyard — and what happens to us happens to everyone here,” she added.

The residents also say they don’t want Goodale’s to shut down; they only want the town to apply zoning rules to the operation.

“I think they should continue to make money but not at our expense,” Ms. Alexander said.

She said bad blood has formed between the company and its residential neighbors. “Words were said that people can’t get out of their ears at this point,” she said.

But she said ideally, the town will work out a way to address the needs of all concerned.

“It is a challenge to see if our town can act in an expeditious fashion, and do something that is right and doesn’t drastically hurt Goodale’s,” she said. “They should not shut down.”