Thursday was a calm November day in North Wildwood, New Jersey, but Rob Fischer was still concerned about the beach near his home that was slowly being washed away.

"If we have another storm, definitely you can tell by the erosion, it's going to completely take away our beaches," said Fischer, who has lived full-time in North Wildwood for three years.

Fischer's concerns amplified after a project led by the Army Corps of Engineers and the New Jersey Department of Environmental Protection was canceled.

The long-planned beach and dune project was supposed to be completed between the Hereford and Cape May inlets.

North Wildwood suffered severe erosion again from recent storms in August and last month.

"We're desperately needing to have these beaches replenished," Fischer said.

In a letter sent last week to the mayors of North Wildwood, Wildwood, Wildwood Crest and Lower Township, NJDEP said in part, "After years of costly engineering work and many attempts by DEP to help resolve local disagreements, the Project reached an impasse necessitating its termination."

"One of the fundamental missions of the NJDEP is to protect the Jersey shore, and in this specific situation, the NJDEP has failed catastrophically in that responsibility," North Wildwood Mayor Patrick Rosenello said.

Rosenello said his city has now filed a lawsuit against the NJDEP for what he called a breach of contract.

The $54 million project was set to replenish all the beaches in the Wildwoods and the Diamond Beach section of Lower Township. Rosenello said the lawsuit is seeking $54 million, or it can be resolved by performing the work.

"The project that was canceled was a 50-year federal commitment to the island that the NJDEP basically let just wash away, no pun intended," he said.

In a letter last week, the Army Corps of Engineers said local disagreements led to the project's termination, saying in part, "Despite prior commitments, the borough of Wildwood Crest and City of Wildwood will not proceed with the project."

The mayors of Wildwood and Wildwood Crest tell CBS News Philadelphia they're good neighbors, but they raised concerns about the project over the amount of sand that would have been pulled from their beaches to help replenish North Wildwood's Beach.

"I didn't agree with the sort of design, quite frankly, from the beginning, but our concerns were not addressed by the NJDEP, so I don't hold anything really against Wildwood or Wildwood Crest in this regard," Rosenello said.

For now, there is no beach replenishment project on the table.

CBS News Philadelphia reached out to NJDEP and was told it does not comment on pending litigation.