A federal judge has rejected the settlement of a lawsuit stemming from the New York Police Department ’s surveillance of Muslims, saying the proposed deal does not provide enough oversight of an agency that he said had shown a “systemic inclination” to ignore rules protecting free speech and religion.

In January, Mayor Bill de Blasio, a Democrat, agreed to appoint a civilian lawyer to monitor the department’s counterterrorism activities as a means of settling two lawsuits accusing the city of violating the rights of Muslims over the past decade.

But the judge, Charles S. Haight Jr., in an opinion published on Monday, said the settlement did not go far enough for an agency that had become “accustomed to disregarding” court orders.

“The proposed role and powers of the civilian representative,” Judge Haight wrote, “do not furnish sufficient protection from potential violations of the constitutional rights of those law-abiding Muslims and believers in Islam who live, move and have their being in this city.”

The decision means lawyers for both sides will have to negotiate changes to the settlement or fight the lawsuit in court. Jethro Eisenstein, a civil rights lawyer in the case, said he and his colleagues planned to discuss the ruling with city lawyers.

A spokesman for the city’s Law Department, Nick Paolucci, said: “To the extent that the court’s decision is based in part on an inspector general’s report containing findings with which both the city and class plaintiffs’ counsel variously disagree, we are disappointed that the settlement was not approved as the parties originally proposed. That said, we will explore ways to address the concerns raised by the judge.”

It was Judge Haight who had first acceded to the city’s requests for relaxed rules after the Sept. 11, 2001, terrorist attacks. The police commissioner at the time, Raymond W. Kelly, said civilian oversight and traditional restrictions on policing made the city less safe, and the judge agreed, saying the old rules “addressed different perils in a different time.” He eliminated civilian oversight and gave wide authority to the commissioner and his intelligence deputy.

By rejecting the deal, Judge Haight, of United States District Court in Manhattan, made a tacit acknowledgment that he had gone too far in that ruling.

It was the authority he granted after Sept. 11, along with help from the Central Intelligence Agency , that allowed the city to turn its eyes on Muslim neighborhoods. Plainclothes officers lingered in coffee shops and halal delis, eavesdropping on conversations and noting the political opinions of owners and customers. They used informers to tape sermons and collected attendance lists at Muslim student groups. Detectives kept files on Muslims who changed their names and conducted long investigations into mosques, cases that put whole congregations under scrutiny for years but never led to charges against a mosque.

In this new ruling, the judge placed surveillance of Muslims in the context of a century of broad police surveillance used against Italians, African-Americans and suspected Communists. “It is a historical fact that as the decades passed, one group or another came to be targeted by police surveillance activity,” he wrote.

The judge also cited an August report by the Police Department’s inspector general, which found that investigators had failed to comply with the court-ordered guidelines. It said investigators kept cases open too long and used boilerplate language to justify the use of confidential informers.

When it was released, officials with the department characterized the report as a vindication, saying it proved they had not investigated anyone unlawfully. They described the problems as minor administrative errors and cast it as a clean bill of health.

But Judge Haight said it was evidence of “near-systemic failure.” He chastised the department for highlighting favorable aspects of the report but not addressing repeated problems.

Judge Haight’s involvement in police oversight dates back nearly half a century. A 1971 class-action lawsuit forced the end of the city’s so-called Red Squads and established the intelligence-gathering rules, known as the Handschu Guidelines, named after one of the plaintiffs. The lawsuit has remained active for decades, serving as a check against overreaching by the police. Judge Haight approved the original rules, relaxed them after Sept. 11, and has now called for them to be tightened.

The settlement he rejected would have placed a lawyer inside the Police Department to review intelligence files and report potential wrongdoing to the police commissioner, the mayor or the judge. Judge Haight said that was not enough. He suggested that the lawyer file confidential quarterly reports with the court. He also opposed a provision that would allow the mayor to eliminate the monitor position after five years.

“One cannot foretell what will happen over time in this sensitive and volatile part of life,” the judge wrote, “and it is questionable whether it is fair or reasonable to give the mayor this unfettered veto power.”

In 2011, The Associated Press published a series of articles and internal documents that revealed the workings of the department’s Intelligence Division. Those documents included a list of “Ancestries of Interest” and maps of locations with notations like “popular meeting location for political activities.”

Judge Haight did not take issue with some aspects of the proposed settlement, including requirements that the department use undercover officers only when other options are impractical and consider the effect that its investigations have on religious groups and people who are not suspected of any wrongdoing. The department also agreed to cap the length of investigations in most cases. City officials have said those policies are already in place.

In meetings this year, Judge Haight heard from community members about the proposed deal. Many in attendance supported the agreement, including Barbara Handschu, for whom the original lawsuit is named. She believed the proposed settlement would offer “far better protections against police surveillance.”

Others, including Ravi Ragbir, the executive director of the New Sanctuary Coalition in New York, expressed concern that the civilian monitor’s authority lacked “teeth.” He asked the judge to “push even harder so the guidelines will be much more stronger, giving the community something to use so that if there is violation there are ways that they can address those problems.”

A settlement, if one can be reached, would result in new policing rules. It would also resolve a separate lawsuit by the American Civil Liberties Union over the surveillance programs.

Ms. Handschu had previously told the judge that she accepted the settlement reluctantly, that it did not go “far enough” and that “there are continuing threats in general to the Muslim community which concern me.”

Correction: October 31, 2016

An earlier version of this article misstated who had previously told the judge that she accepted the settlement reluctantly, that it did not go “far enough” and that “there are continuing threats in general to the Muslim community which concern me.” It was Barbara Handschu, not Hina Shamsi.

A version of this article appears in print on November 1, 2016, on page A1 of the New York edition with the headline: Judge Halts Deal on Surveillance by Police Dept. Order Reprints | Today’s Paper | Subscribe Continue reading the main story