BRUSSELS — The warships were escorting a vessel suspected of smuggling weapons into Libya, violating a United Nations arms embargo. Challenged by a French naval frigate, the warships went to battle alert. Outnumbered and outgunned, the French frigate withdrew.

But this mid-June naval showdown in the Mediterranean was not a confrontation of enemies. The antagonists were France and Turkey, fellow members of NATO, sworn to protect one another.

A similarly hostile encounter between Turkey and a fellow NATO member happened just two weeks ago, when Turkish warplanes buzzed an area near the Greek island of Rhodes after Greek warships went on alert over Turkey’s intent to drill for undersea natural gas there.

Turkey — increasingly assertive, ambitious and authoritarian — has become “the elephant in the room” for NATO, European diplomats say. But it is a matter, they say, that few want to discuss.

A NATO member since 1952, Turkey is too big, powerful and strategically important — it is the crossroads of Europe and Asia — to allow an open confrontation, alliance officials suggest.

Turkey has dismissed any criticism of its behavior as unjustified. But some NATO ambassadors believe that Turkey now represents an open challenge to the group’s democratic values and its collective defense.

A more aggressive, nationalist and religious Turkey is increasingly at odds with its Western allies over Libya, Syria, Iraq, Russia and the energy resources of the eastern Mediterranean. Turkey’s tilt toward strongman rule after 17 years with President Recep Tayyip Erdogan at the helm also has unsettled other NATO members.

“It’s getting hard to describe Turkey as an ally of the U.S.,” said Philip H. Gordon, a foreign policy adviser and former assistant secretary of state who dealt with Turkey during the Obama administration.

Despite that, Turkey is getting a kind of free pass, analysts say, its path having been cleared by a lack of consistent U.S. leadership, exacerbated by President Trump’s contempt for NATO and his clear admiration for Mr. Erdogan.

“You can’t say what U.S. policy on Turkey is, and you can’t even see where Trump is,” Mr. Gordon said. “It’s a big dilemma for U.S. policy, where we seem to disagree strategically on nearly every issue.”

Those strategic divides are proliferating. They include Turkey’s support for different armed groups in Syria; its 2019 purchase of a sophisticated Russian antiaircraft system over fierce objections by the United States and other NATO members; its violation of the arms embargo in Libya; its aggressive drilling in the eastern Mediterranean; its constant demonization of Israel; and its increasing use of state-sponsored disinformation.

But NATO officials’ general meekness in standing up to Turkey has not helped, analysts say, pointing to the group’s secretary-general, Jens Stoltenberg, whose job is to keep the 30-nation alliance together, but who is considered excessively tolerant of both American and Turkish misbehavior.

The last serious discussion of Turkey’s policies among NATO ambassadors was late last year, despite the purchase of the antiaircraft system, the S-400.

Other countries, like Hungary and Poland, also fall short on the values scale, argued Nicholas Burns, a former NATO ambassador now at Harvard. But only Turkey blocks key alliance business.

NATO operates by consensus, so Turkish objections can stall nearly any policy, and its diplomats are both diligent and knowledgeable, “on top of every ball,” as one NATO official said. France has also used its effective veto to pursue national interests, but never to undermine collective defense, NATO ambassadors say. But Turkey has blocked NATO partnerships for countries it dislikes, like Israel, Armenia, Egypt and the United Arab Emirates.

More seriously, for many months Turkey blocked a NATO plan for the defense of Poland and the Baltic nations, which all border Russia. And Turkey wanted NATO to list various armed Kurdish groups, which have fought for their independence, as terrorist groups — something that NATO does not do.

Some of these same Kurdish groups are also Washington’s best allies in its fight against Islamic State and Al Qaeda in Syria and Iraq.

A deal was supposedly worked out at the last NATO summit meeting in December in London, but Turkey created bureaucratic complications, and it was only in late June that Turkey relented — after considerable pressure from official Washington, which has lost patience with Mr. Erdogan and is infuriated by his insistence on buying the S-400.

If deployed, the S-400 would put Russian engineers inside a NATO air defense system, giving them valuable insights into the alliance’s strengths while threatening to diminish the capability of the expensive fifth-generation fighter, the F-35.

The assumption is that Mr. Erdogan, who has grown significantly more suspicious since a failed 2016 coup against him, wants to be able to shoot down American and Israeli planes like the ones his own air force used in the coup attempt.

“Every time we discuss Russia” in NATO, “everyone thinks of the S-400 and no one says anything,” said one European diplomat, who spoke on the condition of anonymity to discuss a sensitive matter. “It’s a major breach in NATO air defense, and it’s not even discussed.”

Instead, NATO assumes that talks between Washington and Ankara will somehow handle the problem. But Washington is divided, and Mr. Erdogan talks only to Mr. Trump.

Yet the confusion is not simply Washington’s, said Amanda Sloat, a former deputy assistant secretary of state who dealt with Turkey in the Obama State Department and wrote a recent essay with Mr. Gordon. The European Union and the United Nations also have no clear policy on Turkey or Libya, she said.

Turkey has pursued its own national interests in northern Syria, where it now has more than 10,000 troops, and in Libya, where its military support for a failing government helped turn the tide in return for a share in Libya’s rich energy resources.

It was near Libya in June that three Turkish warships confronted the French frigate.

While the European Union has a mission to help enforce the arms embargo on Libya, NATO does not. The frigate, the Courbet, was engaged in a different NATO mission aimed at migration flows, but since Turkey and France support different sides in the Libyan civil war, the confrontation between NATO allies was troubling.

Turkey said the ship was carrying aid rather than arms, and has denied harassing the Courbet. NATO officials say that its military committee is investigating and that the evidence is not as clear-cut as the French suggest.

Still, President Emmanuel Macron of France has used the clash as another moment to assert that NATO is nearing “brain death,” because it seems incapable of reining in Turkey or acting in a coordinated political way.

His first accusation also involved Turkey, when Mr. Trump, after a call with Mr. Erdogan last October, unilaterally decided to pull U.S. troops out of northern Syria, where NATO is fighting the Islamic State, leaving the French and other allies exposed. Ultimately, the Pentagon persuaded Mr. Trump to leave some American troops there.

French-Turkish tensions at NATO date to the 2011 decision to intervene against Col. Muammar el-Qaddafi in Libya, noted Ivo Daalder, who was then the American ambassador to NATO.

France, with its policy of secularism, fears that Mr. Erdogan’s reinsertion of Islam into politics will spread in North Africa, encourage Islamist militias and damage France’s “sphere of influence,” said Soner Cagaptay, the director of the Turkish Research Program at the Washington Institute for Near East Policy. “They are quite worried.”

The latest flash point is over Turkey’s demand to share in discoveries of natural gas made in 2015 in the eastern Mediterranean, which led to deals and alliances among Greece, Cyprus, Israel and Egypt.

Maritime claims are disputed, and Mr. Erdogan complained in June that “their aim was to imprison our country, which has the longest coastline in the Mediterranean, into a coastal strip from which you can only catch fish with a rod.”

He then sent survey and drilling ships to explore off Cyprus, prompting European sanctions, and said he would do the same near Rhodes, bringing the Greeks to threaten warfare. Last week, Chancellor Angela Merkel of Germany got Mr. Erdogan to hold off while talks proceed.

While many looked to Turkey as a moderate democratic model during the Arab spring a decade ago, Turkey is a different country under Mr. Erdogan, who has mobilized the more religious voters in the countryside.

A devout Muslim, Mr. Erdogan has become more nationalist and authoritarian, especially in the aftermath of the 2016 coup attempt, when he purged and jailed many Turkish secularists, judges, journalists and military commanders.

He has broken definitively with Turkish secularism, symbolized by his recent decision to turn Hagia Sophia from a museum back into a mosque. He has pushed hard into the region with a neo-Ottoman ambition, downgrading older alliances to press Turkish interests.

For NATO, Ms. Sloat said, “The question now is whether Turkey is still a Western country and shares our values.”

Ibrahim Kalin, Mr. Erdogan’s spokesman, brushes off criticism and says Mr. Trump and Mr. Macron are the ones questioning NATO’s value.

“I guess Macron is trying to assert some sort of leadership in North Africa, the kind he doesn’t have in Europe,” Mr. Kalin said. “He called Turkey criminal, and it is incredible for France to call that to another NATO member.’’

As for Brussels, Mr. Kalin said, “the E.U. should look into the mirror.” Greece “uses E.U. membership as a way to pressure Turkey, but this language of sanctions will not work,” he said, arguing that Turkey wants only “an equitable and fair sharing of energy resources.”

The public American position is essentially to urge Turkey “to halt operations that raise tensions,” Philip T. Reeker, the acting U.S. assistant secretary of state for Europe and Eurasia, said of the eastern Mediterranean.

“We want our friends and allies — and let’s remember, we’re all, Turkey, Greece and the United States, all NATO allies — we want friends and allies in the region to approach these issues in a spirit of cooperation,” he said.

“There is a big conversation to have about what to do about Turkey,” a senior European diplomat said. “But it’s not for now.”

Carlotta Gall contributed reporting from Istanbul, and Matina Stevis-Gridneff from Brussels.