From left, Japan's Prime Minister Shinzo Abe, U.S. President Barack Obama, Australia's Prime Minister Tony Abbott and China's President Xi Jinping in Beijing on Nov. 11. Messrs Abe, Obama and Abbott will hold a trilateral meeting this weekend in Australia. Reuters

CANBERRA, Australia—The U.S. is expected to urge Japan and Australia to step-up military and security cooperation to help contain simmering territorial tensions in Asia, as the leaders of the three allies meet for the first time in seven years on the sidelines of summit of the Group of 20 major economies in Brisbane this weekend.

The meeting risks antagonizing Beijing, which bristles at perceptions that its rise in the region is being challenged, but also comes as the U.S. and Japan are working to repair ties with China, while Australia is looking to deepen its increasingly important economic relations with the country.

Earlier this week in Beijing, U.S. President Barack Obama and China’s President Xi Jinping in Beijing made unexpectedly significant strides in improving an often-thorny relationship, striking new climate, trade and visa agreements. Messrs. Xi and Obama also reached two new agreements designed to avert military confrontations in Asia, one on notifying each other of major activities, such as military exercises, and the other on rules of behavior for encounters at sea and in the air. 

Also this week in Beijing, Japanese Prime Minister Shinzo Abe shook hands and met briefly with Mr. Xi, in what was seen as a first step in a potential thaw in the chilly relationship between Beijing and Tokyo.

In Australia, the U.S. president will urge Mr. Abe and Prime Minister Tony Abbott, to work more closely as a stabilizing regional influence, officials said. Canberra and Tokyo have already strengthened military ties in recent times, including a possible 25 billion Australian dollar (US$22 billion) Australian purchase of Japanese submarines.

The so-called trilateral meeting will coincide with a speech by Mr. Obama on U.S. leadership in the Asia-Pacific that officials say will attempt to quell regional cynicism among other Asian nations about the effectiveness of a U.S. pivot to Asia designed to consolidate its influence on the region.

“Obama’s Brisbane speech is the opportunity to restore confidence in the rebalance,” said Rory Medcalf, an Australian security analyst at the Sydney-based Lowy Institute for International Policy.

Australia remains a close U.S. military and political ally. But China is an increasingly important economic partner for the resource-rich nation as the Chinese get richer. China’s demand for commodities helped shield Australia from a recession following the 2008 global financial crunch. Messrs. Xi and Abbott are set to sign a free-trade pact following the Group of 20 summit that Australia’s conservative government hopes will provide new drivers for growth as a long mining boom slows, opening up access for Australian banks, universities and legal firms to tap into increased demand for services as China’s middle class grows. It estimates the deal may be worth up to $A20 billion a year to the A$1.5 trillion economy. The deal highlights Australia’s difficult balancing act between its security alliance with Washington and a growing economic reliance on Beijing.

China’s president will make a speech of his own to Australia’s parliament on Monday, likely urging Mr. Abbott’s conservative government to recognize the changing political and security reality of China’s rise and—with its allies—allow Beijing more space to meet its own ambitions under Mr. Xi’s preferred model of great-power relations, security analysts say.

The last time an American leader met with his Australian and Japanese counterparts together was in 2007, when a meeting between George W. Bush, former Australian Prime Minister John Howard and Mr. Abe was kept under wraps to avoid stoking Chinese perceptions of encirclement during the Asia-Pacific Economic Cooperation summit in Sydney.

The meeting between Mr. Obama and his two counterparts this time will look at increasing trilateral cooperation exercises, maritime security measures and lowering tensions by working with other regional partners, including involving China more regularly in military exercises, officials said.

White House Special Assistant and Senior Director for Asian Affairs Evan Medeiros told journalists that the meeting would “raise up and highlight the important and growing degree of cooperation on diplomatic and security challenges in the region.”

But the significance of the meeting may be less about the substance of talks and more about the visible reaffirmation of the three-way alliance, with less determination this time to keep the meeting from Chinese attention, according to security analysts.

“It is of course, despite what they say, a direct response to what all three powers see as an increasing challenge to the existing regional order by China,” says veteran security analyst Hugh White. Meeting publicly is “a very suggestive thing for them to do,” said Mr. White, a former senior defense official now at the Australian National University.

Japanese troops are likely to take part in a large biennial U.S-Australian military exercise in Australia’s tropical Queensland state in June next year, according to Australian defense sources. That will build on existing military cooperation between Australia and the U.S., where the U.S. already maintains a rotating air force and Marine presence in northern Australia. Both Australia and Japan are seeking to build amphibious forces and the last exercise in 2013 involved 30,000 soldiers and Marines.