Mr Mattis said Beijing was intimidating and coercing others in the region by putting weapons systems on manmade islands.
US President Donald Trump, backing the remarks by his Pentagon chief at an international security forum in Singapore, said on Twitter: “Very surprised that China would be doing this?”
Mr Mattis said the Trump administration’s recent decision to disinvite China from a multinational naval exercise was an “initial response” to Beijing’s island activity.
He called the US action a “relatively small consequence”.
“I believe there are much larger consequences in the future,” he said.
China’s reliance on military muscle to achieve its goals “is not a way to make long-term collaboration the rule of the road in a region that’s important to China’s future,” Mr Mattis said, when asked to elaborate.
“There are consequences that will continue to come home to roost, so to speak, with China, if they don’t find a way to work more collaboratively with all of the nations who have interests,” he said.
The bluster comes at a delicate time in US-China relations, as the Trump administration is warily asking for China’s help with North Korea ahead of a June 12 summit, while also threatening a trade war.
Mr Trump renewed his threat to hike tariffs on Chinese goods, just days before his commerce secretary arrived in Beijing on Saturday for trade talks.
Mr Mattis said there was little doubt about Beijing’s intentions.
“Despite China’s claims to the contrary, the placement of these weapons systems is tied directly to military use for the purposes of intimidation and coercion,” Mr Mattis said at the annual Shangri-La Dialogue.
China recently has deployed anti-ship missiles, surface-to-air missiles, electronic jammers and other equipment on the Spratly Islands, and landed a bomber aircraft at Woody Island.
Mr Mattis also touched on Taiwan, a longstanding dispute between the US and China.
He said Washington would continue to provide defence equipment and services to Taiwan and oppose any effort to alter the status quo.
The Chinese government and the Chinese people resolutely oppose US arms sales to Taiwan as well as any official contact with Taiwan and upgrading ties with Taiwan in any way, said He Lei, vice president of the Academy of Military Science of the Chinese People’s Liberation Army.
“The Chinese people and the Chinese government will never allow any person, any political party and any organisation under any name to separate any piece of Chinese territory from China at any time and in any form,” Mr He said.
“The Chinese People’s Liberation Army has the determination, confidence and ability to safeguard China’s sovereignty, security, unity and development interest.”
China claims the self-governing island as its own territory to be brought under its control by force, if necessary.
But Mr Mattis also said the US welcomed cooperation with China “wherever possible,” and announced that he had accepted Beijing’s invitation to visit soon.
It was not immediately clear whether that invitation would stand after the conference.
Mr Mattis’s comments drew a pointed reaction from a Chinese official at the meeting.
Senior Colonel Zhao Xiaozhuo said a US move to send two warships into China’s “territorial waters” was a violation of law, and an “obvious provocation to China’s national security and territorial integrity”.
Mr Mattis responded that the question reflected a fundamental disconnect with the way international tribunals have spoken on the matter.
“We do not see it as a militarisation by going through what has traditionally been an international water space,” said Mr Mattis of the US ship movements through the South China Sea.
“What we see it as, is a reaffirmation of the rules-based order.”
Two US warships sailed close to the Paracel Islands on May 27 in the latest freedom of navigation operation designed to challenge Beijing’s claims. China protested the maneuver.