CHICHI JIMA, Japan — Every morning, as the sun rises over this remote Pacific Island and its tiny port with typically Japanese low-slung concrete buildings, John Washington commits a quiet act of defiance against the famously insular Japan: he hoists an American flag over his inn.

Mr. Washington, 63, whose white skin and blond hair, which is turning white, mark him as something of an outsider, is a great-great-grandson of the island’s founding father, an American sailor named Nathaniel Savory who set sail in 1830 with a band of adventurers for this island, which was known as a lawless natural wonder.

These days, Mr. Washington’s attempt to evoke that history seems increasingly like an act of desperation. His community — descendants of those settlers — is vanishing as young people leave this isolated outpost, a 25-hour ferry ride from Tokyo in a chain once known as the Bonins, or assimilate, dropping the Anglican religion and English language of their forebears.

“I feel it will all die out with my generation,” Mr. Washington said. “They don’t teach the history of the Bonin Islands to kids, don’t teach about Nathaniel Savory. The Japanese hide these things.”

And what they are hiding, he says, is a tale as colorful and lurid as it is disputed.

Since it was settled by Mr. Savory’s American and European followers — fortune seekers, deserters, drunkards — and their Hawaiian wives, the island has been pillaged by pirates, gripped by murder and cannibalism, and tugged back and forth between Japan and the United States in their battle for supremacy in the Pacific. There was a brief revival of the island’s Western culture after World War II, when it was ruled by the United States Navy.

Even the island’s V.I.P. visitor list seems outsized for a spit of land just five miles long. It includes Commodore Matthew C. Perry, who stopped here on the 1853 voyage in which he opened Japanese ports at gunpoint, and Jack London, who visited as a 17-year-old deckhand and later wrote about the Bonins.

Today, the island is a sleepy place. Its rhythms are set by the arrival once every six days of the ferry that makes the 600-mile journey from Tokyo, which has administered Chichi Jima as part of what are now known as the Ogasawara Islands, after the United States returned them to Japan in 1968.

About 2,000 people live here, mostly Japanese from the mainland who came after the transfer. Over time, they have overwhelmed the descendants of the original settlers — known here as Obeikei, or the Westerners — who are now estimated to number fewer than 200.

Most of the Obeikei are Japanese citizens. Most of those who still speak English and retain distinctly Western or Polynesian features are over the age of 50.

In a country that prides itself on its homogeneity and avoids tackling uncomfortable situations directly, many of Chichi Jima’s Japanese residents profess to having little knowledge of or interest in the Westerners. They instead focus on running the whale-watching and diving tours for the tourists drawn to a pristine island chain that last year was listed as a Unesco World Heritage site.

Some Japanese residents say the Westerners have made their own lot by being standoffish, using both Western and Japanese names, and pining to return to the “Navy time” after World War II, when they had the island virtually to themselves.

An old graveyard with Christian tombstones is one of the few visible traces of the Westerners’ history. And the official account of the island’s history, presented at the village-run visitor center, plays down the Westerners’ role in settling the island.

It says the island chain was discovered in 1593 by a samurai named Sadayori Ogasawara, for whom the chain was later named. The “Euro-American natives” are presented as little more than squatters who occupied what officials say was already Japanese territory, despite a consensus among modern Japanese and Western historians that Ogasawara never visited the islands.

“They are not the same as indigenous natives who have been here for hundreds of years,” said Kazuhiko Ishida, the island’s vice mayor. He said that while no efforts are being made to preserve the Westerners’ culture, they are not mistreated, either.

Westerners agree, but even some of those with close Japanese friends and spouses say feeling marginalized is not much better.

“They call me foreigner,” said Sutanrii Minami, 64, a tour guide who also goes by Stanley Gilley and who looks Polynesian. “I’m not a foreigner. I was born on this island.”

What is undisputed is that the island was left largely to rule itself until 1875, when Japanese settlers and officials took over in what the historian Daniel Long calls the first act of territorial expansion by a budding Japanese empire.

“Chichi Jima was probably the only case where the island was claimed by an Asian power and the natives were English-speaking Westerners,” said Mr. Long, who has written several books on the island.

It is also agreed upon that the island was untouched when sailors’ tales of an “uninhabited paradise” drew the 35-year-old Mr. Savory and about 20 settlers. They eked out a living selling provisions to passing Yankee whalers and British warships.

Many visiting captains remarked on the lawlessness of the island, recording tales of murder and polygamy. It also proved vulnerable to pirates, who in 1849 made off with Mr. Savory’s gold — and his wife. Witnesses later told a passing captain that the abduction was a tall tale: they said the woman, who was much younger than Mr. Savory, eagerly joined the marauders, leading them to his hidden wealth.

Islanders say that such raids may have led the settlers to peacefully accept the Japanese as rulers, who treated them with benign neglect.

That changed with the approach of World War II. Although they were not interned, the Westerners were forced to take Japanese names and were watched as possible spies. In 1944, most were evacuated along with the Japanese residents to the mainland, where they say they suffered discrimination.

“We are loyal Japanese, but they treated us as enemies when they saw the color of our faces and our eyes,” said Aisaku Ogasawara, 81, an Anglican pastor who also goes by Isaac Gonzales.

During the war, some of the Western men entered the Japanese Army, joining the garrison that defended the island. They witnessed a different horror, historians say, when eight captured American airmen were beheaded and then eaten by the starving Japanese defenders.

After the war, the United States Navy used the island for a submarine base. The Navy allowed the Western-descended settlers to return in 1946, but Japanese former residents were barred from coming back — possibly because of the nuclear warheads that historians say were stored on the island.

When the island was returned to Japan in 1968, the Westerners were given a choice of becoming either Japanese or American citizens. Many left for the United States.

Some wish that Japan and the United States had allowed them to decide the island’s future themselves.

“This island was returned without our control,” said Rokki Sebori, 52, who also goes by Rocky Savory and runs the island’s cooperative supermarket. “We still feel in our hearts that this is our island.”