WASHINGTON--Hewlett Packard has found a new club to use to pound researchers who unearth flaws in the company's software: the Digital Millennium Copyright Act.

Invoking both the controversial 1998 DMCA and computer crime laws, HP has threatened to sue a team of researchers who publicized a vulnerability in the company's Tru64 Unix operating system.

In a letter sent on Monday, an HP vice president warned SnoSoft, a loosely organized research collective, that it "could be fined up to $500,000 and imprisoned for up to five years" for its role in publishing information on a bug that lets an intruder take over a Tru64 Unix system.

HP's dramatic warning appears to be the first time the DMCA has been invoked to stifle research related to computer security. Until now, it's been used by copyright holders to pursue people who distribute computer programs that unlock copyrighted content such as DVDs or encrypted e-books.

If HP files suit or persuades the federal government to prosecute, the company could set a precedent that stifles research into computer security flaws, a practice that frequently involves publishing code that demonstrates vulnerabilities. The DMCA restricts code that "is primarily designed or produced for the purpose of circumventing protection" of copyrighted works.

On July 19, a researcher at SnoSoft posted a note to SecurityFocus.com's popular Bugtraq mailing list with a hyperlink to a computer program letting a Tru64 user gain full administrator privileges. The researcher, who goes by the alias "Phased," said in the message: "Here is the warez, nothing special, but it does the job."

That public disclosure drew the ire of Kent Ferson, a vice president in HP's Unix systems unit, who alleged in his letter on Monday that the post violated the DMCA and the Computer Fraud and Abuse Act.

"HP hereby requests that you cooperate with us to remove the buffer overflow exploit from Securityfocus.com and to take all steps necessary to prevent the further dissemination by SnoSoft and its agents of this and similar exploits of Tru64 Unix," Ferson wrote, according to a copy of the letter seen by CNET News.com. "If SnoSoft and its members fail to cooperate with HP, then this will be considered further evidence of SnoSoft's bad faith."

Ferson also said that HP reserves the right to sue SnoSoft and its members "for monies and damages caused by the posting and any use of the buffer overflow exploit."

HP refused to discuss Ferson's letter. "We're not going to comment on this," spokesman Jim Dunlap said on Tuesday.

Last year, Adobe Systems persuaded the Justice Department to prosecute Dmitry Sklyarov, a Russian programmer who allegedly violated the DMCA by writing an e-book unscrambler. Charges against Sklyarov were eventually dropped in exchange for his testimony in his company's trial, which begins Aug. 26 in San Jose, Calif.

Researcher Phased did not reply to a request for comment. But in an e-mail sent to SnoSoft on Tuesday, Phased said he was not worried about legal action because he released it independently of SnoSoft, adding, "I'm not American; the law doesn't apply to me." SnoSoft representatives said they did not know where Phased lived.

SnoSoft began talking with HP this spring about the group's research into Tru64 Unix's security flaws and had not intended to release the code publicly.

SnoSoft co-founder Kevin Finisterre said on Tuesday afternoon that Phased released the C language code, which was created by another SnoSoft programmer, without authorization from the group.

SecurityFocus.com, which is in the process of being acquired by Symantec, said it had already deleted a copy of the C source code from its Web site at the request of SnoSoft.

"Shortly after (the Bugtraq post), we were contacted by SnoSoft to suggest that this was leaked by a member who was not following the rules, and it should not have made its way onto the list," said Dave Ahmad, the moderator of the Bugtraq list. When an organization that contributed an exploit wants to modify or delete it, SecurityFocus.com's policy is to comply, Ahmad said.

Ahmad said that while the source code had been removed, the original post remained in the Bugtraq archives. Whether to delete it or not is "still a decision that I have to make," Ahmad said.

Triggering penalties
Robin Gross, an attorney at the Electronic Frontier Foundation (EFF), predicted HP would be one of many companies striving for broad interpretations of the DMCA. "These are the kinds of letters that we can expect to see now that the DMCA has granted such broad powers to copyright holders," Gross said. "Any information that can bypass controls will trigger DMCA penalties.

"The DMCA is so broad in what it prohibits it does include preventing researchers from revealing security weaknesses in operating systems--even though that has nothing to do with protecting copyright."

The EFF represented Princeton University professor Ed Felten after he was threatened with a DMCA lawsuit for exposing weaknesses in a music watermarking scheme. The San Francisco-based nonprofit group also backed hacker publication 2600, which was successfully sued by eight movie studios for distributing a DVD-decrypting utility.

SnoSoft representatives stressed in an interview that they wanted a cordial relationship with HP. They provided a copy of an e-mail message sent before the July 19 posting in which HP had discussed a deal with SnoSoft, asking what it would "cost for you to share, under NDA, the problems you have discovered to date for Tru64 Unix V5.1 and/or V5.1a."

HP has known about the Tru64 vulnerability "for some time," SnoSoft's Finisterre said, but never fixed the problem. An HP spokesman said he did not know if a patch had been released.

Another researcher, who uses the alias K2 and is part of the ADM hacking group, released a similar exploit in 2001 that also gave a person complete access to a Tru64 Unix system.

Finisterre said that while he wanted to resolve the dispute with HP, he resented receiving DMCA threats. "We are like the guys that found out that Firestone tires have issues on Ford explorers," he said. "It's not our fault your Explorer has crap tires. We just pointed it out. We should not get attacked for pointing out issues in someone’s product nor for proving it is possible."

Ahmad of SecurityFocus.com said that HP's Tru64 operating system is no more secure than other mainstream Unix variants.

"A lot of the time, when a major Unix has some vulnerability, Tru64 Unix will also be vulnerable just as a result of shared code," Ahmad said. "Also it's old code, and it's my belief that much of it was written without an understanding of the modern code problems that can be exploited by hackers."

Tru64 Unix came in last place in a recent survey by a computing research firm. As a result of HP's acquisition of Compaq Computer, Tru64 is being phased out over the next few years, and its features are supposed to be folded into HP-UX.

In an unrelated incident last week, HP asked one of its employees not to engage in a public demonstration that would have arguably violated the DCMA.