The M5 (from the episode "The Ultimate Computer") is built by a supergenius named Richard Daystrom, and it is installed on the Enterprise for a series of wargame-style tests that will put the M5-controlled Enterprise against several other starships. Everything is going swimmingly, until the wargames get intense enough that the M5 assumes that it is actually under attack, at which point it opens a computer-driven can of whoop-ass. It turns out that Daystrom used engrams from his own mind to program the M5, so it thinks the way he does – which is unfortunate, as Daystrom turns out to have some psychological issues. The computer is going all-out, which forces Starfleet to also go all out, planning to destroy the Enterprise, unless Captain Kirk can save the day. Which he does, of course, using his tried-and-true method of talking the computer into a logical contradiction that forces it to shut itself down.
Kirk would do this a few other times in the course of the series, and it became enough of a cliché that in the DC Comics Trek series in the late 80s, there was an issue where the Enterprise crew comes up against a supercomputer and the first thing Kirk does is say something like, "Well, I suppose I could try talking it into a logical contradiction. That usually works." Needless to say, it doesn't.
But supercomputers with a malevolent bent are certainly not unique to Star Trek. There was the computer in Wargames, the one designed to manage the entire nuclear defense of the United States. There was the Master Control Program in TRON, and, of course, the granddaddy of all malevolent supercomputers, the HAL-9000 in 2001: A Space Odyssey, whose insanity would later be explained in 2010: Odyssey Two, which ends on some tension as the entire crew's survival depends on HAL, whose sanity is open to question. (To the extent that one can even talk about sanity in a computer.)
And let's not forget about Skynet. That one turned out poorly, didn't it? Beware the supercomputers! They'll kill us all!!!