Tamiroff was born in either Tiflis or Baku, Russian Empire, to an Armenian family. He trained at the Moscow Art Theatre drama school for nine years. He arrived in the U.S. for the first time, in January 1923 on a three-month tour with a troupe of actors. He returned in November and stayed until 1924. His final trip with his theatre group was in October 1927 when he decided to stay permanently. Tamiroff managed to develop a career in Hollywood despite his thick accent.

Tamiroff's film debut came in 1932 in an uncredited role in Okay, America!. He performed in several more uncredited roles until 1935, when he co-starred in The Lives of a Bengal Lancer. He also appeared in the lavish epic China Seas in 1935 with Clark Gable, Wallace Beery, Jean Harlow, Rosalind Russell and Robert Benchley. The following year, he was cast in the titular role in The General Died at Dawn. He appeared in the 1937 musical High, Wide, and Handsome with Irene Dunne and Randolph Scott, and the 1938 proto-noir Dangerous to Know opposite Anna May Wong, frequently singled out as his best role.

In the following decade, he appeared in such films as The Buccaneer (1938) with Fredric March, The Great McGinty (1940), The Corsican Brothers (1941), Tortilla Flat (1942) with Spencer Tracy, Hedy Lamarr and John Garfield, Five Graves to Cairo (1943) with Erich von Stroheim as Field Marshal Erwin Rommel, Frank Borzage's His Butler's Sister (1943), For Whom the Bell Tolls (1943) with Gary Cooper and Ingrid Bergman, for which he received another Oscar nomination,  and Preston Sturges' The Miracle of Morgan's Creek (1944).

In later years, Tamiroff appeared in Ocean's 11 (1960) with Frank Sinatra and Dean Martin's Rat Pack, Topkapi (1964) with Peter Ustinov and Simone Signoret, Alphaville (1965), and had a long collaboration with Orson Welles including Touch of Evil (1958) with Charlton Heston, Mr Arkadin (1955), The Trial (1962) and Welles' unfinished version of Don Quixote, in which he played Sancho Panza.

While Tamiroff may not be a household name now, his malapropistic performance as the boss in The Great McGinty is thought to have been the inspiration for the cartoon character Boris Badenov, the male half of the villainous husband-and-wife team Boris and Natasha on The Rocky and Bullwinkle Show. He was also spoofed in a 1969 episode of the TV show H.R. Pufnstuf entitled "The Stand-in" in which a frog named "Akim Toadanoff" directs a movie on Living Island.

Tamiroff's accepted birth year was 1899, although in at least two instances this appeared to be different. In his second trip to America in November 1923 his age is given as 27 and in the 1930 census as 32. He married fellow actress Tamara Shayne, with whom he performed nightclub acts,^[3] in February 1933 in Los Angeles. Yet, according to the above-mentioned 1930 census, the couple was living in Chicago, Illinois as married under the (misspelled) name Tameriroff. It appears also that this was his second marriage.

Tamiroff died on September 17, 1972, from cancer.