Martin Scorsese wanted to direct after Mean Streets (1973) but the producers wanted Sydney Pollack. Scorsese has gone on record that he would very much have liked to direct the film and was disappointed that he was passed over. However, he got to direct Alice Doesn't Live Here Anymore (1974) instead after being sought out by Ellen Burstyn. Alice Doesn't Live Here Anymore ended up making more than 20 times it's budget and won Burstyn an academy award while The Yakuza became a box office bomb.
This is film critic-turned-screenwriter Paul Schrader's feature-film writing debut. He had also written Taxi Driver (1976) for Martin Scorsese and Obsession (1976) for director Brian De Palma. On this story, Schrader had worked with his brother Leonard Schrader who had been an American teacher in Japan. Together they came up with the premise involving a former American soldier who travels back to Japan to help a friend is in trouble with the yakuza. The Schrader brothers sold this idea to Warner Brothers for $300,000 (plus 30% of the net profit for his original screenplay), an unheard-of sum for a script at the time, particularly for two brand-new screenwriters. Schrader fleshed out their story with Robert Towne brought in for rewrites. Also Towne was also about to have several hits, including Chinatown (1974) released the very same year as The Yakuza (1974).
Sydney Pollack spoke no Japanese, and his cinematographer, Kôzô Okazaki, spoke no English. "We communicated in a very strange but effective way." They each had a gray-scale card going from white to black in ten stages, and they used that to communicate light density in various areas of a shot. "He did a lovely job."
The star who was originally set to play the central character was Lee Marvin with his The-Dirty-Dozen director Robert Aldrich also attached. Warner Brothers wanted Charles Bronson for the lead, and screenwriter Paul Schrader preferred Robert Mitchum. Warner Brothers refused to make the script changes Marvin wanted so he dropped out then Aldrich too, replaced by Mitchum with director Sydney Pollack. William Holden was also considered for leading man before the part finally went to Mitchum. Pollack briefly considered Robert Redford for the lead role.
"He was capable of a lot," says Sydney Pollack about Robert Mitchum, "but you had to push him." He thinks the actor, who often referred to himself as "an actress," didn't consider himself to be all the good without being ridden hard. "He was a real mule. He would give you what you wanted, but you had to beat him."