Sean Penn told me not to be friends with Michael J. Fox

Stephen Baldwin says Sean Penn took his job very seriously while shooting War victims.

Speaking with Jamie Kennedy on a recent episode of the One bad movie podcast, the 58-year-old said “Sean Penn was one of my favorite actors” when Baldwin signed on for the 1989 war drama, which marked his biggest film at the time.

Baldwin was originally cast as in War victims, alongside Penn, 63, and Michael J. Fox. Because Penn and Baldwin’s characters did not get along with Fox’s in the film, Baldwin said Penn told him, “We can’t be friends with Mike Fox.”

“Our characters, we hate him. So you must hate him like I do. Do what I tell you,” Baldwin recalled Penn telling him.

Representatives for Penn and Fox, 63, did not immediately respond to PEOPLE’s requests for comment.

Never miss a story: Sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

But as Baldwin recalled to Kennedy, he ultimately did not follow Penn’s advice.

“When Sean Penn told Stephen Baldwin of Massapequa, New York, ‘Don’t talk to Mike Fox’… the first thing I did was (say to Fox), ‘Hey, I thought you were great at Family tiesBaldwin said to the 54-year-old Kennedy, laughing.

Baldwin, who was fired by director Brian De Palma during the making of the film, also recalled the “gnarly” experience of letting go. During a rehearsal with Penn, Baldwin said he asked for filming to stop because Penn was “mumbling” his lines. “Sean just walks away,” he recalled, adding that at that point De Palma told Baldwin, “You’re not cutting off my camera.”

Baldwin said he apologized to the director and asked that Penn speak that way during the scene. He then claimed that De Palma, now 83, punched him in the face and shouted: “You’re not asking what Sean’s going to do! Do you understand me?!”

After continuing to shoot for about a week, Baldwin said he was called to producer Art Linson’s office, where Linson told him they were letting him go.

“He says, ‘We’re sending you home… you’re on the plane tonight,’” Baldwin said.

After thanking Linson, now 82, and telling the producer he would go to his hotel room and pack his things to leave, Baldwin claimed Linson told him, “Everything in your room is already packed.” ”

Representatives for De Palma and Linson did not immediately respond to PEOPLE’s requests for comment.