There was this scene in the motion picture "Captain America: The First Avenger" where Steve Rogers and Agent Peggy Carter are driving through Brooklyn. The below dialogue ensued:

> **Steve Rogers**: I know this neighborhood. I got beat up in that alley.
> And that parking lot. And behind that diner. 

>  **Peggy Carter**: Did you
> have something against running away? 

>**Steve Rogers**: You start running
> they'll never let you stop. You stand up, push back. ***Can't say no
> forever, right***?

Who is Steve Rogers referring to when he says "Can't say no forever"?  Is he referring to himself, meaning he can't say no to a fight forever or is he referring to the men who used to beat him up?