I don't see how so many people are missing this. Sam chimped out and shouldn't have, absolutely, but there's no question that Harland was intentionally trying to piss him off through out the podcast. And, no, it's not the same as his handling of other guests. "You look like you got raped" isn't some kind of dry humor or quirky "tee hee, I said a sexual thing!" comment. It's literally just "You're a faggot" ever so slightly dressed up.
Never in the history of the word has "You look like a bitch who gets raped" been a playful dig that is intended to solicit some kind of playful retort in response. There's nothing even to work with there. It's not like "I noticed you have a pink shirt on, so you must be gay" where there's some actual observation or medium for a reply such as picking out something Harland was wearing and making a similar insult based on that.
It's just blatant shit talking with a veneer of 'comedy' because it's a 'comedy' podcast. So, if you get upset with it, you're going to get told you can't take a joke or you're thinned skinned, even though there's no joke there. So, Sam initially just said "Thank you" which I think is a good dismissal of the it. But, it's obviously irritated him enough that when we get to the "You're a retarded chimp, go dance for me" part, he loses his cool.