The crucial problem is right from the start: Colin interprets the task of defining man and woman as to define what he himself understands as man and woman. Stephen on the contrary refers to how society "usually" defines man and woman, but he refuses to provide a definition himself. So he plays motte and baily:
Whenever, he is criticized that he cannot a provide a definition, he refers to society. Then, whenever he is critiziced that these definitions are stereotypes, he says that he does not agree with them. The first suspicious moment was, when Stephen's definition of man and woman were exactly the same, so he did not provide any specific difference, which a classical definition must in any case provide.
The irony: When Stephen says, that he does not believe in the utility of gender categories and that he would rather live in a world without them, Colin could have just said: Welcome to my world, get rid of the gender talk, do not reinforce it - and let us rather look at what remains: biological sex differences.
Stephen's whole argument is a long winded circle that ends up where Colin would actually start: with sex, not with gender.
