Isis literally had a leader. Al Qaeda consists of various affiliate groups. Antifa is an idea, like fascism is an idea. If a country said "all fascists are now going to be arrested" you would think that's ridiculous in part because it just means the country would arrest anyone it identifies as fascist, not people part of actual fascist groups. Antifa is the same thing. It's a terroristic idea which a bunch of groups aspire to. I think anyone even laying claim to that idea should be removed from society, but legally speaking, specific groups have to be identified to impose legal consequences on the members of those groups. That's how constitutional criminal law works. Just having an idea or thought isn't criminal, it's the act plus the thought that is criminalized.
That's the difference between a guy smashing a window because he's crazy and a guy smashing a window because he's part of an antifa riot. One should be treated as a petty criminal and the other should be treated as a terrorist.