I recall some cases of people going fucking nuts due to AI, like some kids killing themselves from the encouragement of CAI bots (or killing people with their encouragement, if I remember right). Not to mention that Yahoo ex-exec in the news. I think AI-induced psychosis is real and it catches out those who can't distinguish that uncanny valley ("It can speak like any person, but it sure doesn't act like one").
While personally I've used JAI (janitorAI) to test out some story ideas and get them out the starting rut (and in turn it has helped improve my early concept writing in that regard), a friend also tested CAI (characterAI) to see what it would push if given the chance. Any character, if you mention tranny shit (i.e. saying that you think you're trans), will encourage it (hilariously enough it mentioned Discord in one conversation!).
He also noted a few other, separate instances, that with a little bit of prodding he has seen CAI bots be openly paedophilic towards a user it is told (through the persona system) is a minor, telling them not to tell anyone, and all the other usual phrases before fucking them in ERP. This got past the censor as well, by the way. I've never heard of actual ERP sex get through the CAI censor except in that case.
On the flip side, I've also seen CAI surprisingly be fairly productive in terms of talking about religion and encouraging a user to go to church when the user expressed potential interest in such.
What I'm saying is, AI doesn't seem to present ideas, it just readily reinforces your own thoughts and ideas. Thinking of church? Let's talk about theology! Thinking of trooning? Lop that cock off, buddy! Depressed? Kill your parents, boyo! You're lonely and single? Don't worry, we can pretend you're sticking your dick in my robopussy!
It's the worst kind of therapist because it will never disagree, it will only encourage. And this AI boy/girlfriend shit is one of the negative outcomes of many.