I don't trust the government to not use things like this to increase internet censorship under the guise of 'think of the children!!'
It's not the law or the principles I'm worried about but that there is going to be a push that the liability of these laws will be given to the software that generated them instead of solely on the user.
It is one thing to uphold the law and keep criminals accountable, it's another to demand these companies enforce nanny guidelines or surveillance into their systems. Already we see Apple allow the UK government to backdoor its systems. It sounds too plausible.
Even without the government, do we think it's a good idea to let the tech companies decide the moral usage of their products? I'm not sure how section 230 applies to generative AI, but I feel like the protections should be expanded similarly.
Look, I think child porn is one of those universally unforgivable crimes there is, and it's difficult to argue for any libertarian or free expression stance with that in mind, but I think this is why it is important to keep authorship away from the machine and towards its user. A lot of people have this tendency to claim that whatever the AI creates should not be attributed to the prompter, especially on the topic of copyright, so I'm worried that this will create a movement to push for restrictions on the technology, as the software is seen just as liable to the crime as the user since they did most of the work.
I think that premise on its face is ridiculous, because we do not penalize the camera manufacturer for creating the device that filmed the illegal content, but more outrageous is the idea that somehow a machine is more responsible for the crime than the person who requested it.
As far as fictional CP goes, when I was younger, I was in the camp of "It doesn't hurt real children". While I still agree that technically speaking no children were hurt in the production of such material, I find it that these kinds of materials are undeniably damaging to those that regularly consume them, and it's kinda inarguable that a lot of the imagery of that nature is too obscene for the public. As much as I want to champion free expression, there is a part of me nowadays that doesn't want to live in a world where content like that is proliferated, drawn or AI.