code generated by an AI is arguably not a “substantial portion” of the software

How do you verify that though?

And does the model need to include all of the licenses? Surely the "all copies or substantial portions" would apply to LLMs, since they literally include the source in the model as a derivative work. That's fine if it's for personal use (fair use laws apply), but if you're going to distribute it (e.g. as a centralized LLM), then you need to be very careful about how licenses are used, applied, and distributed.

So I absolutely do believe that building a broadly used model is a violation of copyright, and that's true whether it's under an open source license or not.