Creating an LLM that can babble plausible nonsense at you, with no particular guarantee of truth although a lot of the time it will be perfectly sensible, is (for as fucking amazing a statement as this is) not overly hard at this point.

Getting it to make sense and not do bad things is the hard part. That was the huge innovation that made ChatGPT different from the nonsense-machines that were its early predecessors.

Rob Miles gave a very unsettling talk where he said, the main danger of AI is that commercial pressures will push it into the implementation phase, once the easy part of making it do stuff is done, before the much harder part is done of making it do reliably what you want it to do. And what we’re seeing now is a pretty solid confirmation that yes, that is absolutely the behavior we can count on in the future, and it will get more and more dangerous as AI models get more and more capable.