I'm a researcher in the field. Not in NLP anymore, but I worked on that as well, years ago, and I keep up with the research. You can't "understand the context" of "War and Peace" unless you have real, actual AGI. I doubt actually it can be fully understood at all when translated to English and read by someone without the right cultural background. This is an extreme example, chosen to make it easy to see that it applies to any non-trivial text.
Let's take a question answering example. Take just about any recent deep learning paper and try to answer detailed, higher level questions against it. To use a concrete example, take MobileNet V3 paper and ask your system "do I put activation before or after squeeze and excitation" (correct answer is "before"), or "do I need a bias in squeeze and excitation module" (correct answer is "it depends on the task"). You won't be able to, because a lot of things are just _assumed_, just like in any other realistic example of text written for human consumption. The facts are encoded externally as information about the world, and they're so fine grained and contextual, that we don't even know how to begin incorporating them into the answers, let alone do so contextually and probabilistically, like human mind does.
Maybe. I take a really optimistic view of attention based mechanisms, like I explained and added to my original post. If you read the recent reformer paper, they produce a new way of computation to start building a model that can in some way, encode the relationships between different parts of war and piece. The bottle neck right now is computation, we don’t know how well these models can learn when that bottle neck is removed!
I’m optimistic because I believe that the contextual information that you are describing, is already there in the vast expanse of the internet.
But I will also add, I think none of this will spawn AI, just that it will spawn new technologies that are categorically different.
I think it will merely spawn technologies that are less fragile, yet still too fragile and unsophisticated to be practical for full blown conversational user interface the likes of which you see in the Avengers movies. It will (and already does) make a difference in simpler tasks where you can get away with just counting numbers, such as search/retrieval, simple summarization, constrained dialogue in chatbots, stuff like that.
Let's take a question answering example. Take just about any recent deep learning paper and try to answer detailed, higher level questions against it. To use a concrete example, take MobileNet V3 paper and ask your system "do I put activation before or after squeeze and excitation" (correct answer is "before"), or "do I need a bias in squeeze and excitation module" (correct answer is "it depends on the task"). You won't be able to, because a lot of things are just _assumed_, just like in any other realistic example of text written for human consumption. The facts are encoded externally as information about the world, and they're so fine grained and contextual, that we don't even know how to begin incorporating them into the answers, let alone do so contextually and probabilistically, like human mind does.