I thought this article by physicist David Deutsch was very thought provoking. It helped me understand the philosophy of Karl Popper better, especially the tenuousness of learning by induction.
It really is a fascinating can of worms. While reading this I thought of my own attitudes to life and learning. When taking surveys, and indeed throughout life, it is customary for "intelligence" to be inferred by proxy from "educational attainment". It’s not a terrible fit. But there are too many spectacular anomalies in the wrong corners of that scatter plot to really feel like finding the most (traditionally) educated people is key to finding the most intelligent. I’ve worked now for over 10 years in a factory designed to churn out post graduate degrees and I can assure you, quality control isn’t 6 sigma. More disturbing than the occasional idiot electron cryo-microscopy scientist is the clearly large percentage of very intelligent people who have no opportunity or interest in traditional educational attainment.
I think what Deutsch is trying to say is that with computers, the practical limit to imparting cleverness, craft, skill, intelligence of an artificial nature is the exact same limitation we have in the educational system. With current educational technology, you can make a PhD, but you can not make a creative, contemplative person capable of new and interesting insights.
I would add "interesting" to the discussion. I don’t actually think it would be hard to genetically breed an artificial intelligence that produces all manner of novel insights. What’s currently seemingly impossible are insights that anyone cares about. But then again what does it mean to care? What is this motivation stuff? More worms.
One of my longstanding controversial beliefs about education is that no one ever teaches anyone anything meaningful. Things can be demonstrated to person B by person A, but to truly stick they must be internalized (intrinsically) by B. Or as Karl Popper is quoted in the article, "there is no such thing as instruction from without… We do not discover new facts or new effects by copying them, or by inferring them inductively from observation, or by any other method of instruction by the environment." Simply put, you can not pour brilliance into someone. This article posits that computers are no different.
To me education is about providing people with the environment and, for most, the motivation, to learn for themselves. Some people need a more conducive environment than others and the requisite motivation is highly variable as well. It may be interesting to try to isolate the factors that contribute to intrinsically motivated learners or people who thrive intellectually despite a difficult environment. Whatever these factors are may be the missing magic ingredient that separates (some) humans from other species and likewise common programming from true artificial general intelligence.