Hi, nice post. Could you expand on this: “Narrative learning (done by AI) will outperform every other explainable machine learning technique.”? I am not sure I understand what you mean by narrative learning here and how this would outperform any XAI technique while staying reliable (not that XAI is reliable either, but that’s another can of worms).
I'm still writing it up. Narrative learning is where you ask an LLM to come up with some English language rules to classify data (or other language). Then you ask another LLM (or the same one in a different session) to apply those rules to the training data set, and identify the {true,false} {negatives,positives}. Then you feed that information (including a few samples) along with the previous rules to an LLM to get a new set of English language rules.
LLMs have been getting better at this quite rapidly, and by the end of the year will outperform decision trees and logistic regression on most metrics.
I would be very surprised if this ever outperforms a hyperparameter tuned xgboost on tabular data though. It probably will flatten out at about the performance level of CORELS or similar interpretable rule learning frameworks, given binary features. Perhaps it can be competitive at learning new features from unstructured data though.
It isn't constrained to binary features of course, since the "model coefficients" is whatever it is that can be expressed in English, which makes it richer than CORELS.
In the grand tradition of "never bet against AI being able to do something", do you want a wager on this? Narrative learning (and its descendants) vs xgboost in say 2035?
Perhaps yes, but we need to write this up properly. I would definitely bet that on small (define) tabular datasets (choose a set of benchmarks) with categorical features that were not part of the LLM's training data the ROC AUC of an xgboost classifier (set an implementation etc.) hyperparameter-tuned with (define method and amount of compute) will be within 5% (above or below) of the ROC AUC of narrative learning in 2030. Not because LLM's won't become all powerful but because we are already quite close to squeezing this kind of dataset so there isn't much room for improvement. I'm on manifold.
The high school & college students of today are pretty disadvantaged by the inability of teaching to adapt to the rapid pace of LLM improvements. The "no-tech" teaching option is laughable, since it could only be more ridiculously inadequate for helping kids enter the real world if they were asked to learn chancery cursive and wear 17th century clothing.
The wrangling path is the one advocated by Ethan Mollick at Wharton, who has been doing some of the best work envisioning post-AI education
But what I don't understand fully is why a student would want to *rely* on AI?
What are their plans for when they're out in the real world and asked a question in a meeting that their degree evidences they should know and were hired to know?
(Speaking as someone who plans on doing a Maths and Stats degree this year despite being closer to 30 than 20.)
I think that's the problem with the university of AI wrangling -- you don't know if their graduates know anything. But they would be so incredibly productive in the workplace, people might hire them anyway. Many students will go with options that bring them employment, even if it doesn't bring them all the other benefits that tertiary education provides.
I’m not entirely sure they would be so much more productive.
In fact, I’m very confident in the long run they risk being far (far) less productive - for several reasons.
I can’t find it now for some reason, but there was an internal group at Microsoft that had their github repo public and you can see a collective of junior coders repeatedly asking copilot to fix X…
There is a hidden assumption there: that being very good at using AI now and in the future will make you enormously more productive both short term and long term. If that's not the case, then there's no point in letting students use AI at university, and the university of no technology would be the #1 university to go to.
But if it is the case that there's a productivity benefit from structured experiences using AI to achieve outcomes, then the university of AI wrangling would be the #1 university to go to.
(That is of course a broad-brush statement. There might be specific students for whom it's still worthwhile to go to UoAIW even if it doesn't increase productivity; and likewise UoNT even if it does.)
Remember though that even 'realised' productivity gains (assuming it were true) may only be realised in the short term. If they introduce a bug that then requires more hours than an initial solution would have took then it has imparted a negative effect but would not be picked up by (external) research (as no company would say such things to the public)
I've been steering more into the belief that now more than ever is a good time *to* specialise deeply in one particular area. I don't understand how this not only wouldn't be more future-proof, but how *couldn't* it be?
Hi, nice post. Could you expand on this: “Narrative learning (done by AI) will outperform every other explainable machine learning technique.”? I am not sure I understand what you mean by narrative learning here and how this would outperform any XAI technique while staying reliable (not that XAI is reliable either, but that’s another can of worms).
I'm still writing it up. Narrative learning is where you ask an LLM to come up with some English language rules to classify data (or other language). Then you ask another LLM (or the same one in a different session) to apply those rules to the training data set, and identify the {true,false} {negatives,positives}. Then you feed that information (including a few samples) along with the previous rules to an LLM to get a new set of English language rules.
LLMs have been getting better at this quite rapidly, and by the end of the year will outperform decision trees and logistic regression on most metrics.
I would be very surprised if this ever outperforms a hyperparameter tuned xgboost on tabular data though. It probably will flatten out at about the performance level of CORELS or similar interpretable rule learning frameworks, given binary features. Perhaps it can be competitive at learning new features from unstructured data though.
It isn't constrained to binary features of course, since the "model coefficients" is whatever it is that can be expressed in English, which makes it richer than CORELS.
In the grand tradition of "never bet against AI being able to do something", do you want a wager on this? Narrative learning (and its descendants) vs xgboost in say 2035?
Perhaps yes, but we need to write this up properly. I would definitely bet that on small (define) tabular datasets (choose a set of benchmarks) with categorical features that were not part of the LLM's training data the ROC AUC of an xgboost classifier (set an implementation etc.) hyperparameter-tuned with (define method and amount of compute) will be within 5% (above or below) of the ROC AUC of narrative learning in 2030. Not because LLM's won't become all powerful but because we are already quite close to squeezing this kind of dataset so there isn't much room for improvement. I'm on manifold.
Having to beat xgboost by 5% and also retain some measure of explainability -- that seems unlikely.
What about a bet vs CORELS?
Who/what will produce the binary features for CORELS though? That makes all the difference.
The high school & college students of today are pretty disadvantaged by the inability of teaching to adapt to the rapid pace of LLM improvements. The "no-tech" teaching option is laughable, since it could only be more ridiculously inadequate for helping kids enter the real world if they were asked to learn chancery cursive and wear 17th century clothing.
The wrangling path is the one advocated by Ethan Mollick at Wharton, who has been doing some of the best work envisioning post-AI education
Here from ACX. Interesting read.
But what I don't understand fully is why a student would want to *rely* on AI?
What are their plans for when they're out in the real world and asked a question in a meeting that their degree evidences they should know and were hired to know?
(Speaking as someone who plans on doing a Maths and Stats degree this year despite being closer to 30 than 20.)
I think that's the problem with the university of AI wrangling -- you don't know if their graduates know anything. But they would be so incredibly productive in the workplace, people might hire them anyway. Many students will go with options that bring them employment, even if it doesn't bring them all the other benefits that tertiary education provides.
I’m not entirely sure they would be so much more productive.
In fact, I’m very confident in the long run they risk being far (far) less productive - for several reasons.
I can’t find it now for some reason, but there was an internal group at Microsoft that had their github repo public and you can see a collective of junior coders repeatedly asking copilot to fix X…
I don’t need to tell you how it went.
There is a hidden assumption there: that being very good at using AI now and in the future will make you enormously more productive both short term and long term. If that's not the case, then there's no point in letting students use AI at university, and the university of no technology would be the #1 university to go to.
But if it is the case that there's a productivity benefit from structured experiences using AI to achieve outcomes, then the university of AI wrangling would be the #1 university to go to.
(That is of course a broad-brush statement. There might be specific students for whom it's still worthwhile to go to UoAIW even if it doesn't increase productivity; and likewise UoNT even if it does.)
Hidden assumption on who's side?
Remember though that even 'realised' productivity gains (assuming it were true) may only be realised in the short term. If they introduce a bug that then requires more hours than an initial solution would have took then it has imparted a negative effect but would not be picked up by (external) research (as no company would say such things to the public)
I've been steering more into the belief that now more than ever is a good time *to* specialise deeply in one particular area. I don't understand how this not only wouldn't be more future-proof, but how *couldn't* it be?