|
An Entropy and Noisy-Channel Model for Rule Induction| title | An Entropy and Noisy-Channel Model for Rule Induction |
|---|
| start_date | 2022/10/03 |
|---|
| schedule | 12h15 |
|---|
| online | no |
|---|
| summary | What triggers the inductive steps from memorizing specific items and combinations of items, to inferring rules (or statistical patterns) between these specific items, and also to forming categories and generalizations that apply to categories of items?
We propose an innovative information-theoretic model both for learning statistical regularities and generalizing to new input. Based on Shannon's noisy-channel coding theory (Shannon, 1948), our entropy model hypothesizes that rule induction (generalization) is an encoding mechanism gradually driven by the dynamics between an external factor – input entropy – and an internal factor – channel capacity.
Input entropy quantifies (in bits of information) the statistical properties of the linguistic input, given by the number of items and their probability distribution. Channel capacity is an information-theoretic measure of the encoding capacity of our brain, and is determined by the amount of entropy that can be encoded per second.
Specifically, in information-theoretic terms, if the input entropy per second is below or matches the channel capacity, the information about specific items and relations between them can be encoded with high-fidelity item specificity by item-bound generalization, at the channel rate (i.e. channel capacity). If the input entropy per second is higher than the channel capacity, then this high-specificity form of encoding becomes prone to errors, due to noise interference. So the form of encoding becomes inefficient, due to loss of information. Thus, the form of encoding is gradually shaped into a high-generality form of encoding – category-based generalization, in order to avoid exceeding the channel capacity.
I will present the results of three artificial grammar experiments (with adults) that tested this model, and aimed at better understanding the generalization mechanism and the type of generalizations that language learners make. Taken together, these results speak to the validity and wide application of this entropy model for the cognitive process of generalization. |
|---|
| responsibles | Hadjadj |
|---|
Workflow history| from state (1) | to state | comment | date |
|---|
| submitted | published | | 2022/11/22 16:23 UTC |
| |
|