Information — Objective vs Subjective

Objective (Shannon) information

Properties of sources/distributions:

• Entropy $H(P)$: expected surprisal over outcomes under source $P$.
• Surprisal $−log P(i)$: information content of a particular outcome $i$.

Subjective (agent) surprisal

Properties relative to an agent’s model:

• Surprisal $−log Q(i)$: how unexpected $i$ is for an agent with beliefs $Q$.
• Varies across agents because Contexts differ.

Bridging them (cross‑entropy, KL)

• Cross‑entropy $H(P, Q) = H(P) + D_{KL}(P \|\| Q)$.
• The “penalty” for using $Q$ when the world runs on $P$ is the KL divergence.
• When $Q \approx P$, subjective and objective views align.

Learning as KL reduction

• Learning updates $Q \to Q'$ to reduce $D_{KL}(P \|\| Q)$.
• Operationally: reduce predictive log‑loss on future Texts under resource budgets.

Further reading: Text–Context–Interpretation, Operational Criteria, Agency & Delegation, Glossary.