---
+title: "Bayesian Probabilistic Knowledge Structures^[Correspondence to:
+ Florian Wickelmaier, Department of Psychology, University of Tübingen,
+ Schleichstr. 4, 72076 Tübingen, Germany, wickelmaier@web.de. Portions of
+ this material were presented at the Psychoco International Workshop on
+ Psychometric Computing, February 5-6, 2026, Padua, Italy. We thank Bob
+ Carpenter for his comments on an earlier version of this manuscript.]"
+author:
+ - Florian Wickelmaier
+ - Julian Mollenhauer
+ - Alice Jenisch
+date: "2026-10-20"
+format:
+ html:
+ toc: true
+ code-download: true
+ code-tools: true
+ embed-resources: true
+bibliography: "BayesianPKS.bib"
+abstract: "Knowledge structure theory (KST) seeks to provide procedures for
+the effective diagnosis of a student's knowledge state in a certain domain
+(such as algebra, geometry, physics, or statistics; Doignon & Falmagne, 1999;
+Heller & Stefanutti, 2024). Here, we introduce Bayesian versions of two
+prevalent probabilistic KST models: the basic local independence model and the
+simple learning model. We demonstrate how these models can be coded in Stan
+and interfaced with R. The Bayesian framework can then be leveraged to assess
+model adequacy via posterior predictive checks and for model comparison via
+cross-validation. These procedures are illustrated with simulated responses
+and a real-data example."
+---
+
+
+```{r}
+#| message: false
+#| code-fold: true
+#| code-summary: "R setup"
+
+rerun_dgp <- FALSE
+refit_model <- FALSE
+
+library(pks)
+library(rstan)
+options(mc.cores = 4)
+set.seed(1617)
+
+## For plotting with Rgraphviz
+myattrs <- list(
+ graph = list(rankdir = "BT", bgcolor = "white"),
+ edge = list(arrowsize = NULL),
+ node = list(fillcolor = "lightblue", fontsize = 10,
+ shape = "rectangle", fixedsize = FALSE)
+)
+```
+
+<!-- --------------------------------------------------------------------- -->
+# Probabilistic models in knowledge structure theory
+<!-- --------------------------------------------------------------------- -->
+
+Knowledge structure theory (KST) seeks to provide procedures for the effective
+diagnosis of a student's knowledge state in a certain domain, such as algebra,
+geometry, physics, or statistics [@DoignonFalmagne99; @FalmagneKoppen90;
+@HellerStefanutti24].
+
+KST offers an alternative approach to the representation of knowledge that is
+distinct from classical test theory [@LordNovick68] and from item response
+theory [IRT, @FischerMolenaar95; @Rasch60]: While the latter theories seek to
+establish a continuous numerical representation of the ability of a student
+(the true score or the latent trait), KST provides a discrete non-numerical
+characterization of the strengths and weaknesses of a student within a
+set-theoretic framework. Probabilistic KST models are latent class models and
+related to cognitive diagnostic assessment models [CDMs, @JunkerSijtsma01;
+@delaTorre09], which were originally devised to identify the qualitative
+attributes necessary for a student to solve certain problems. The connections
+between KST, IRT, and CDMs have recently been explicated [@HellerStefanutti15;
+@HellerStefanutti16; @NoventaHeller24; @NoventaHeller25; @NoventaHeller26].
+
+The basic notions in KST are
+
+- the knowledge domain: a finite set $Q$ of (dichotomous) problems (also
+ called items);
+
+- the knowledge state: the subset $K \subseteq Q$ of problems in the domain
+ $Q$ a student masters;
+
+- the knowledge structure: a collection $\mathcal{K}$ of knowledge states that
+ contains at least the empty set $\emptyset$ and the set $Q$.
+
+Only a subset of all possible response patterns $\mathcal{R}$ are also
+knowledge states in $\mathcal{K}$.
+
+## Example: High school geometry
+
+@Lakshminarayan95 presented 959 undergraduate students with the five problems
+in high school geometry shown in @fig-items.
+
+::: {#fig-items layout-ncol=2}
+
+{#fig-item_a}
+
+{#fig-item_b}
+
+{#fig-item_c}
+
+{#fig-item_d}
+
+{#fig-item_e}
+
+Five high school geometry problems.
+:::
+
+With the knowledge domain $Q = \{a,b,c,d,e\}$ consisting of five items, there
+are $2^{|Q|} = 32$ possible response patterns ranging from *no problem solved*
+($00000$) to *all problems solved* ($11111$). Not all of them are plausible
+knowledge states because solving a more difficult problem may require that a
+simpler problem is solved first. @DoignonFalmagne99 introduced a knowledge
+structure that is used here for illustration; it consists of $|\mathcal{K}| =
+9$ knowledge states (@fig-K-DF7).
+
+```{r}
+#| label: fig-K-DF7
+#| fig-cap: "Graph of a knowledge structure with nine knowledge states. A
+#| label such as $ab$ is an abbreviation for the set $\\{a, b\\}$. An
+#| ascending path between two states represents set inclusion."
+#| fig-width: 4
+
+data(DoignonFalmagne7, package = "pks") # provides knowledge structure
+K <- DoignonFalmagne7$K
+rownames(K) <- as.pattern(K, useNames = TRUE)
+rownames(K)[rownames(K) %in% c("{}", "abcde")] <- c("\u2205", "Q")
+plot(relations::as.relation(is.subset(K)), attrs = myattrs, main = "")
+```
+
+A knowledge structure like the one displayed in @fig-K-DF7 may be obtained in
+different ways: informally, by postulating plausible implications among
+problems; empirically, by analyzing all pairwise solution patterns and
+establishing a binary precedence relation between problems [@VanLeeuwe74;
+@Schrepp99]; by systematically querying experts (like teachers) whether
+failing certain problems would entail -- based on logical or curricular
+dependencies -- failing other problems [@FalmagneKoppen90; @KambouriKoppen94].
+
+## Deterministic and probabilistic theory
+
+The knowledge structure in @fig-K-DF7 is special in two ways: First, it is
+closed under union and thus called a knowledge space. Second, it is
+well-graded: it can be decomposed into learning paths along which a student
+can traverse -- by learning how to solve one problem at a time -- from a state
+of ignorance $\emptyset$ to full mastery $Q$. Consequently, $\mathcal{K}$ is
+downgradable: For any nonempty state $K$ there is some problem $q$ such that
+$K \smallsetminus \{q\} \in \mathcal{K}$. If learning proceeds by mastering
+one additional problem in each step, the outer fringe collects the problems a
+student in state $K$ is ready to learn next; it is defined as
+$$
+ K^\mathcal{O} = \{q \notin K \mid K \cup \{q\} \in \mathcal{K}\}.
+$$
+@fig-K-DF7-PathFringe shows one possible learning path through $\mathcal{K}$
+and illustrates the outer fringe of the knowledge state $\{a, b\}$: $c$ and
+$d$ are the problems a student in this state is ready to learn.
+
+```{r}
+#| label: fig-K-DF7-PathFringe
+#| fig-cap: "Panel a: Knowledge states marked in blue are along a learning
+#| path from $\\emptyset$ to $Q.$ Panel b: Knowledge states marked in green
+#| are the ones relevant for obtaining the outer fringe of $\\{a, b\\}$."
+#| fig-subcap:
+#| - "Learning path $b \\to a \\to c \\to d \\to e$"
+#| - "Outer fringe $\\{a, b\\}^\\mathcal{O} = \\{c, d\\}$"
+#| layout-ncol: 2
+#| fig-height: 7
+#| fig-width: 6
+#| code-fold: true
+
+rel <- relations::as.relation(is.subset(K))
+mycol <- setNames(rep("white", 9),
+ unlist(relations::relation_domain(rel)[[1]]))
+mycol["\u2205"] <-
+mycol["b"] <-
+mycol["ab"] <-
+mycol["abc"] <-
+mycol["abcd"] <-
+mycol["Q"] <- "lightblue"
+plot(rel,
+ nodeAttrs = list(fillcolor = mycol),
+ attrs = myattrs, main = "")
+mycol <- setNames(rep("white", 9),
+ unlist(relations::relation_domain(rel)[[1]]))
+mycol["ab"] <- "lightblue"
+mycol["abd"] <-
+mycol["abc"] <- "lightgreen"
+plot(rel,
+ nodeAttrs = list(fillcolor = mycol),
+ attrs = myattrs, main = "")
+```
+
+In deterministic KST, there is no explicit formulation of response errors. For
+example, the pattern $10010$ does not coincide with any state in
+$\mathcal{K}$. The pattern could arise if a student in state $\{a, b, d\}$
+makes a mistake at $b$ or if a student in state $\{a\}$ guesses the solution
+to problem $d$. In order to formulate a probabilistic model, the implicit
+distinction between knowledge state and response pattern has to be made
+explicit [@FalmagneDoignon88BJMSP; @FalmagneDoignon88JMP]. A knowledge state
+$K$ is a latent class, and the response pattern $R$ is a manifest indicator of
+the knowledge state. As a consequence, this gives rise to two types of
+response errors,
+
+- a careless error, which occurs if the response is incorrect although the
+ problem is contained in $K$;
+
+- a lucky guess, which occurs if the response is correct although the problem
+ is not contained in $K$.
+
+The latent knowledge states themselves are considered to be random variables
+with a probability distribution. Formally, a probabilistic knowledge structure
+consists of
+
+- a knowledge structure $\mathcal{K}$ on a knowledge domain $Q$ of problems;
+
+- the conditional probabilities $P(R \mid K)$ to observe response pattern $R$
+ given state $K$;
+
+- the marginal distribution $P(K)$ on the knowledge states $K \in
+ \mathcal{K}$.
+
+The probability of the response pattern $R \in \mathcal{R} = 2^Q$ then is
+predicted by
+$$
+ P(R) = \sum_{K \in \mathcal{K}} P(R \mid K) \cdot P(K),
+$$
+and the response frequencies $y = (N_R)_{R \in \mathcal{R}}$ follow a
+multinomial likelihood:
+$$
+ p(y \mid \theta) = \prod_{R \in \mathcal{R}} P(R)^{N_R}.
+$$
+
+Two prevalent models that make specific assumptions about the conditional and
+marginal probabilities are the basic local independence model and the
+simple learning model.
+
+### The basic local independence model (BLIM)
+
+For the conditional distribution $P(R \mid K)$, the BLIM assumes that given
+the knowledge state $K$ of a student, responses are stochastically independent
+over problems (local independence). This implies that the response to each
+problem $q$ only depends on the probabilities of a careless error $\beta_q$
+and a lucky guess $\eta_q$. Accordingly, the probability of the response
+pattern $R$ given the knowledge state $K$ becomes
+$$
+ P(R \mid K) =
+ \prod_{q \in K \setminus R} \beta_q
+ \cdot \prod_{q \in K \cap R} (1 - \beta_q)
+ \cdot \prod_{q \in R \setminus K} \eta_q
+ \cdot \prod_{q \in \bar{R} \cap \bar{K}} (1 - \eta_q).
+$$
+For example, the probability that a student in state $\{a,b\}$ solves the
+problems $b$ and $d$ is
+$$
+ P(01010 \mid \{a,b\}) =
+ \beta_a \cdot (1 - \beta_b) \cdot (1 - \eta_c) \cdot \eta_d \cdot
+ (1 - \eta_e).
+$$
+For the marginal distribution $P(K)$, the BLIM does not imply any
+restrictions; for each knowledge state $K$ it introduces a parameter $\pi_K$
+with $\pi_K \in (0, 1)$ and $\sum_K \pi_K = 1$. Thus, in total it has $2
+\cdot |Q| + |\mathcal{K}| - 1$ free parameters.
+
+### The simple learning model (SLM)
+
+Just like the BLIM, the SLM assumes local independence for the conditional
+distribution of responses given a knowledge state $P(R \mid K)$. On top of
+that, the SLM establishes restrictions on the marginal distribution of
+knowledge states $P(K)$. It requires $\mathcal{K}$ to be a well-graded
+knowledge space, that is a knowledge space that is downgradable. For each
+problem, it introduces a solvability parameter $g_q$, and the probability of a
+knowledge state $K$ becomes
+$$
+ P(K) = \prod_{q \in K} g_q \!
+ \prod_{q \in K^\mathcal{O}} \! (1 - g_q).
+$$
+This is the probability of mastering all problems in $K$ while not mastering
+any problems accessible from $K$. Accessible here means: problems that are in
+the outer fringe $K^\mathcal{O}$ of $K$ as defined above. For example, with
+the well-graded knowledge space in @fig-K-DF7-PathFringe, the probability of
+being in state $\{a,b\}$ is
+$$
+ P(\{a,b\}) = g_a \cdot g_b \cdot (1 - g_c) \cdot (1 - g_d).
+$$
+In total, the SLM has $3 \cdot |Q|$ free parameters.
+
+
+<!-- --------------------------------------------------------------------- -->
+# Fitting models to artificial responses
+<!-- --------------------------------------------------------------------- -->
+
+## The BLIM as data-generating model
+
+### Simulating responses
+
+Using the `simulate` function from the *pks* package, we generate response
+patterns for 2000 students using the knowledge structure in @fig-K-DF7 and the
+BLIM as data generating process (dgp). We assume a fixed distribution over the
+knowledge states and set all careless error and guessing probabilities to 0.1.
+
+```{r}
+dgp <- function(K) {
+ nitem <- ncol(K)
+ m0 <- list(
+ P.K = setNames(c(0.15, 0.1, 0.1, 0.08, 0.1, 0.12, 0.1, 0.1, 0.15),
+ as.pattern(K)),
+ beta = rep(0.1, nitem),
+ eta = rep(0.1, nitem),
+ K = K,
+ ntotal = 2000
+ )
+ class(m0) <- "blim"
+ y <- simulate(m0, method = "multinomial", zeropad = TRUE)
+ attr(y, "dgp") <- m0
+ y
+}
+```
+
+```{r, eval = rerun_dgp}
+N.R <- dgp(K)
+dput(N.R, file = "data/data_dgp_blim.R")
+```
+
+```{r}
+N.R <- dget("data/data_dgp_blim.R")
+datlist <- list(
+ R = as.binmat(N.R), # distinct response patterns
+ K = K, # knowledge states
+ Ko = getKFringe(K), # outer fringes
+ y = c(N.R), # by-pattern multinomial count
+ ycat = match(rep(names(N.R), N.R), names(N.R)), # by-subj response category
+ nstate = nrow(K),
+ nitem = ncol(K),
+ N = sum(N.R)
+)
+datlist$npattern <- nrow(datlist$R)
+datlist$y |> print()
+datlist$ycat |> table()
+```
+
+The list of data that is passed on to Stan includes two versions of the
+generated responses: `y` is a named vector that contains the multinomial
+counts $N_R$, some may be zero, but its length is always `npattern`, which is
+the number of all possible distinct response patterns; `ycat` is a vector that
+contains the response category of each student as a position index in `y`, it
+must be a number between 1 and `npattern`. Whenever some response patterns
+were not observed, the response frequency vector `y` should be completed with
+zeros, as is achieved by the `zeropad = TRUE` setting above. In addition, we
+assume that every student provides a response to every problem as the models
+below do not account for missingness of single item responses [see
+@deChiusoleStefanutti15].
+
+### Fitting the BLIM
+
+In Stan, the BLIM can be written as follows:
+
+```{stan, file = "stan/blim.stan", eval = FALSE, output.var = ""}
+```
+
+The conditional distribution $P(R \mid K)$ is implemented as a separate
+`getPRK` function, which is called in the model block to build the multinomial
+likelihood. The weakly informative priors for the $\beta_q$ and $\eta_q$
+parameters are beta distributions assuming low error rates. In addition, the
+error rates are restricted to satisfy $\beta_q + \eta_q < 1$. This ensures
+that a correct response is more likely if the problem is mastered than if it
+is not mastered, and it prevents the case $\beta_q = \eta_q = 0.5$ for all $q
+\in Q$, which would render the model nonidentifiable [@Heller17]. The prior
+for the $\pi_K$ parameters is a diffuse Dirichlet distribution, as both high
+and low probabilities for the knowledge states are plausible a priori. In
+Stan, a $\mathrm{dirichlet}(1, \dots, 1)$ distribution corresponds to a
+constant added to the log-posterior, so it can be omitted. If a more
+informative prior is desired, it can be added to the model block as `pi ~
+dirichlet(...);` with a suitable row vector of constants.
+
+In summary, the Bayesian BLIM can be written as:
+\begin{align*}
+ \beta_q &\sim \mathrm{beta}(2, 9), \quad \beta_q + \eta_q < 1\\
+ \eta_q &\sim \mathrm{beta}(2, 9)\\
+ \pi_K &\sim \mathrm{dirichlet}(\mathbf{\alpha}),
+ \quad \mathbf{\alpha} = (1, \dots, 1)\\
+ y &\sim \mathrm{multinomial}(P(R)),
+ \quad y = (N_R)_{R \in \mathcal{R}},
+ \quad \sum_{R \in \mathcal{R}} N_R = N\\
+ P(R) &= \sum_{K \in \mathcal{K}} P(R \mid K) \cdot P(K)\\
+ P(R \mid K) &=
+ \prod_{q \in K \setminus R} \beta_q
+ \cdot \prod_{q \in K \cap R} (1 - \beta_q)
+ \cdot \prod_{q \in R \setminus K} \eta_q
+ \cdot \prod_{q \in \bar{R} \cap \bar{K}} (1 - \eta_q)\\
+ P(K) &= \pi_K, \quad \pi_K \in (0, 1),
+ \quad \sum_{K \in \mathcal{K}} \pi_K = 1,
+\end{align*}
+where $R$ denotes the response pattern, $N_R$ its frequency, and $K$ a
+knowledge state.
+
+Stan fits the model to the generated responses and draws posterior samples for
+its parameters:
+
+```{r, eval = refit_model}
+## Code takes time to run; instead use saved output
+blim1 <- stan("stan/blim.stan", data = datlist, seed = 1634)
+saveRDS(blim1, "saved_fit/blim1.rds")
+```
+
+```{r}
+blim1 <- readRDS(file = "saved_fit/blim1.rds")
+print(blim1, pars = c("beta", "eta", "pi"), probs = c(.025, .975))
+```
+
+The convergence diagnostics and effective sample sizes do not indicate
+problems for any of the parameters. @fig-recovery-blim shows the posterior
+distribution of the BLIM parameters and their true values.
+
+```{r}
+#| label: fig-recovery-blim
+#| fig-cap: "Box plot of the posterior distribution of BLIM parameters, their
+#| posterior mean, and their true value."
+#| fig-height: 6
+#| fig-width: 5
+#| code-fold: true
+
+posteriorBlimPlot <- function(samples, npar = 5 + 5 + 9,
+ N.R = NULL, dgp = FALSE) {
+ with(samples, {
+ boxplot(cbind(beta, eta, pi)[, npar:1], horizontal = TRUE,
+ border = "gray", axes = FALSE, xlab = "Parameter value")
+ rev(colMeans(cbind(beta, eta, pi))) |>
+ points(y = 1:npar, col = "darkblue")
+ })
+ points(rev(coef(attr(N.R, "dgp"))), 1:npar, pch = 4, col = "darkblue")
+ axis(1)
+ axis(2, at = 1:npar, rev(c(
+ expression(beta[a]),
+ expression(beta[b]),
+ expression(beta[c]),
+ expression(beta[d]),
+ expression(beta[e]),
+ expression(eta[a]),
+ expression(eta[b]),
+ expression(eta[c]),
+ expression(eta[d]),
+ expression(eta[e]),
+ expression(pi[symbol("\306")]),
+ expression(pi[a]),
+ expression(pi[b]),
+ expression(pi[ab]),
+ expression(pi[abc]),
+ expression(pi[abd]),
+ expression(pi[abcd]),
+ expression(pi[abce]),
+ expression(pi[Q])
+ )), las = 1)
+ legend("topright", if(dgp) "BLIM (dgp)" else "BLIM", bty = "n")
+ box()
+}
+par(mai = c(.6, .5, .02, .07),
+ mgp = c(2, .7, 0))
+posteriorBlimPlot(extract(blim1), N.R = N.R, dgp = TRUE)
+legend("bottomright", inset = c(-.14, -.02),
+ legend = c("posterior mean", "true"), pch = c(1, 4),
+ col = "darkblue", bty = "n", ncol = 2, xpd = NA)
+```
+
+Regarding parameter recovery, we obtain mixed results: On one hand, for about
+half of the parameters the 50% regions of the box plots overlap with the true
+values, which indicates successful recovery. On the other hand, systematic
+deviations from the true values suggest that there are certain tradeoffs
+between parameters due to nonidentifiability. As a consequence of the
+knowledge structure being a well-graded knowledge space, some parameters of
+the BLIM are not identifiable [@Heller17; @StefanuttiHeller12].
+
+```{r}
+c(
+ space = pks:::is.knowledgespace(K),
+ downgradable = is.downgradable(K),
+ forward = names(which(is.forward.graded(K))),
+ backward = names(which(is.backward.graded(K)))
+)
+```
+
+The knowledge structure (@fig-K-DF7) is forward graded in problems $a$ and $b$
+(they can be added to any state and still yield a state) and backward graded
+in $d$ and $e$ (they can be removed from any state and still yield a state).
+Thus, the error and guess probabilities of these problems trade off with some
+of the state probabilities. The BLIM's partial nonidentifiability, however,
+does not compromise its predictive performance as shown below
+(@fig-postpred-blim).
+
+### Fitting the SLM
+
+In Stan, the SLM can be written like this:
+
+```{stan, file = "stan/slm.stan", eval = FALSE, output.var = ""}
+```
+
+For the marginal state distribution $P(K)$, the code includes the `getPK`
+function that is called in the model block. The solvability parameters $g_q$
+have a beta prior. Its parameters should be chosen to produce an a priori
+plausible marginal state distribution. The model-implied marginal distribution
+evaluated at the prior mean can be checked via:
+
+```{r}
+getSlmPK(g = rep(5/6, 5), K = K, Ko = getKFringe(K)) |> print(digits = 2)
+```
+
+In summary, the Bayesian SLM can be written as:
+\begin{align*}
+ \beta_q &\sim \mathrm{beta}(2, 9), \quad \beta_q + \eta_q < 1\\
+ \eta_q &\sim \mathrm{beta}(2, 9)\\
+ g_q &\sim \mathrm{beta}(5, 1)\\
+ y &\sim \mathrm{multinomial}(P(R)),
+ \quad y = (N_R)_{R \in \mathcal{R}},
+ \quad \sum_{R \in \mathcal{R}} N_R = N\\
+ P(R) &= \sum_{K \in \mathcal{K}} P(R \mid K) \cdot P(K)\\
+ P(R \mid K) &=
+ \prod_{q \in K \setminus R} \beta_q
+ \cdot \prod_{q \in K \cap R} (1 - \beta_q)
+ \cdot \prod_{q \in R \setminus K} \eta_q
+ \cdot \prod_{q \in \bar{R} \cap \bar{K}} (1 - \eta_q)\\
+ P(K) &= \prod_{q \in K} g_q \! \prod_{q \in K^\mathcal{O}} \! (1 - g_q),
+\end{align*}
+where $R$ denotes the response pattern, $N_R$ its frequency, $K$ a knowledge
+state, and $K^\mathcal{O}$ its outer fringe.
+
+Fitting in Stan yields:
+
+```{r, eval = refit_model, include = FALSE}
+slm1 <- stan("stan/slm.stan", data = datlist, seed = 1652)
+saveRDS(slm1, "saved_fit/slm1.rds")
+```
+
+```{r}
+#| code-fold: true
+
+slm1 <- readRDS(file = "saved_fit/slm1.rds")
+print(slm1, pars = c("beta", "eta", "g"), probs = c(.025, .975))
+```
+
+Also for the SLM, the convergence diagnostics and effective sample sizes do
+not indicate problems for any of the parameters.
+
+### Fitting the saturated multinomial model (MNM)
+
+The MNM serves as a reference for the two substantive models. It can be
+written like this:
+
+```{stan, file = "stan/multinom.stan", eval = FALSE, output.var = ""}
+```
+
+The MNM has one parameter for each distinct response pattern, so in total
+$2^{|Q|} - 1$ free parameters. We obtain the following abbreviated posterior
+summary:
+
+```{r, eval = refit_model, include = FALSE}
+mnm1 <- stan("stan/multinom.stan", data = datlist, seed = 1721)
+saveRDS(mnm1, "saved_fit/mnm1.rds")
+```
+
+```{r}
+#| code-fold: true
+
+mnm1 <- readRDS(file = "saved_fit/mnm1.rds")
+print(mnm1, pars = c("theta[1]", "theta[2]", "theta[32]"),
+ probs = c(.025, .975))
+```
+
+Again convergence diagnostics and effective sample sizes look good.
+
+### Posterior predictive checks
+
+@fig-postpred-blim shows the BLIM-generated and predicted frequencies for each
+distinct response pattern. The BLIM achieves a close fit between observations
+and predictions, not much worse than the MNM. The SLM's predictions are
+further off; this is especially visible for patterns with higher frequency.
+
+```{r}
+#| label: fig-postpred-blim
+#| fig-cap: "Box plot of the posterior predictive distribution of the response
+#| pattern frequencies for all three models. Data (marked by crosses) were
+#| generated by the BLIM."
+#| fig-height: 5
+#| fig-width: 8
+#| code-fold: true
+
+postPredPlot <- function(blim, slm, mnm, N.R, dgp = 1) {
+ par(mfrow = c(1, 3), omi = c(.4, .25, 0, .03), mai = c(.1, .1, .3, .02),
+ mgp = c(2, .7, 0))
+ blim$yrep |>
+ boxplot(border = "gray", horizontal = TRUE, axes = FALSE,
+ main = if(dgp == 1) "BLIM (dgp)" else "BLIM")
+ points(N.R, 1:32, pch = 4, col = "darkblue")
+ axis(1)
+ axis(2, 1:32, names(N.R), las = 1, tick = FALSE, line = -1,
+ outer = TRUE, cex.axis = 0.82)
+ box()
+ #
+ slm$yrep |>
+ boxplot(border = "gray", horizontal = TRUE, axes = FALSE,
+ main = if(dgp == 2) "SLM (dgp)" else "SLM")
+ points(N.R, 1:32, pch = 4, col = "darkblue")
+ axis(1)
+ axis(2, 1:32, labels = FALSE)
+ box()
+ #
+ mnm$yrep |>
+ boxplot(border = "gray", horizontal = TRUE, axes = FALSE, main = "MNM")
+ points(N.R, 1:32, pch = 4, col = "darkblue")
+ axis(1)
+ axis(2, 1:32, labels = FALSE)
+ box()
+ mtext(paste0("Frequency (N = ", sum(N.R), ")"),
+ side = 1, line = 1.5, outer = TRUE, cex = .8)
+}
+postPredPlot(extract(blim1), extract(slm1), extract(mnm1), N.R)
+```
+
+### LOO-CV model comparison
+
+When each model computes the observation-wise log-likelihood in the generated
+quantities block, approximate leave-one-out cross-validation [LOO-CV,
+@VehtariGelman17] is available to compare the models with respect to their
+ability to predict out-of-sample observations. This ability is quantified by
+the model's expected log predictive density (ELPD), which is defined as
+$$
+ \int \log(p(\tilde{y} | y)) p(\tilde{y}) d\tilde{y},
+$$
+where $p(\tilde{y} | y)$ is the posterior predictive distribution. The true
+data distribution $p(\tilde{y})$ is unknown, so the LOO estimate of ELPD is
+$$
+ \frac{1}{m} \sum_{i=1}^m \log p(y_i | y_{-i}),
+$$
+where $m$ is the number of observations and $y_{-i}$ are the data without the
+$i$th data point. Actual LOO-CV would require fitting the model $m$ times, the
+*loo* package avoids this by approximating the ELPD estimate via importance
+sampling. For the three models, we obtain the following summary:
+
+```{r, cache = TRUE}
+#| code-fold: true
+
+l1 <- loo(blim1)
+l2 <- loo(slm1)
+l3 <- loo(mnm1)
+loo::loo_compare(list(BLIM = l1, SLM = l2, MNM = l3)) |>
+ _[1:8] |> print(simplify = FALSE)
+```
+
+The `elpd_loo` and `se_elpd_loo` columns show the mean ELPD values and their
+standard error. The model with the largest ELPD is on top. The `elpd_diff` and
+`se_diff` columns show the mean ELPD difference values and their standard
+error relative to the top model. The standard error of the differences is
+smaller than the sum of the individual standard errors because the ELPD values
+are correlated. The `p_worse` column contains the probability for each model
+of having lower ELPD than the top model based on a normal approximation. The
+`diag_diff` and `diag_elpd` columns may contain diagnostic information about
+the ELPD difference being small or about the approximation to LOO-CV being
+non-trustworthy. More explanation of the output can be found in the
+documentation via `?loo::loo_compare`.
+
+For none of the three models are there any diagnostic flags, so these columns
+are empty here. The model most successful at predicting new response
+frequencies is the BLIM, followed by the MNM. The SLM is the least successful
+of the three. Remember that all three models were estimated based on the
+BLIM-generated data.
+
+## The SLM as data-generating model
+
+### Simulating responses
+
+We generate a new set of response frequencies from a rudimentary SLM. The
+solvability parameters $g_q$ are all set to 0.8; this gives rise to a
+distribution over the knowledge states that is different to the one above,
+where the BLIM was the data generator. Otherwise, the settings are the same as
+above.
+
+```{r}
+dgp <- function(K) {
+ nitem <- ncol(K)
+ m0 <- list(
+ beta = rep(0.1, nitem),
+ eta = rep(0.1, nitem),
+ g = rep(0.8, nitem),
+ K = K,
+ ntotal = 2000
+ )
+ m0$P.K <- getSlmPK(g = m0$g, K = K, Ko = getKFringe(K))
+ class(m0) <- c("slm", "blim")
+ y <- simulate(m0, method = "multinomial", zeropad = TRUE)
+ attr(y, "dgp") <- m0
+ y
+}
+```
+
+```{r, eval = rerun_dgp, include = FALSE}
+N.R <- dgp(K)
+dput(N.R, file = "data/data_dgp_slm.R")
+```
+
+We obtain the following response pattern frequencies:
+
+```{r}
+#| code-fold: true
+
+N.R <- dget("data/data_dgp_slm.R")
+datlist$ycat <- match(rep(names(N.R), N.R), names(N.R))
+datlist$N <- sum(N.R)
+datlist$y <- c(N.R) |> print()
+```
+
+### SLM parameter recovery
+
+```{r, eval = refit_model, include = FALSE}
+blim2 <- stan("stan/blim.stan", data = datlist, seed = 1742)
+saveRDS(blim2, "saved_fit/blim2.rds")
+```
+
+```{r, echo = FALSE, results = "hide"}
+blim2 <- readRDS(file = "saved_fit/blim2.rds")
+print(blim2, pars = c("beta", "eta", "pi"), probs = c(.025, .975))
+```
+
+```{r, eval = refit_model, include = FALSE}
+slm2 <- stan("stan/slm.stan", data = datlist, seed = 1746)
+saveRDS(slm2, "saved_fit/slm2.rds")
+```
+
+```{r}
+#| code-fold: true
+
+slm2 <- readRDS(file = "saved_fit/slm2.rds")
+print(slm2, pars = c("beta", "eta", "g"), probs = c(.025, .975))
+```
+
+@fig-recovery-slm shows the posterior distribution of SLM parameters and their
+true values. Again, the results are mixed. About half of the parameters are
+covered by their 50% posterior intervals as expected. But there are also
+systematic deviations from the true values visible as a tradeoff between the
+solvability and careless-error parameters for problems $d$ and $e$: Higher
+solvability goes along with more careless errors and vice versa.
+
+```{r}
+#| label: fig-recovery-slm
+#| fig-cap: "Box plot of the posterior distribution of SLM parameters, their
+#| posterior mean, and their true value."
+#| fig-height: 6
+#| fig-width: 5
+#| code-fold: true
+
+posteriorSlmPlot <- function(samples, npar = 5 + 5 + 5,
+ N.R = NULL, dgp = FALSE) {
+ with(samples, {
+ boxplot(cbind(beta, eta, g)[, npar:1], horizontal = TRUE,
+ border = "gray", axes = FALSE, xlab = "Parameter value")
+ rev(colMeans(cbind(beta, eta, g))) |>
+ points(y = 1:npar, col = "darkblue")
+ })
+ points(rev(coef(attr(N.R, "dgp"))), 1:npar, pch = 4, col = "darkblue")
+ axis(1)
+ axis(2, at = 1:npar, rev(c(
+ expression(beta[a]),
+ expression(beta[b]),
+ expression(beta[c]),
+ expression(beta[d]),
+ expression(beta[e]),
+ expression(eta[a]),
+ expression(eta[b]),
+ expression(eta[c]),
+ expression(eta[d]),
+ expression(eta[e]),
+ expression(g[a]),
+ expression(g[b]),
+ expression(g[c]),
+ expression(g[d]),
+ expression(g[e])
+ )), las = 1)
+ legend("topright", if(dgp) "SLM (dgp)" else "SLM", bty = "n")
+ box()
+}
+par(mai = c(.6, .5, .02, .03),
+ mgp = c(2, .7, 0))
+posteriorSlmPlot(extract(slm2), N.R = N.R, dgp = TRUE)
+legend("bottomright", inset = c(-.14, -.02),
+ legend = c("posterior mean", "true"), pch = c(1, 4),
+ col = "darkblue", bty = "n", ncol = 2, xpd = NA)
+```
+
+```{r, eval = refit_model, include = FALSE}
+mnm2 <- stan("stan/multinom.stan", data = datlist, seed = 1750)
+saveRDS(mnm2, "saved_fit/mnm2.rds")
+```
+
+```{r, echo = FALSE, results = "hide"}
+mnm2 <- readRDS(file = "saved_fit/mnm2.rds")
+print(mnm2, pars = "theta", probs = c(.025, .975))
+```
+
+### Posterior predictive checks
+
+In addition to the SLM, the BLIM and the MNM were fitted to the new data.
+@fig-postpred-slm shows the SLM-generated and predicted frequencies for each
+distinct response pattern. All three models achieve a close fit between
+observations and predictions.
+
+```{r}
+#| label: fig-postpred-slm
+#| fig-cap: "Box plot of the posterior predictive distribution of the response
+#| pattern frequencies for all three models. Data (marked by crosses) were
+#| generated by the SLM."
+#| fig-height: 5
+#| fig-width: 8
+#| code-fold: true
+
+postPredPlot(extract(blim2), extract(slm2), extract(mnm2), N.R, dgp = 2)
+```
+
+### LOO-CV model comparison
+
+```{r, cache = TRUE}
+#| message: false
+#| code-fold: true
+
+l1 <- loo(blim2)
+l2 <- loo(slm2)
+l3 <- loo(mnm2)
+loo::loo_compare(list(BLIM = l1, SLM = l2, MNM = l3)) |>
+ _[1:8] |> print(simplify = FALSE)
+```
+
+Now that the SLM has generated the responses that were used for parameter
+estimation, it also turns out to be the model with the highest predictive
+accuracy, closely followed by the BLIM. The MNM is worst of the three.
+Considering the uncertainty as indicated by the `se_diff` and `p_worse`
+values, the differences are not large.
+
+
+<!-- --------------------------------------------------------------------- -->
+# An empirical application
+<!-- --------------------------------------------------------------------- -->
+
+## High school geometry problems: Responses and knowledge structure
+
+Lakshminarayan [-@Lakshminarayan95; -@Lakshminarayan96] presented the high
+school geometry problems shown in @fig-items to 959 undergraduate students
+before and after a lesson. The data were split into a training and a held-out
+set; only the 800 responses in the training data set are reported. In this
+application, the posttest responses (@fig-N.R) will be analyzed. They are
+referred to by Doignon and Falmagne [-@DoignonFalmagne99, chap. 8]. The
+response frequencies are:
+
+```{r}
+#| label: fig-N.R
+#| fig-cap: "Response frequencies before and after a lesson on geometry."
+#| fig-height: 7
+#| fig-width: 6
+#| code-fold: true
+
+data(hsgeometry, package = "pks")
+dotchart(angles$N.R[, "posttest"], pch = 4,
+ main = "High school geometry problems",
+ xlab = "Response frequency (pre- [o] and posttest [x])")
+points(angles$N.R[, "pretest"], 1:32)
+mtext("Lakshminarayan (1995)", side = 3, line = 0.5)
+N.R <- angles$N.R[, "posttest"] |>
+ print()
+```
+
+@fig-K-Lakshmi displays the knowledge structure proposed by @Lakshminarayan95
+based on the content of the problems.
+
+```{r}
+#| label: fig-K-Lakshmi
+#| fig-cap: "Graph of a knowledge structure with eleven knowledge states."
+#| fig-width: 4
+#| code-fold: true
+
+K <- angles$K
+rownames(K) <- as.pattern(K, useNames = TRUE)
+rownames(K)[rownames(K) %in% c("{}", "abcde")] <- c("\u2205", "Q")
+plot(relations::as.relation(is.subset(K)), attrs = myattrs, main = "")
+```
+
+## Fitting no-guessing models in Stan
+
+With an open-answer format like in this problem set, it is plausible to assume
+that guessing the correct response is impossible or very unlikely. There are
+two options to implement this in Stan: Option 1 is to keep the guessing
+parameters in the model together with strong priors that favor no guessing.
+Option 2 is to set all guessing parameters $\eta_q$ to zero [as in
+@DoignonFalmagne99, chap. 8]. For Option 2, the `getPRK` function has to be
+adapted, so a no-guessing BLIM can be written in Stan like this:
+
+```{stan, file = "stan/blim-noguess.stan", eval = FALSE, output.var = ""}
+```
+
+The above code assumes that when response patterns are missing, their
+frequencies are set to zero such that the frequency vector is always complete.
+An alternative strategy is to leave out missing response patterns: `R` only
+contains the observed patterns and `y` their frequencies; `npattern` $<
+2^{|Q|}$ is their number. Then, the likelihood in the model block should avoid
+the `multinomial` distribution statement (which requires a simplex as an
+argument). Instead, a custom likelihood can be coded like this:
+
+```{stan, eval = FALSE, output.var = ""}
+transformed parameters {
+ vector[npattern] logPR = log(getPRK(beta, eta_filter, K_t, R) * pi);
+}
+model {
+ beta ~ beta(2, 9);
+ target += sum(y .* logPR); // custom likelihood
+}
+```
+
+### Posterior distributions
+
+We pass the data to Stan and obtain posterior samples for the no-guessing
+version of the BLIM.
+
+```{r, echo = FALSE, results = "hide"}
+datlist <- list(
+ R = as.binmat(N.R),
+ K = K,
+ Ko = getKFringe(K),
+ y = N.R,
+ ycat = match(rep(names(N.R), N.R), names(N.R)),
+ nstate = nrow(K),
+ nitem = ncol(K),
+ N = sum(N.R)
+)
+datlist$npattern <- nrow(datlist$R)
+```
+
+```{r, eval = FALSE, include = FALSE}
+blim3 <- stan("stan/blim-noguess.stan", data = datlist, seed = 1025)
+blim3_ml <- replicate(10, blim(K, N.R, method = "ML", etafix = rep(0, 5),
+ randinit = TRUE), simplify = FALSE)
+saveRDS(blim3, "saved_fit/blim3.rds")
+saveRDS(blim3_ml, "saved_fit/blim3_ml.rds")
+```
+
+```{r}
+#| code-fold: true
+
+blim3 <- readRDS(file = "saved_fit/blim3.rds")
+print(blim3, pars = c("beta", "pi"), probs = c(.025, .975))
+```
+
+```{r, eval = FALSE, include = FALSE}
+slm3 <- stan("stan/slm-noguess.stan", data = datlist, seed = 1026)
+slm3_ml <- replicate(10, slm(K, N.R, method = "ML", etafix = rep(0, 5),
+ randinit = TRUE), simplify = FALSE)
+saveRDS(slm3, "saved_fit/slm3.rds")
+saveRDS(slm3_ml, "saved_fit/slm3_ml.rds")
+```
+
+We also estimate the no-guessing version of the SLM.
+
+```{r}
+#| code-fold: true
+
+slm3 <- readRDS(file = "saved_fit/slm3.rds")
+print(slm3, pars = c("beta", "g"), probs = c(.025, .975))
+```
+
+@fig-posterior-blimslm displays the posterior distribution of the BLIM and SLM
+parameters. The pattern for the careless-error parameters $\beta_q$ is similar
+for both models. In order to compare the results with maximum likelihood
+estimation (MLE), the models were fitted using the *pks* package. When MLE is
+repeated with different starting values, the estimates show high variability
+for those parameters with the highest posterior uncertainty, indicating
+tradeoffs due to nonidentifiability.
+
+```{r}
+#| label: fig-posterior-blimslm
+#| fig-cap: "Box plot of the posterior distribution of BLIM and SLM
+#| parameters, their posterior mean, and maximum likelihood estimates (ten
+#| replications with different starting values)."
+#| fig-height: 5
+#| fig-width: 6
+#| code-fold: true
+
+posteriorBlimPlot <- function(samples, mlfit, npar = 5 + 11,
+ N.R = NULL, dgp = FALSE) {
+ with(samples, {
+ boxplot(cbind(beta, pi)[, npar:1], horizontal = TRUE,
+ xlim = c(1, 16), ylim = 0:1, border = "gray", axes = FALSE)
+ for(m in mlfit)
+ rev(coef(m)[!grepl("^eta\\.", names(coef(m)))]) |>
+ points(y = jitter(1:npar), pch = 16, cex = 0.6, col = "tan2")
+ rev(colMeans(cbind(beta, pi))) |>
+ points(y = 1:npar, col = "darkblue")
+ })
+ axis(1)
+ axis(2, at = 1:npar, rev(c(
+ expression(beta[a]),
+ expression(beta[b]),
+ expression(beta[c]),
+ expression(beta[d]),
+ expression(beta[e]),
+ expression(pi[symbol("\306")]),
+ expression(pi[a]),
+ expression(pi[b]),
+ expression(pi[ab]),
+ expression(pi[ad]),
+ expression(pi[bc]),
+ expression(pi[abc]),
+ expression(pi[abd]),
+ expression(pi[bcd]),
+ expression(pi[abcd]),
+ expression(pi[Q])
+ )), las = 1)
+ legend("topright", if(dgp) "BLIM (dgp)" else "BLIM", bty = "n")
+ box()
+}
+posteriorSlmPlot <- function(samples, mlfit, npar = 5 + 5,
+ N.R = NULL, dgp = FALSE) {
+ with(samples, {
+ boxplot(cbind(beta, g)[, npar:1], at = 1:npar + 6, horizontal = TRUE,
+ xlim = c(1, 16), ylim = 0:1, border = "gray", axes = FALSE)
+ for(m in mlfit)
+ rev(coef(m)[!grepl("^eta\\.", names(coef(m)))]) |>
+ points(y = jitter(1:npar) + 6, pch = 16, cex = 0.6, col = "tan2")
+ rev(colMeans(cbind(beta, g))) |>
+ points(y = 1:npar + 6, col = "darkblue")
+ })
+ axis(1)
+ axis(2, at = 1:npar + 6, rev(c(
+ expression(beta[a]),
+ expression(beta[b]),
+ expression(beta[c]),
+ expression(beta[d]),
+ expression(beta[e]),
+ expression(g[a]),
+ expression(g[b]),
+ expression(g[c]),
+ expression(g[d]),
+ expression(g[e])
+ )), las = 1)
+ legend("topright", if(dgp) "SLM (dgp)" else "SLM", bty = "n")
+ box()
+}
+blim3_ml <- readRDS(file = "saved_fit/blim3_ml.rds")
+slm3_ml <- readRDS(file = "saved_fit/slm3_ml.rds")
+par(mfrow = 1:2, mai = c(.4, .5, .02, .03), omi = c(.3, 0, 0, 0),
+ mgp = c(2, .7, 0))
+posteriorBlimPlot(extract(blim3), blim3_ml)
+posteriorSlmPlot(extract(slm3), slm3_ml)
+mtext("Parameter value", side = 1, outer = TRUE)
+legend("bottomright", inset = c(-.28, -.02), pch = c(1, 16),
+ legend = c("posterior mean", "MLE"), pt.cex = c(1, 0.6),
+ col = c("darkblue", "tan2"), bty = "n", ncol = 2, xpd = NA)
+```
+
+```{r, eval = FALSE, include = FALSE}
+mnm3 <- stan("stan/multinom.stan", data = datlist, seed = 1027)
+saveRDS(mnm3, "saved_fit/mnm3.rds")
+```
+
+```{r, echo = FALSE, results = "hide"}
+mnm3 <- readRDS(file = "saved_fit/mnm3.rds")
+```
+
+### Posterior predictive checks
+
+@fig-postpred-hsgeometry shows the observed response frequencies and their
+predictions from the BLIM, SLM, and MNM. The BLIM predictions are almost as
+close to the observations as the MNM predictions. The SLM is further off,
+especially for response patterns with high frequency.
+
+```{r}
+#| label: fig-postpred-hsgeometry
+#| fig-cap: "Box plot of the posterior predictive distribution of the response
+#| pattern frequencies for all three models. Crosses denote the observed
+#| response frequencies."
+#| fig-height: 5
+#| fig-width: 8
+#| code-fold: true
+
+postPredPlot(extract(blim3), extract(slm3), extract(mnm3), N.R, dgp = 0)
+```
+
+### LOO-CV model comparison
+
+```{r, cache = TRUE}
+#| message: false
+#| code-fold: true
+
+l1 <- loo(blim3)
+l2 <- loo(slm3)
+l3 <- loo(mnm3)
+loo::loo_compare(list(BLIM = l1, SLM = l2, MNM = l3)) |>
+ _[1:8] |> print(simplify = FALSE)
+```
+
+According to LOO-CV, the BLIM is not much worse than the saturated multinomial
+model at predicting replication data. The uncertainty reflected by the
+standard error of this difference is high. The SLM is noticeably worse than
+the other two models.
+
+
+# Summary
+
+In this case study, we introduced Bayesian versions of two prevalent models in
+knowledge structure theory, the basic local independence model and the simple
+learning model. We showed how to code these models in Stan, obtain posterior
+estimates, run diagnostic checks, and compare them using cross validation. For
+the two models illustrated here, there are classical methods available for
+estimating their parameters and evaluating their fit. We see the greatest
+benefit from adopting a Bayesian framework when we want to extend these
+models: (1) Bayesian methods allow us to implement models for which classical
+methods do not exist and would be difficult to obtain. (2) Because the prior
+distribution serves as a regularization mechanism, models with many parameters
+(as are common in KST) become feasible to estimate.
+
+
+# Computational details
+
+The *pks* package version 0.8-0 [@HellerWickelmaier13] was used to generate
+artificial responses, for data handling, for providing the high school
+geometry data, and for plotting knowledge structures leveraging facilities
+from *relations* [@relations] and *Rgraphviz* [@rgraphviz]. Stan was
+interfaced from R using *RStan* version 2.32.7 [@RStan]. The *loo* package
+version 2.10.1 [@loo] was used for running LOO-CV model comparison.
+
+
+# References
+