Unlike the original Transformer model, masking is applied to all multi-head attention layers in the proposed model. Because only information from previous interactions is needed for this problem (i.e., $$e_1, \cdots, e_n$$ and $$l_1, \cdots, l_{n-1}$$), masking is used across all layers.

Subsequent Masking (Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment)

The proposed model has a similar structure to the transformer, consisting of $$n$$ stacked encoders and decoders. The encoder consists of self-attention and feedforward layers; after each of these layers, there is a residual connection and layer normalization. Similarly, decoders contain the same layers with an additional fully connected layer for prediction. The encoder and decoders have distinctive inputs: for the encoder we have $$e_i$$s (question metadata features), and for the decoders we have response features shifted by one ($$(S, l_1, \cdots, l_{n-1})$$) and the output of the encoder. In their model, the authors use MultiHeadAttention and upper triangular masking.

Model Description (Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment)

As an input to the model, the features include question metadata ($$e_i$$) and response metadata ($$l_i$$). Their embedding vectors are defined as:

$$e_i = emb_e(id_i, c_i, st_i, p_i, sp_i)$$

$$l_i = emb_l(r_i, et_i, st_i, iot_i, d_i, p_i, sp_i)$$

Input Representation (Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment)

1. Input Representation
2. Model Description
3. Subsequent Masking

San Diego State University

The paper "Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment" proposes the Deep Attentive Study (DAS) model to predict the probability of student dropout from an ongoing study session. Specifically tailored for mobile learning environments, the study utilizes the Santa dataset (containing student interactions with an AI tutor) to evaluate the DAS model against baseline models such as LSTM and GRU networks.

Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment

Besides education, there are several different areas for which dropout prediction is also important problem like gaming, telecommunication etc. In all of these areas it is important to increase satisfactory level for the user and make their experience more enjoyable. In MOOC and school/college dropout prediction several different models were used: Support Vector Machines (SVM), Random Forrest Classifier, LSTMs. In other areas, very similar problem is Spotify Sequential Skip Prediction Challenge, here whether user will skip the song is predicted based on his/her playlist.  

Related Work (Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment)

In this paper Santa dataset is used which contains data collected from more than a million students. Santa is AI tutor that generates 100 questions per each session for the users. For this research only the data collected from mobile environment is utilized. Santa dataset prepares students for TOEIC (Test Of English for International Communication) exam. Here more than 200000 users interaction is used and the data is divided into training/validation/test sets.
This how the dataset looks like:


Dataset (Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment)

The experiment for predicting study session dropout in a mobile learning environment involves specific training configurations and baseline comparisons:

### 1. Training Details
* The model with the best Area Under the ROC Curve (AUC) score was selected for the testing phase.
* The optimal model configuration included 4 stacked encoder/decoder layers, a latent dimension of 512, and 8 heads in the multi-head attention layer.
* Xavier initialization and the Adam optimizer were used during training.

### 2. Baseline Comparison and Input Representation
* Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks were evaluated as baselines alongside the Deep Attentive Study (DAS) model.
* The input representations for these baseline models were:
  $$in_{i-1} + e_i$$
  and
  $$in_{i-1} = l_{i-1} + e_{i-1}$$
* Different sequence sizes of 5 and 25 were tested across all models.

Experiment (Deep Attentive Study Session Dropout Prediction in Mobile Learning Environment)

Intelligent tutoring systems often emphasize learning efficiency, but learner engagement and continued participation also affect the learning experience. Predicting dropout within an ongoing mobile study session differs from predicting withdrawal from a school or MOOC because mobile applications, messaging, and other interruptions can disrupt a session. In this study, consecutive interactions belong to the same study session when they are separated by no more than one hour; a longer period of inactivity ends the session and is treated as dropout. The paper proposes the transformer-based Deep Attentive Study (DAS) model and evaluates it using mobile interactions from the Santa dataset.

Motivation and Problem Setting for Mobile Study Session Dropout Prediction

In this study, a study session is a sequence of a user's interactions in which consecutive interactions are separated by less than one hour. Inactivity lasting more than one hour constitutes study-session dropout. A single session captures short-term user behavior, while combining sessions can represent longer-term behavior and help define user paths. The prediction task is to estimate, during an ongoing session, the probability that the student will drop out.

Learn Before

Related

Learn After