Core Methods in Educational Data Mining HUDK4050 Fall 2014 The Homework

Lets go over the homework Was it harder or easier than basic homework 1?

What was the answer to Q1? What tool(s) did you use to compute it? What was the answer to Q2? What tool(s) did you use to compute it?

What was the answer to Q3? What tool(s) did you use to compute it? What was the answer to Q4?

What tool(s) did you use to compute it? What was the answer to Q5? What tool(s) did you use to compute it?

What was the answer to Q6? What tool(s) did you use to compute it? What was the answer to Q7? What tool(s) did you use to compute it?

What was the answer to Q8? What tool(s) did you use to compute it? What was the answer to Q9?

What tool(s) did you use to compute it? What was the answer to Q10? Who did Q11? Challenges?

Questions? Comments? Concerns? Textbook/Readings

Detector Confidence Any questions about detector confidence? Detector Confidence What are the pluses and minuses of making

sharp distinctions at 50% confidence? Detector Confidence Is it any better to have two cut-offs?

Detector Confidence How would you determine where to place the two cut-offs? Cost-Benefit Analysis

Why dont more people do cost-benefit analysis of automated detectors? Detector Confidence Is there any way around having intervention

cut-offs somewhere? Goodness Metrics Exercise

Detector Academic Suspension Detector No Academic Suspension

Data Suspension 2

3 Data No Suspension

5 140 What is accuracy?

Exercise Detector Academic Suspension

Detector No Academic Suspension Data Suspension

2 3

Data No Suspension 5 140

What is kappa? Accuracy Why is it bad?

Kappa What are its pluses and minuses? ROC Curve

Is this a good model or a bad model? Is this a good model or a bad model?

Is this a good model or a bad model? Is this a good model or a bad model? Is this a good model or a bad model?

ROC Curve What are its pluses and minuses? A

What are its pluses and minuses? Any questions about A? Precision and Recall

Precision = TP TP + FP Recall =

TP TP + FN Precision and Recall

What do they mean? What do these mean? Precision = The probability that a data point classified as true is actually true

Recall = The probability that a data point that is actually true is classified as true Precision and Recall What are their pluses and minuses?

Correlation vs RMSE What is the difference between correlation and RMSE? What are their relative merits?

What does it mean? 1. 2. 3.

4. High correlation, low RMSE Low correlation, high RMSE High correlation, high RMSE

Low correlation, low RMSE RMSE vs MAE RMSE vs MAE Radek Pelanek argues that MAE is inferior to

RMSE (and notes this opinion is held by many others) Radeks Example

Take a student who makes correct responses 70% of the time And two models Model A predicts 70% correctness Model B predicts 100% correctness

In other words 70% of the time the student gets it right Response = 1

30% of the time the student gets it wrong Response = 0 Model A Prediction = 0.7 Model B Prediction = 0.3

MAE 70% of the time the student gets it right Response = 1 Model A (0.7) Absolute Error = 0.3

Model B (1.0) Absolute Error = 0 30% of the time the student gets it wrong Response = 0 Model A (0.7) Absolute Error = 0.7

Model B (1.0) Absolute Error = 1 MAE Model A (0.7)(0.3)+(0.3)(0.7) 0.21+0.21

0.42 Model B (0.7)(0)+(0.3)(1) 0+0.3

0.3 MAE Model A (0.7)(0.3)+(0.3)(0.7)

0.21+0.21 0.42 Model B is better. (0.7)(0)+(0.3)(1)

0+0.3 0.3 MAE Model A

(0.7)(0.3)+(0.3)(0.7) 0.21+0.21 0.42 Model B is better. Do you buy that?

(0.7)(0)+(0.3)(1) 0+0.3 0.3 RMSE

70% of the time the student gets it right Response = 1 Model A (0.7) Squared Error = 0.09 Model B (1.0) Squared Error = 0 30% of the time the student gets it wrong

Response = 0 Model A (0.7) Squared Error = 0.49 Model B (1.0) Squared Error = 1 RMSE

Model A (0.7)(0.09)+(0.3)(0.49) 0.063+0.147 0.21

Model B (0.7)(0)+(0.3)(1) 0+0.3 0.3

RMSE Model A is better. (0.7)(0.09)+(0.3)(0.49) 0.063+0.147 0.21

Model B (0.7)(0)+(0.3)(1) 0+0.3 0.3

RMSE Model A is better. Does this seem more reasonable? (0.7)(0.09)+(0.3)(0.49)

0.063+0.147 0.21 Model B (0.7)(0)+(0.3)(1) 0+0.3

0.3 AIC/BIC vs Cross-Validation AIC is asymptotically equivalent to LOOCV BIC is asymptotically equivalent to k-fold cv

Why might you still want to use crossvalidation instead of AIC/BIC? Why might you still want to use AIC/BIC instead of cross-validation? AIC vs BIC

Any comments or questions? LOOCV vs k-fold CV Any comments or questions?

Other questions, comments, concerns about textbook? Creative HW 2

Creative HW 2 Due October *8* Creative HW 2 Yes, you get to breathe for a few days

Creative HW 2 Yes, you get to breathe for a few days (Sorry about assignment timing; my getting sick the second week of class threw off the

class timeline a little) Questions about Creative HW 2? Other questions or comments?

No Class Next Week Next Class Monday, October 6 Feature Engineering -- What

Baker, R.S. (2014) Big Data and Education. Ch. 3, V3 Sao Pedro, M., Baker, R.S.J.d., Gobert, J. (2012) Improving Construct Validity Yields Better Models of Systematic Inquiry, Even with Less Information. Proceedings of the 20th International Conference on User Modeling,

Adaptation and Personalization (UMAP 2012),249-260. The End