Student Experience: how to improve the learning experience at university

learning artificial intelligence

Donostia. 2021

How would you rate the learning experience you had/are having at the university? Could you say what makes you most satisfied with a teacher?

When we talk about experiences, it is difficult to put our feelings into words. Furthermore, these sensations of one person can be very different from those of another, and it is not easy to find common patterns. However, we are all capable of fondly remembering a teacher from our pre-university stage, a person who transmitted something different to us from the rest.

Wouldn't it be ideal to be clear about what makes that teacher generate a positive experience for us? Wouldn't it be useful for that teacher (and the coordination team) to take this into account as a critical factor in measuring their performance?

And, let's be clear. In addition to the experience, we highly value the final result of the process. The extrinsic motivation of the final evaluation is a very relevant factor, so wouldn't it be important to know what makes a teacher improve the academic results of a group? In this way, not only have a happy group, but a group that offers its greatest potential. And, with the university so close to the labor market, this is even more important than in other training stages.

Why have we gotten into this?

When we started thinking about these questions, we believed that most of them would already be answered. But, it seems the measurement of the college experience is not a topic of national debate. And the rankings are almost more dedicated to how well MBAs are ranked than to student satisfaction.

So since Saturdays.AI Bilbao a team made up of two university students (Gorka Legarreta Ibarra and Rubén García Pedrejón) and a server, university professor, (Iñaki Fernández López-Zuazo) we got to work. The 3 of us, for obvious reasons, are interested in making the experience and academic results the best possible. And, from a very personal point of view as a teacher, if there is something that irritates me, it is that everyone believes that they are right about what is the best way to teach/educate. So, quoting Deming “in order not to be another person with an opinion” we are going to work to reach conclusions based on data.

None of the 3 of us had experience in programming, but by dint of practicing, practicing, and practicing (and a little bit of controlC+controlV, it must be said) we have reached some interesting conclusion.

If you are a student, are you prepared to know what you have to ask of your university to have the best experience and grades possible?

If you are a university professor, are you prepared to know the elements on which you need to focus the most to improve your professional performance?

The dataset

Without data there is no paradise, and it has been difficult to obtain a good database that contains enough information to reach conclusions of interest. A university has kindly provided a dataset, anonymizing any characterization attributes, and introducing multipliers to some attributes to prevent their identification. These changes have not affected the result of the project in any case, since both datasets (the original and the modified one) yield the same conclusions. Finally, although the comments provided by the students in the dataset were used in this analysis, they were subsequently deleted, as they contained information that made it easy to identify teachers and specific situations.

This dataset contains information on more than 20,000 satisfaction surveys carried out on students from February 2015 to December 2020. The survey information has been completed with identifying data of the teacher who taught the subject and the average grade of the group.

Going into detail, the information that will be worked on the most throughout the dataset is:

Answers to specific satisfaction questions: The teacher's knowledge, his/her way of explaining, the methodology used in the classroom and the feedback he/she gives are evaluated. Finally, a general grade is given.

Average grade: An average has been made of the entire group that responds to the survey. That is, a record does not contain the grade that the student obtained in the evaluation, but rather the average grade of the entire group to which he or she belongs.

Teacher's identification data and his subject: Sex, age, campus where you usually work, type of subject taught...

EDA: How right they were...

In the first sessions of Saturdays.AI The importance of data cleaning is always mentioned, and that it is a task that takes more than 80% of the time of almost any project. Honestly, it seemed like an exaggeration, but perhaps we have reached 90% :)

In order not to get too confused at this point, these have been the biggest transformations:

• Elimination of records with NaN: Having such a large database, we believed that it was not worth inferring results, and we were left with only those records that had all the information.
• Focus on a particular grade: We had information on several grades, but the information on the rest of them was partial, and we also did not have their notes, which were key to the project. So we decided to focus on a single grade.
• Homogenization and elimination of attributes: In a specific year, the learning model was changed towards co-teaching, and for each classroom there are 3 teachers. Therefore, the student gave grades to the three, and that solved some problems for the homogenization of the dataset. All solved with a lot of effort and determination 🙂
• Categorization of attributes: To improve subsequent analysis, the responses to the satisfaction questions (with the Net Promoter Score) and the grades were categorized. In satisfaction we categorize into detractors (0 to 6), passive/neutral (7 and 8) and promoters (9 and 10). In the grades: failed (0 to 4.9), approved (5 to 7.9) and outstanding (8 to 10).

Exploring the data: reaffirming intuitions

With the dataset ready to work on, we start with a Heatmap to know the correlation between all the variables:

If we look at the variables related to satisfaction, we can see that the methodology is what correlates most with the teacher's overall grade (although explaining and feedback are very close) and the teacher's knowledge, the least. Come on, we begin to reaffirm something that we already imagined: no matter how much a person knows, since they do not have the appropriate methodology, they may not satisfy the students enough. But be careful, knowing is also correlated with explaining, therefore, to be able to explain well you have to know well what is being taught. Necessary condition, but not sufficient.

We also found it interesting to know if gender and age influence student satisfaction, so we started to group them with these criteria:

Well, it seems that younger people generally obtain better scores. There are no notable differences between the sexes, although it is true that older women (>55) seem to offer a better experience than men their age.

Finally, we wanted to know if, beyond age, what was relevant was the professor's seniority at the university. The teaching staff is ordered according to their entry into the faculty, so it was enough to plot this order with respect to the satisfaction score.

Well yes, it seems that the new signings have fewer negative scores than the local veterans. Of course, it is more difficult for them to reach 10.

Now that we have some ideas about the dataset, we move on to the models.

From supervised to unsupervised learning, seasoned with preschool NLP

We have worked on 4 models, each one with a goal.

-Linear regression: To be able to predict general satisfaction if we have the 4 satisfaction items, and know the importance of each of them.

-Association: To know which attributes “mix” more with others.

-Decision Trees: To easily classify promoters/detractors/neutrals.

-Clustering: To identify the relationship between grade and satisfaction, and most importantly, describe the groups of teachers that are formed.

-NLP: To know which comments are repeated the most according to satisfaction and the group's grade.

Linear regression

We chose the 4 satisfaction variables as independent variables, and general satisfaction as the dependent variable of the model. We assumed, given the correlations, that we were going to have good results.

And so it was, using the Ridge regression technique we obtained an accuracy of… 67%! Surely by playing with the train and test data we can reach a better result, but we were satisfied. To better contextualize this data, we measure the relative importance of each variable.

That is, we can predict the result, and we verify that explaining is the element that varies this result the most. So you know teachers, if you think you are doing badly with a group, explain better!

Association

We already have several clues about what affects satisfaction most, but we still do not know if there is a relationship between a group with good grades and a good teacher. To verify this, we implemented the “a priori” algorithm to visualize the associations between variables. Next, we attach the associations with a lift>1 (they occur more than expected).

Although there is a little bit of everything, the association with greater lift and confidence is “outstanding” with “promoter.” Therefore, we can intuit that those groups that have an outstanding average have a teacher that they have valued very positively, but not the other way around. And we cannot conclude that bad grades are mostly associated with bad teachers.

To go deeper into what makes that teacher have promoters or detractors, we start with the decision trees.

Decision Trees

The first test we did was with a depth of 2, to begin to see the first results.

Conclusions similar to the above for teachers: Explaining is what most differentiates promoters from detractors/neutrals. But to ensure a greater number of student promoters, it is better to have a good methodology in the sessions. And, if explaining is not your strong point, focus on giving good feedback so as not to have detractors.

But hey, the score of the tree is 0.35, so you have to take the result with some skepticism.

If we expand the depth to 5, we already see that new attributes enter, and the score increases to 0.45. A insight What we discovered with this tree is that, if the feedback is not good enough, but the knowledge perceived by the student is high, the possibility of having promoters increases.

Given all this, we are going to focus on what the teachers are like according to the satisfaction of the students and the grades they give.

Clustering

First of all, we apply the K-Means algorithm to identify the optimal number of clusters: 6. We now plot this relationship between grades and satisfaction:

We have 6 different groups, but we are going to focus on 3 of them:

• Teachers who only have detractors, regardless of the group's average grade:. He is over 55 years old, from the western area of ​​Gipuzkoa, with enough knowledge to be well valued, but without the appropriate methodologies according to the opinion of his students. That is to say, older professors with unattractive (perhaps outdated?) methodologies are likely to have detractors.
• Teachers who They only have promoters, having their groups average grades <6. Between 35 and 55 years old, they teach finance subjects, they are from the east of Gipuzkoa and are well valued for their knowledge. On the other hand, the feedback they offer does not seem to be enough. It can be inferred that no matter how satisfied a student is, if they are not given the feedback necessary for their improvement, they will not have notable results.
• Teachers who have mostly promoters and their minimum average grade is 7: Under 35, in strategy subjects, with a lot of knowledge and good explanations.

To go a little deeper into how to classify these teachers, let's take a look at NLP.

NLP

Like any project, there is little energy left in the final phases. And if the last thing is NLP, which is not exactly the simplest algorithm, it is difficult to reach revealing conclusions. However, with a simple count of which are the most repeated phrases (when asked about aspects to maintain) of the students based on the grade they give to the teachers and the grades they receive, we obtain the following insights.

In the case of detractors, the comments revolve around teamwork. Come on, the positive thing about the subject has been his classmates more than the class itself. On the other hand, the most repeated comments with the promoters praise the teacher: his/her knowledge, different ways of teaching, good feedback on all the work...

If we analyze the responses according to the grade obtained, the most repeated comments in the case of failures refer to the material provided. That is, the only good thing you have to say is that the subject or the PPTs are good. And in the case of the outstanding ones, many comments already appear (for the first time) about the teacher's disposition: attention, attitude, motivation, desire to help...

Let's finish: 3 big conclusions

• The teacher's way of explaining is the key element for satisfaction.
• Generally, teachers with promoters have groups with better results. Especially if they are young and perceived to have a lot of knowledge.
• To improve the results of a group, the attitude and disposition of the teacher seem to be the differential element.

They are not revealing conclusions that will win us the Nobel Prize in education, and they may not be extrapolated to other universities and contexts. But at least there is a basis to start from, and although now the complicated thing is precisely how to improve those explanations or the attitude, the teachers already know where to influence, and the students what to demand 😉

If we had to close with a final conclusion, it would be precisely the importance of attitude. It is the most repeated comment, by far, in the case of the groups with outstanding grades. It is no longer a question of whether they are more or less satisfied, but rather whether they obtain better results. And although better grades do not necessarily equate to better future development, two closing questions arise.

As teachers, or from the coordination: Are sufficient efforts made to improve the attitude and disposition of the teachers towards the students? Can bureaucracy or the student experience do more?

As students: To what extent is there an attitude towards learning? How much responsibility does the teacher have in motivating us? Wouldn't it be more logical to come from home motivated?

Líneas futuras

There is still much to do…

• A good NLP analysis, beyond counting the most repeated phrases. Lots of potential to extract value from more than 4,000 comments.
• Clustering: both the clustering done to teachers could be improved, as well as introducing new student variables to make a new clustering.
• And beyond programming, implement a “close the loop” system to take actions and decisions based on the results of the surveys. That lead to actionable projects.

Closing

In the following link of GitHub you will find the dataset and the different notebooks used in the project.

If you want to see the presentation that was made of the project, you have it here.

And finally, a thank you to the entire team at Saturdays.AI Bilbo. From counting rows in Excel we have moved on to a presentable project, nothing would have been possible without the community. Mila esker denoi!

Thanks to Rubén García Pedrejón. 

Repository

In the following repository you will find the code used to develop this application: https://github.com/SaturdaysAI/Projects/tree/master/Donostia/Donostia2021/StudentExperience-ResultsAI-main

More artificial intelligence!

The mission of Saturdays.ai is to make artificial intelligence more accessible (#ai4all) while carrying out projects for good (#ai4good). The workshops we carry out are part of the program AI 4 Schools so that anyone “learn by doing” AI regardless of their background or starting point.

Explore our AI master’s programme at https://saturdays.ai/master-ia-online/

If you want to learn more about artificial intelligence join our community through this link or visit us on our website www.saturdays.ai Join us!

Open video or resource