Using Big Data to Assess Student Lab Skills in Labflow
The teaching laboratory is an important component of a comprehensive science education program in chemistry, biology, and physics. In colleges and universities, there is a strong cultural belief that students cannot become skilled scientists by only learning the theory of the discipline: it’s important for them to also get experience with the hands-on investigation and inquiry of the science as well.
However, behind this shared sentiment, instructors often differ in whether they see the lab as merely enriching a lecture course or whether lab courses stand on their own because of their focus on introducing novel scientific ideas related to experimentation, measurement, and evidenced-based reasoning. This latter perspective has led to a movement in education research toward better understanding and assessing students’ laboratory skills and techniques in addition to their conceptual knowledge of a discipline.
Insight Into Developing Experimental Technique
There are many lab skills that are important for students to master. They include engaging in the scientific process and developing critical thinking skills, communication skills, and mastery of laboratory techniques. One of the strengths of Labflow is the resolution we have on particular lab skills related to experimental techniques.
For example, data collection is built into many Labflow lab reports. Students record every value from an experimental procedure and get feedback in real time about whether that value is reported to the correct precision of the lab equipment specified in the experiment. In addition to serving as a record of student data, Labflow also maintains a trace of every value that students attempted to report for a specific measurement until their final value was accepted. This gives you as an instructor valuable perspective into a student’s thinking as lab-specific skills around measurement and instrumentation develop. Trace data like this also make visible the common mistakes that students make as they learn in the lab. The following figure shows an example of the trace data a student generated while attempting to enter a volume value for a trial in the Glassware, Techniques, and Measurements OER report.
One exciting area of research at Catalyst Education is our work on transforming trace data into proxy measures of student understanding of specific lab skills and techniques. We have found that we can use trace data to highlight common measurement mistakes that students make—the data is so high-resolution that it’s possible to see an understanding of lab techniques develop on a moment-to-moment basis. Our approach quantifies how long it takes students to enter valid measurements and, most interestingly, how many errors they make across repeated trials.
The analysis we’ve done to date suggests that students learn quickly from feedback embedded in Labflow reports, which directly improves the number of tries it takes them to enter valid measurements. Students become more adept both through performing multiple trials of a measurement and, compellingly, they are able to transfer their understanding of concepts like precision and accuracy between two types of equipment. In the following graph, you can see that students were more error-prone when using graduated cylinders (first in the procedure) than with the buret (second in the procedure).
Validating and Extending This Approach
This research is quickly evolving, and we hope to continue our investigation by looking for evidence that these process metrics change as students move through their course. Future work will also establish whether these process measures are good proxies for student lab techniques through both qualitative work (e.g., student interviews) and triangulation (e.g., correlation with established assessment techniques).
To learn more about how we use student data from Labflow to track laboratory skill development, check out my on-demand presentation from this year’s Biennial Conference on Chemical Education (BCCE).