Frequently Asked Questions about Class Experiments

Why should I use economics experiments to teach?

How should I reward student participation in experiments?

How many experiments should I conduct in a semester?

What if the results of an experiment are different than expected?

How do I make sure an experiment runs well?

How do I discuss the results of an experiment?

Where should I conduct the experiments?

Should I run computerized or hand-run experiments?

Do I need IRB approval to conduct class experiments?

Can I run experiments in a large lecture class?

Should an experiment come before or after a lecture on a topic?

Why should I use economics experiments to teach?

To begin with, there is evidence that the use of economics experiments actually improve student performance (Emerson and Taylor, 2004; Durham, McKinnon, and Schulman, 2007).  Because students are actively involved in the experiment, the topic is more real to them, and they have a deeper level of understanding.  By combining the experiment with guided discussion, students frequently discover some of the economic principles on their own, which further increases learning.

In addition, students almost always enjoy the activity (especially when it substitutes a lecture!)  Many are more interested in the topic after participating in an experiment (which many view as a "game") and are more likely to participate in the follow-up discsussion.

How should I reward student participation in experiments?

There are several approaches to this, and little agreement about what is the "best" way to reward participation.  Here is an overview of your options:

Hypothetical: Some instructors use only hypothetical rewards (Arlington Williams sometimes tells his students that they are playing for "oodles of utils").  One advantage of this approach (beyond saving the expense of cash payments) is that one does not have to record and verify experiment earnings for future payment.  However, others are concerned that students will not think about the experiment as carefully if the rewards are hypothetical.  In addition, some may be concerned that the desired (or typical) results are not as reliably attained with hypothetical payments.  Students also have a tendency to suggest that they may have made different decisions (leading to different aggregate results) if they had been paid.  Because of this, if hypothetical rewards are used, you should emphasize that you want students to think about what they would actually do if they were being paid in cash.

Market experiments can generally be counted on to produce equilibrium outcomes, even with hypothetical incentives, although it may take longer for the market to converge.  There is some concern that real incentives may be required in settings that involve externalities or strategic behavior.  However, some instructors have used many of the modules on our Teaching page using only hypothetical rewards with consistent results.

Extra Credit: Experiment earnings may be converted into extra credit points.  The conversion may be based on absolute earnings (for example, 1 extra credit point for every dollar earned in an experiment) or based on one's earnings relative to a benchmark (for example, your earnings relative to the maximum possible earnings in the experiment or your earnings relative to those with similar earnings opportunities - for example, buyers in a market with the same values).

Extra credit incentives typically encourage students to take the experiment seriously, and there is some evidence that results from experiments using extra credit rewards are consistent with those using cash payments.

However, some instructors worry about using extra credit rewards in settings where one person's decisions has a direct impact on another's earnings.  For example, in a public goods experiment, one's contribution decision affects their own earnings and also the earnings of others in the group.

If you "curve" the final course grade, we suggest adding any extra credit rewards AFTER the curve has been determined.

Cash: Depending on the instructor's budget, one can pay all students their full earnings, pay all a fraction of earnings, or randomly select one (or more) students to pay.  Some instructors establish a lab fee at the beginning of the semester to fund these payments.

If you plan to use cash payments, you should be careful about how you set up the parameters of the experiment so that it does not wind up costing a lot more than you expect.  For example, subjects sometimes make mistakes in a market experiment that lead them to lose money; in one case, a buyer with a value of $15 intended to submit a bid of $10 but typed an extra "0" and entered a $100 bid instead.  A seller quickly sold to this buyer; the buyer therefore incurred a loss of $85 (the difference between the $100 price paid and his value of $15), and the seller earned a correspondingly large profit.  With cash payments, the instructor could not have collected $85 from this buyer, but the seller would have liked to have been paid cash for this profit.

Other Considerations: You can also use more than one of thes methods of rewarding subjects.  For example, if there is an experiment in which you think cash payments are especially important you could use them, but use extra credit or hypothetical rewards for others.  Some instructors prefer to use hypothetical incentives for any experiment conducted during class, but use extra credit rewards for any that are done outside of class.

How many experiments should I conduct in a semester?

While there is no exact answer to this question, it is probably best to not schedule too many experiments in a course if you have never run any before.  Once you begin to use class experiments on your own, you will develop a better feel for how many you want to do, and you will also become more experienced and comfortable with running more experiments.

What if the results of an experiment are different than expected?

In our teaching modules, we present "typical" results from classroom experiments.  For many types of experiments, results are quite consistent and so you can reliably expect to have similar results.  However, sometimes you will see results that differ from what is expected.  There is nothing wrong with this, and you can use it to your advantage to get students involved in the discussion.  One approach is to present students with the unexpected results along with more typical results, and then discuss possible explanations for why the two data sets are different.  Care should be taken that students don't feel that they made a mistake as they are should be encouraged to think about the reasons why they made their decisions, and why students in other classes may have made different decisions.

Of course, you will need to be prepared with a typical data set (such as those described in the teaching modules) in order to best use this approach.

 

How do I make sure an experiment runs well?

While prior experience is usually the best way to insure experiments run well (just like with your lectures), there are several things you can do to make things run more smoothly:

1. You need to spend some time understanding a new experiment before you ever use it in your class – this can not be overemphasized.  If possible, do a trial run outside of class with a few assistants.  This will allow you to understand the mechanics of the experiment, as well as make sure that everything is clear to the students when they actually do participate. If you plan to run a computerized experiment, set up the experiment and then test it using a couple of computers or by opening extra web browsers on your own computer.

2. Before your experiment, make sure you give your students any needed information about the experiment.  This may include instructions you want them to read in advance, or it could be details about where and when and how to participate.  Often, you will have a better feel for what information you need to give to your students after you have followed the suggestion in the prior comment concerning your test run of the experiment.

How do I discuss the results of an experiment?

The post-experiment discussion is the time when you really can interact with your students, and you will find many students become quite engaged in the discussion.  You should not be too willing to give away the story of the experiment by simply lecturing about the results.  Rather, allow the students to discover and articulate to the class the central ideas of the experiment, while you help your students to articulate their thoughts about the experiment.  You should have a set of key discussion points about the experiment, and redirect the discussion if necessary to hit all of your key points.

For examples, see our teaching modules.  Many of them offer a plan that includes questions and typical student responses.

Where should I conduct the experiments?

Classroom:  There are many hand-run experiments that can be conducted in any classroom.

Computer Lab: If you have a computer lab available, you can conduct the computerized experiments during class-time in the lab.

Out of Class: Our software was designed to allow you to conduct computerized experiments outside of classtime.  Students can log on wherever they have an internet connection (home, coffee shop, computer lab, etc.) and participate in the experiment.  For many experiments, this requires that a group of students log in at the same time to participate (for example, you might decide to conduct a market experiment and tell your students to log in at 7pm; if students try to log in after the experiment has started they will not be allowed to join the experiment).  For large classes, this may mean that you need to schedule more than one experiment to accommodate everyone's schedule.  Some instructors use these computerized experiments when they must be out of town.  They instruct students to log-in at class time; the instructor can log-on remotely and start the experiment at this time.

Some instructors prefer to run experiments outside of classtime, and saving the regular class period for discussing the results of the experiment.  They feel like this enables them to use these active-learning exercises without giving up valuable class time.  Others prefer to conduct them in class.  One colleague stated that she likes to run experiments in class because they have the "wow" factor; she describes this as the point in the experiment when students start to really understand the incentives, and feels that discussion right after the experiment (as opposed to a day or more later if the experiment is run outsidde of class) maximizes the impact.

Should I run computerized or hand-run experiments?

There is no simple answer to this question.  Some instructors use only hand-run experiments, while others use only computerized versions, and still others use a combination.  That being said, there are factors which will influence your own decision on this matter.

With hand-run experiments, you don't have to worry about having special computer equipment, internet connections, and figuring out how to set up the software to conduct the experiment.  In addition, some instructors enjoy the direct interaction with students that occurs with a hand-run experiment (for example, having students shout out bids and offers in a market experiment).

With a small class, there are a wide variety of hand-run experiments that you can conduct, including market experiments, public goods, and game-theory experiments.  In larger classes, it is harder to conduct hand-run experiments, but our teaching modules give some suggestions for larger classes.  We also have an entry about this in our FAQ.

Computerized experiments make record-keeping easier, and allow the instructor to present results to students much more quickly and easily.  Also, because the computer collects, collates, and organizes all of the data, these experiments typically can be conducted more quickly than hand-run experiments.  In market experiments, for example, this means that you can conduct experiments with more rounds; this gives makes it easier for the market to converge to equilibrium and also allows you to add extra features to the experiment such as taxes or price controls.

Most instructors who use classroom experiments start with hand-run experiments conducted during class time.  As you become more comfortable with classroom experiments you may want to try one or more computerized experiments.

Do I need IRB approval to conduct class experiments?

Many instructors think that you need to get approval from your University's Institutional Review Board only if you use cash rewards in your classroom experiments.

In fact, it really depends on your purpose for conducting the experiment and what you plan to do with the results.

According to Federal guidelines, IRB approval is required for any research involving human subjects, where reserach is defined as a systematic investigation designed to contribute to generalizable knowledge.

In other words, if you are using the experiment in a class, but will not use the results of the experiment in any other setting, this is not research and therefore does not require IRB approval.  On the other hand, if you will use the results of the experiment when you make a presentation or to write a paper, then this is research and must be approved by your IRB.  It does not matter whether or not you pay your subjects.

Can I run experiments in a large lecture class?

Absolutely!  While some experiments are more manageable in small classes, there are a lot of experiments you can conduct in large classes.

One option is to conduct computerized experiments outside of classtime.

In addition, some of our teaching modules give suggestions for adapting experiments for use in large classes.

One technique that some instructors use is to put the experiment instructions online a day or two before class.  You can tell your students that they must print out the instructions, read them,  and bring them to class in order to participate in the experiment.  This has the added advantage that you can go through the instructions more quickly if students have read them before class.  Any students who do not bring instructions are told to sit in the back rows and observe the experiment, while those with instructions sit in the front and participate.  Of course, in this case not all students gain the benefit of participating, all observe the experiment and can take part in the follow-up discussion.

Should an experiment come before or after a lecture on a topic?

Most instructors conduct an experiment before they lecture on the topic covered by the experiment.  This has two advantages:

1. It avoids giving away the expected outcome before students participate, and allows them to discover the economic principles for themselves.
2. It can save class time; the experiment may reduce the amount of time you have to spend on the lecture, and so you can let the class discussion guide you in terms of how much additional lecture material you need.