Selection bias is often a very destructive force when trying to determine what works and what doesn’t, it’s often impossible to evaluate if programmes are a success or not unless you’re under something extreme like a totalitarian regime.
Let’s take hospitalization for instance; if you were to ask people coming out of the hospital how their health status was, it would probably be worse for people coming out of the hospital rather than randomly selected people from the population.
So let’s assign some simple terms here then. First let’s have the treatment dummy:
So its 0 if they don’t get treated and 1 if they do get treated. Then we observe the actual outcome (health status) for each individual with Y.
So to actually measure how much of an effect treatment has we need to compare people receiving treatment to those not receiving it.
Yet the world presents problems since what we actually observe in the real world is:
So there is no one number that represents what actually happens. We can measure the, the average effect of treatment on the treated (ATT), average effect of treatment on the untreated (ATU) or the average treatment effect (ATE).
In public institutions there is this top down selection problem and the more private an institution is the more bias comes from self-selection.
The ATT can tell us the effect of going to the hospital on the people who went to the hospital. The first part of the equation is the observed part, where we measure the health status of individuals who went to the hospital had they went to the hospital. The second part is unobserved because it measures the potential health status of patients who went to the hospital had they not gone to the hospital.
ATT can show us how much people gain from going to the hospital. In the real world private enterprises are much more likely to survive if people notice that there is a positive effect and so the market eliminates a low output hospital. A government hospital on the other hand might have trouble eliminating waste because it will receive customers who might not necessarily think the hospital is any good but will still go just because it is free.
The ATU tells us the expected effect of going to the hospital on someone who did not go to the hospital. The unobserved portion is the effect of the hospital on those who did not go. Second part which we observe is the effect of the hospital on those who went to the hospital.
So it seems pretty obvious that in the real world ATU and ATT are very close to impossible to measure accurately. However what we can measure accurately is the average treatment effect. This is represented by:
To accurately measure this we must make sure to apply a RCT (randomized control trial), in other words randomly allocate if people will go to the hospital or not, and here a paradox arises. We can understand if something works if we randomly allocate it, but if we randomly allocate it we are not maximizing the use of the hospital since we are sending in healthy people. Yet if we don’t randomly allocate it we cannot observe if the hospital is working.
This also is the case with control areas if we decide to do something differently in one country/state/city and leave the rest untouched we can then compare them to see if the area where we applied the new method is better off. So what’s the next step? If they are better off then we mass produce this method to the control areas and we can no longer see if it’s working(think long term effects), and if we don’t mass produce it then those control areas are not benefiting from this new method that could be improving their standard of living.
So the paradox here is between knowing something works versus making it work for everyone. Whether we apply a randomized control trial or use control groups, chances are we are not helping out the best we can, and if we do not apply these methods then we are ignorant as to whether the program is helping people at that specific period in time. To properly understand treatment effects we need a sacrificial lamb to exist. Though what’s generally done is to assume that if something worked in the past, it will keep doing so.
diomavro 