National Research Council ranks ORIE's doctoral program among the very top

You don't have to already have a Ph.D. in Operations Research to understand how the National Research Council (NRC) arrived at its ranking of ORIE's doctoral program as one of the top programs in the country—but it might help.   The NRC recently released its rankings of more than 5,000 programs in 61 fields of study at 222 universities.  As a field of graduate study at Cornell, ORIE ranks among the top Operations Research programs nationally and fares among the top fields at Cornell. It takes some digging to find out how the NRC came to this conclusion. 

In addition to developing two different overall rankings, the NRC ranked programs on the basis of research activity, student support and outcomes, and diversity.  ORIE ranks especially high in student support and outcomes and in research activity.  All of the rankings are based on data submitted in the 2005-06 academic year and do not take into account changes since then.   Programs in Operations Research were ranked together with Systems Engineering and Industrial Engineering (IE) programs.

NRC responds to criticism of past rankings

Earlier NRC rankings, such as the most recent one done in 1995, relied on surveys of the reputations of the ranked programs among faculty in the same field (Operations Research was not rated in 1995).  In response to criticism of these so-called reputational rankings, the NRC devised a new procedure.  This time they built rankings from measured data variables, such as publications per faculty member and citations per publication, time to student degree, gender and ethnic diversity, and student quantitative GRE scores—20 variables, some with subcomponents, in all.   These data were submitted by the participating universities and derived from other sources. 

Of ranks and ranges

Rather than provide simple top-to-bottom rankings, the NRC has published ranges, derived from the submitted data, for the ranking of each school and field.  The overall rank for ORIE is measured by the range from the 5th to the 95th percentile of the distribution of possible rankings based on the data.  So measured, ORIE's rank lies between #3 and #11, which turns out to be one of the top half-dozen ranges in the country for Operations Research.  

For student support and outcomes, ORIE's rank among all programs in its category is between #2 and #5, which is the highest range of any Operations Research program in the country (two or three IE programs ranked higher).  For research activity, ORIE's range is among the top seven for such programs.  ORIE rates relatively low only for diversity, an area that has strengthened since the NRC survey was taken.

Delving into NRC's methodology

Deriving a ranking from twenty variables is not easy, since experts might disagree on the relative importance, or weight, to be assigned to each variable after they are all adjusted, or 'standardized', to have a mean of 0 and standard deviation of  1.  So the NRC used two approaches to arrive at weights to apply to the variables, which they call S (for survey) and R (for regression).  [By contrast, the annual US News and World Report ranking of undergraduate institutions is based on weights assigned more or less arbitrarily by the magazine, and published along with the rankings.]

For the S weightings, the NRC surveyed a randomly selected sample of faculty in the field and asked these faculty to score the top two most important variables, the next two most important variables, and the relative importance of each of three groups into which the variables were divided.  From their responses, NRC analysts were able to come up with final weights for that field and sample of faculty.

The R weightings were arrived at through a more indirect approach that attempts to determine the relationship between the reputational rankings and the measured variables,  implicitly assuming that reputational rankings are a function of such facts about the programs.  The NRC asked the faculty sample group in each field for reputational input:  members were asked to rate, on a scale of 1 to 6, a sample of programs (typically 15) in the field.  The organization then used a statistical regression method to fit weights to the 20 data variables that best predicted the ratings given by the sample group.  The weights derived in this manner were then used to rate and rank all of the programs in the field.  In this way, rankings were developed that relied on expert input about the weights but reflected program reputations only indirectly, through the collected data.

Using simulation to cope with variability

ORIE Ph.D. alum and Senior Associate Dean for Graduate Studies and Planning at Northwestern University's McCormick School of Engineering and Applied Science Ajit Tamhane is a statistical expert who analyzed the rankings for his school.  He points out that "the data used in the calculation of ranks were subject to random variability arising from many different sources, e.g. the faculty were randomly selected and the programs they ranked on reputation were also randomly selected."  There is variability in the way faculty assigned weights and scores, and in the submitted program variables (for example, publications and citations vary from year to year). 

Asking a random set of faculty members their opinions about anything—not just the importance of specific measured variables to the ranking of a field or the reputations of departments in the field—will yield a distribution of outcomes that can be characterized by the range into which 90%, say, of outcome values fall.   Moreover, the distribution and associated range of outcomes are also subject to the random variability that has been noted by Prof. Tamhane.  Simply using the range of rankings derived from the random samples might not adequately represent the input that would have come from a full faculty about a full set of programs in the field.   So the NRC employed a simulation technique in an effort to get a more robust estimate of the distribution of outcomes. 

This technique, one of a class called "bootstrapping" (tantamount to lifting the data by its own bootstraps), entails drawing random 50% samples from the actual faculty responses, recalculating weights and scores and deriving a rank each time.  For each field, the NRC used a computer to draw 500 of these 50% samples and come up with a distribution of ranks as well as the ranges determined by the top 5th and the bottom 5th percentiles of these 500 ranks.  These are the ranges reported by the NRC.

Replicating the approach

Although the R and S rankings differed for fields at many schools, both rankings showed nearly identical results for the top schools in the Operations Research, IE, and Systems Engineering category. Phds.org has attempted to replicate the NRC approach using the NRC's raw data and displays histograms for each field at each school.  (While they used the same bootstrapping method the results perforce are not completely identical because different subsets may be drawn at random).   The histograms show that ORIE is among the top half-dozen schools in both rankings.   Phds.org also permits the user to influence the weights applied to the raw data and thereby generate customized rankings (a process that Science magazine calls "the Mr. Potato Head of graduate school rankings.") 

Appraising of the results

Publication of the NRC results has led to widespread criticism.  In some cases, data errors or inconsistencies have been identified in the submissions. University of Chicago professor of statistics Stephen M. Stigler argues that "the variables in the NRC's study actually aren't very good at making distinctions among programs, especially among programs that are clustered in the middle of the quality spectrum."

On the other hand, "we already knew that ORIE is at or near the top as a program for Ph.D. study," says ORIE Director Adrian Lewis with a wry smile, "so it is nice to have that view confirmed by such an extensive analysis—and to understand how the analysis was carried out,"