Source Notes

TheCenter Measures

Total Research Expenditures
Federal Research Expenditures
Source: NSF/SRS Survey of R&D Expenditures at Universities and Colleges, FY 2003.

Each year, the National Science Foundation (NSF) collects data from hundreds of academic institutions on expenditures for research and development in science and engineering fields and classifies them by source of funds (e.g., federal government, state and local government, industry, etc.). These data are the primary source of information on academic research and development (R&D) expenditures in the United States. Included in this survey are all activities specifically organized to produce research outcomes that are separately budgeted and accounted for. This “organized research” may be funded by an external agency or organization (“sponsored research”) or by a separately budgeted organizational unit within the institution (“university research”). This report excludes activities sponsored by external agencies that involve instruction, training (except training in research techniques, which is considered organized research), and health service, community service or extension service projects.

All Federally Funded Research Labs (FFRLs) are excluded from these academic expenditures data, including the following: Jet Propulsion Laboratory (California Institute of Technology); Los Alamos National Lab, Lawrence Livermore Lab, Lawrence Berkeley Lab (University of California); Software Engineering Institute (Carnegie Mellon); Argonne National Laboratory (University of Chicago); National Astronomy and Ionospheric Center (Cornell); Ames Laboratory (Iowa State University); Lincoln Laboratory (MIT); Plasma Physics Lab (Princeton); and Linear Accelerator Center (Stanford). The NSF data no longer classify the Applied Physics Lab (APL) at Johns Hopkins as an FFRL, but federal funds support the vast majority of research conducted there. The APL makes up nearly one-half of Johns Hopkins’ total R&D expenditures and 53 percent of its federal R&D expenditures.

While inconsistencies in reporting (known and unknown) do exist here, as in any survey of this type, problems arise mostly when one breaks out the data by source of funds. NSF expects institutions to use year-end accounting records to complete this report, and there are nationally recognized accounting guidelines for higher education institutions. However, there are also countless variations in institutional policy that determine whether the university reports a particular expenditure as coming from one source or another, or possibly not counted at all. Take federal formula funds for agriculture (e.g., Hatch-McIntire, Smith-Lever) as an example. We conducted an informal survey of the appropriate institutions in the Association of American Universities (AAU) and found that two out of eleven land grants did not include any of these federal funds in their 1997 NSF data, while others included all or some of these monies. Because these funds make up a very small percentage of the total research expenditures in any given year, the impact on our total research rankings is slight. The agriculture formula funds will have a somewhat greater, but still small, impact on the federal research rankings. NSF notes, “An increasing number of institutions have linkages with industry and foundations via subcontracts, thus complicating the identification of funding source. In addition, institutional policy may determine whether unrestricted state support is reported as state or as institutional funds.” ¹

We believe that the reporting inconsistencies in the data are relatively minor when using the total research expenditures and the federal research expenditures component. Federal and state government audits of institutional accounting make deceptive practices highly unlikely, even though these entities do not audit the NSF data directly. NSF goes to great lengths to verify the accuracy of the data, especially federal expenditure data—checking them against several other federal agencies that collect the same or similar information. In fact, all major federal agencies and their subdivisions submit data to NSF identifying research obligations to universities each year. Historically, the NSF data have tracked very closely the data reported by universities.² Further, for their National Patterns of R&D Resources series, NSF prefers to use the figures reported by the performers of the work (that is, academic institutions, industry, nonprofits) because they believe that the performers are in the best position to accurately report these expenditures.

In some sections of this report, these expenditure data are deflated to constant 1983 dollars to show real change over time. While NSF uses the Gross Domestic Price (GDP) implicit price deflator in its reports on federal trends in research, we use the Higher Education Price Index (HEPI) because of its narrower focus. Developed by Research Associates of Washington, the HEPI illustrates the effect of inflation on college and university operations.³ In contrast, the GDP implicit price deflator is based on change in the entire U.S. economy and, as noted by NSF itself, “[its] use more accurately reflects an ‘opportunity cost’ criterion [i.e., the value of R&D in terms of the amount of other goods and services that could have been spent with the same amount of money], rather than a measure of cost changes of doing research.”⁴

The research trend data always reflect the most recent published data available because NSF allows institutions to submit revised figures for up to two years. Each year, NSF reports data for the current year as well as for the previous seven years. Specifically, we use the 2003 Survey data for fiscal years (FYs) 2001-02, the 2002 Survey for FYs 1995-2000 data, and the 2001 Survey for FY 1994 data. NSF’s published nationwide totals for academic R&D expenditures will not always match the corresponding totals in this study due to NSF's sampling procedures for smaller or non-reporting institutions. In some years, rather than identifying the institutions individually, NSF provides one aggregate figure for all sampled institutions.
__________________________
1. Academic R&D Expenditures, FY 2000: Technical Notes (On-line: http://www.nsf.gov/sbe/srs/nsf02308/secta.htm )

2. National Patterns of R&D Resources, 1998: Technical Notes (On-line: http://www.nsf.gov/sbe/srs/nsf99335/appa.htm )

3. College and University Higher Education Price Index, 2005 Update, Research Associates of Washington, Washington, DC.

4. National Patterns of R&D Resources, 1998: Technical Notes (On-line: http://www.nsf.gov/sbe/srs/nsf99335/appa.htm )

Endowment Assets
Source: NACUBO Endowment Study as reported in the Chronicle of Higher Education, endowment market value as of June 30, 2004.

Institutions report the market value of their endowment assets as of June 30 to three different sources, and they quite often use three different values. For this project, we use the National Association of College and University Business Officers (NACUBO) Endowment Study because of NACUBO’s long history of reporting endowments of higher education institutions, their emphasis on using audited financial statements, and their focus on net assets (i.e., includes returns on investments and excludes investment fees and other withdrawals). NACUBO conducts its study annually and reports the results each February in the Chronicle of Higher Education.

Another source for endowment assets is the Council for Aid to Education’s (CAE) annual Voluntary Support of Education (VSE) survey, cosponsored by the Council for Advancement and Support of Education (CASE) and the National Association of Independent Schools. The VSE survey is useful as a secondary resource because it provides more single-campus data than the other two sources. For those institutions that report a system-wide total to NACUBO, we often use the VSE data to calculate a campus’ percentage contribution to the entire system, applying that factor to the NACUBO figure. In other cases, we may substitute the VSE figure when the institution indicates that this is a good data source.

The National Center for Education Statistics (NCES) Integrated Postsecondary Education Data System (IPEDS) Finance Survey also collects information on endowment assets, but these figures are often much lower than the two other sources and also are available much later. Although IPEDS instructions say to report endowment assets for “the institution and any of its foundations or affiliated organizations,” it appears that not all institutions do so.

The fact that the NACUBO study requests net assets, while IPEDS and the VSE survey request gross assets, cannot explain the large differences found in some cases. In calling various institutions, we found it very difficult to determine exactly why the numbers vary so greatly. Oftentimes, two or more individuals at an institution independently report figures for these three reports with no clear understanding of how or why the numbers differ. An examination of the 1997 endowment figures provided by these institutions showed only one university (University of North Carolina at Chapel Hill) that submitted the same figure to each of the three organizations. We discovered no consistent pattern to explain reporting variations among the institutions. This area definitely warrants more study.

Annual Giving
Source: Council for Aid to Education’s Voluntary Support of Education (VSE) Survey, FY 2004.

The Council for Aid to Education (CAE), an independent subsidiary of RAND, has produced the Voluntary Support of Education (VSE) Survey since 1986. The annual giving data include all contributions actually received during the institution’s fiscal year in the form of cash, securities, company products, and other property from alumni, non-alumni individuals, corporations, foundations, religious organizations, and other groups. Not included in the totals are public funds, earnings on investments held by the institution, and unfulfilled pledges.

CAE's VSE Data Miner service, available on-line, provides 10 years of data for all participating institutions (more than 1,600 universities and colleges). Although this is a subscription-based service and requires a user ID and password, limited access is available on-line at
[http://www.cae.org/vse].

National Academy Members
Source: National Academy of Sciences, National Academy of Engineering, and Institute of Medicine membership directories for 2004.

One of the highest honors that academic faculty can receive is membership in the National Academy of Sciences (NAS), the National Academy of Engineering (NAE), or the Institute of Medicine (IOM). All three are private, nonprofit organizations and serve as advisors to the federal government on science, technology, and medicine. Nominated and voted on by active members, newly elected members of these organizations receive life terms. Individuals elected to membership come from all sectors—academia, industry, government, and not-for-profit agencies or organizations. Member election dates are in February (NAE), April (NAS), and October (IOM).

The data collected for these rankings use active or emeritus members at their affiliated work institution, as reported in the on-line membership directories. In all cases, we were able to determine the specific campus for individual members. We re-check institutional affiliation annually to account for established members who have changed employers or whose membership is no longer active.

Faculty Awards in the Arts, Humanities, Science, Engineering, and Health
Source: Directories or web-based listings for multiple agencies or organizations.

For this category, we collect data from several prominent grant and fellowship programs in the arts, humanities, science, engineering, and health fields. Included in this measure are:

• American Council of Learned Societies (ACLS) Fellows, 2003-04
• Beckman Young Investigators, 2004
• Burroughs Wellcome Fund Career Awards, 2004
• Cottrell Scholars, 2004
• Fulbright American Scholars, 2004-05
• Getty Scholars in Residence, 2004-05
• Guggenheim Fellows, 2004
• Howard Hughes Medical Institute Investigators, 2004
• Lasker Medical Research Awards, 2004
• MacArthur Foundation Fellows, 2004
• Andrew W. Mellon Foundation Distinguished Achievement Awards, 2004
• National Endowment for the Humanities (NEH) Fellows, 2005
• National Humanities Center Fellows, 2004-05
• National Institutes of Health (NIH) MERIT (R37) FY 2004
• National Medal of Science and National Medal of Technology, 2003
• NSF CAREER awards (excluding those who are also PECASE winners), 2004
• Newberry Library Long-term Fellows, 2004-05
• Pew Scholars in Biomedicine, 2004
• Presidential Early Career Awards for Scientists and Engineers (PECASE), 2004
• Robert Wood Johnson Policy Fellows, 2004-05
• Searle Scholars, 2004
• Sloan Research Fellows, 2004
• US Secretary of Agriculture Honor Awards, 2004
• Woodrow Wilson Fellows, 2004-05

While the vast majority of these programs clearly identify a particular campus, in a few instances we used the institution’s web-based phone directory to determine the correct campus.

Doctorates Awarded
Source: NCES IPEDS Completions Survey, doctoral degrees awarded between July 1, 2003, and June 30, 2004.

Each year, universities report their degrees awarded to the NCES in the IPEDS Completions Survey. IPEDS provides straightforward instructions for reporting doctoral degrees awarded, and we do not find any inconsistencies in reporting among the universities included in our rankings. IPEDS asks each institution to identify the number of Doctor of Education, Doctor of Juridical Science, Doctor of Public Health, and Doctor of Philosophy degrees awarded between July 1 and June 30.

Most institutions in our study submit degree data by campus or offer doctoral degrees solely or primarily at the main campus.

In addition to doctorate degrees, TheCenter presents degrees awarded at other levels—associate’s, bachelor’s, master’s, and professional degrees—in the Student Characteristics table (see Data Tables: http://mup.asu.edu/AnyFed1990-2000-II/Student_Characteristics.xls).

Postdoctoral Appointees
Source:NSF/Division of Science Resource Statistics (SRS) Survey of Graduate Students and Postdoctorates in Science and Engineering, Fall 2003.

Each year, NSF and NIH collect data from all institutions offering graduate programs in any science, engineering, or health field. The Survey of Graduate Students and Postdoctorates in Science and Engineering (also called the Graduate Student Survey or GSS) reflects graduate enrollment and postdoctoral employment at the beginning of the academic year. Postdoctorates are defined in the GSS as “individuals with science and engineering PhD’s, MD’s, DDS’s or DVM’s and foreign degrees equivalent to U.S. doctorates who devote their primary effort to their own research training through research activities or study in the department under temporary appointments carrying no academic rank.” The definition excludes clinical fellows and those in medical residency training programs unless the primary purpose of their appointment is for research training under a senior mentor.

In the methodological notes for this survey,⁵ NSF indicates that it verifies the data with the institutional coordinator when dramatic year-to-year fluctuations are noted. In addition, in this data set, it is unclear whether an institution has actually reported zero postdocs or NSF has simply assigned a zero for non-response (rather than imputing by using prior-year or peer data, as described in NSF methodological notes). This year, in cases where we suspect it is not a true zero, we left the field blank.

Although each doctorate-granting campus submits data separately, NSF often aggregates them in its published reports. In all cases, we obtain the single-campus data for these schools directly from NSF.

__________________________
5. Survey Methodology: Survey of Graduate Students and Postdoctorates in Science and Engineering (On-line: http://www.nsf.gov/statistics/srvygradpostdoc/)

SAT Scores
Source: the Annual Survey of Colleges of the College Board and Data Base, 2004-2005. Copyright © 2004 College Entrance Examination Board. All rights reserved.

The College Board reports the 25th and 75th percentiles for verbal and quantitative SAT I scores for most institutions in our study. The institutions submit these data to The College Board each spring through its Annual Survey of Colleges. For our measure, we calculated the median of that range. Some institutions report the ACT instead of the SAT to The College Board. In those cases, we use a conversion table provided by The College Board to generate a comparable SAT equivalent score.⁶ When an institution submits neither an SAT nor ACT score, we substitute data from the prior year reported.
__________________________
6. Concordance Between SAT I and ACT Scores for Individual Students, Research Notes 07, June 1999 (On-line:
http://www.collegeboard.org/research/html/rn_indx.html)

Other Measures of Undergraduate Quality

National Merit and Achievement Scholars
Source: The 2003-04 National Merit Scholarship Corporation Annual Report, which reflects the 2004 freshman class.

The National Merit Scholarship Corporation (NMSC) is an independent, nonprofit organization that awards scholarships to the nation’s outstanding high school seniors based on their academic achievement, qualifying test scores, high school principal and counselor recommendations, and their activities, interests, and goals. The NMSC names approximately 16,000 National Merit Finalists each February. Of these, about one-half will receive a National Merit $2,500 Scholarship, a corporate-sponsored scholarship, or a college-sponsored scholarship.

National Achievement Scholars are selected and funded in a similar fashion and represent the nation’s outstanding African-American students. Ideally, the National Hispanic Scholars Program should also be included in this category, but it does not track the enrollment of its scholarship winners. Should it do so in the future, we will include these students in TheCenter’s data. In this study, Merit and Achievement scholarships are credited to the main campus if the NMSC Annual Report does not indicate a branch campus.

While the number of National Merit and National Achievement award winners in the entering class provides an indication of the attractiveness of a university’s undergraduate program to outstanding students, it is also an indicator that is sensitive to institutional policies on financial aid. Because the number of Merit Scholars is small, relatively small changes in institutional aid policies can have a significant impact on the number of National Merit Scholars enrolling in institutions. The average SAT score provides a broader-based and more reliable measure of overall undergraduate quality; for those reasons, we prefer the SAT scores to the number of National Merit and Achievement Scholars as an indicator of undergraduate quality.

Institutional Characteristics

Medical Schools
Source: NCES IPEDS Completions Survey, MD degrees awarded between July 1, 2003, and June 30, 2004.

Although the IPEDS Institutional Characteristics Survey does have a “medical” field that indicates whether an institution grants a medical degree, we choose not to use its data because it includes medical degrees in Veterinary Medicine, Dentistry, and other professional health-related fields. For our measure, we determined whether a particular campus awarded any MD degrees during the academic year. If the institution did not submit any data to IPEDS for that year, we then looked at whether it was accredited by the American Medical Association to determine whether the institution has a medical school.

Land Grant Institutions
Source: National Association of State Universities and Land Grant Colleges.

The first Morrill Act in 1862 appropriated federal funds for universities to provide agricultural and technical education to their citizens. A second Morrill Act in 1890 expanded eligibility to include several historically black colleges and universities, and in 1994 several Native American tribal colleges were recognized as land grant institutions. Today, there is at least one land grant institution in each state and U.S. territory and in the District of Columbia. Of the 105 institutions, most are public universities. Federal land grant institutions receive both federal and state dollars in support of their agricultural and extension activities.

While land grant status technically applies to some university systems, such as the University of California and the University of Nebraska, for our study we designate as land grant institutions only those schools that actually perform that function (e.g., UC-Berkeley, UC-Davis, UC-Riverside, Nebraska-Lincoln). In these cases, the land grant field will identify whether an institution is part of a system-wide land grant and whether the vast majority of the activity occurs on that campus. For example, UC-Davis is coded as “Yes-System” while UCLA is coded as “No-System.” We consider the 1890 institutions as land grant institutions, but we identify them separately because they do not perform extension activities.

Research Focus
NSF/SRS Survey of R&D Expenditures at Universities and Colleges, FY 2003.

In addition to reporting expenditure data by source of funds, NSF identifies in what major disciplines the money is expended. In the Research by Discipline table (Data Tables, pp. 58), we provide the proportion of total and federal expenditures in each discipline for those institutions with more than $20 million in federal research. These data are useful for developing groups of similar institutions for peer analysis.

The Institutional Characteristics table (Data Tables, pp. 154) provides a summary measure of an institution’s research strength and concentration based on these discipline-level expenditures. Universities with 95-100% of their federal research dollars spent in one particular discipline are coded as “all.” We identify institutions with 75-94% in one area as “heavy” and label those with 50-74% of their expenditures concentrated as “strong.” Other universities with 25-49% in one or more disciplines we describe as “moderate.” A few institutions (but none in the more than $20 million group) have expenditures distributed fairly evenly across the disciplines; those we code as “mixed.”

In some cases, where an institution reports as a multi-campus entity, we made adjustments to break out the discipline-level expenditure data by single campus. Typically, this involved moving all or a portion of the life sciences expenditures to the health or medical center campus. IPEDS fall enrollment and graduate degrees by discipline data also were used to help in this effort.

While these data offer some insight as to the research structure of a university, their usefulness is limited. For example, we may be tempted to use the life sciences as a surrogate for medical research, but we must remember that they also include agricultural and biological sciences. Further, the growing trend toward multidisciplinary and interdisciplinary projects may make it more difficult for universities to accurately reflect expenditures by discipline or sub-discipline. TheCenter chooses not to break out these sub-disciplines because the data are increasingly prone to error as further adjustments are made.

Student Characteristics

Fall Enrollment
Source: NCES IPEDS Fall Enrollment Survey, 2003.

Each November, institutions report their current fall headcount enrollment to the IPEDS Fall Enrollment Survey. Enrollment figures include both degree seeking and non-degree seeking students. TheCenter provides the headcount enrollment by level as presented by IPEDS, along with the percentage of those attending part-time. Graduate students include those seeking specialist degrees in engineering and education. First professional students include those seeking degrees in medical fields, such as Chiropractic, Dentistry, Medicine, Optometry, Osteopathic Medicine, Pharmacy, Podiatry, and Veterinary Medicine, as well as those seeking degrees in Law and Theology.

Each campus in our study submits enrollment data by campus, except for the few institutions identified in our Data Notes section. Because this is an informational item and not one of TheCenter’s nine quality measures, we did not attempt to adjust these figures.