Wednesday, July 05, 2006

B-School Ranking Survey gets "F" Grade

The annual media ritual has started. Like earlier years, Business Today (July 16, 2006) has come out with the annual B-School Survey Ranking...

It is called "perception survey" based on ACNielsen ORG-MARG BEI (Brand Equity Index) model.

In 2003, when BT had decided to do away with facts and rely on perceptions - and had adapted this methodology, it had explained:

"It is based on ACNielsen's trademarked Brand Equity Model, and it involves a perceptual survey of (list of dimensions)etc.... By not depending on questionable factual information, and focusing exclusively on subjective information, the BT survey ends up being most objective of them all."
!!!?
(Yes, you read it correctly)

This year, the survey results come with a "word of caution". The survey, to quote, is

"based on the perceptions of B-school stakeholders, comprising recruiters, functional heads, MBA wannabes, MBA students and young executives... the survey works like an exit poll - the B-school ranks reveal how our 526 respondents voted on each of the 30 schools. Unfortunately, that also means some high-decibel B-School advertisers end up garnering higher salience, while some others (better schools, but not as aggressive advertisers) rank lower on popular perception..."

One surely appreciates this candidness, and the appeal to reader's rational caution.... Reminds one of the statutary warning on the cigarette packets.

This warning about the validity of results is even more appreciated, since the shortlisted Top 30 "Best" B-Schools of India also include at least two "high decibel" advertisers:

  • 1. two campuses of a ponytail B-school, which carries out full-page ads (with a small print at the bottom mentioning that it is beyond the "purview of AICTE and UGC"... or something of that nature), meaning: its degree/diploma will not be accepted as a post-graduate qualification in India or abroad (and, perhaps also to get a work visa in some of the countries).

    &

  • 2. a certain B-School in the NCR region which advertises itself as the "No-1 Private B-School" on TV, whose accredition for its MBA course was revoked last year

    ... Which perhaps also reflects how the survey was conducted:

    SAMPLE:
    The section describing the methodology of the survey highlights that this year's survey is "more ambitious than the ones before" - which is good news (someone had commented on my last year's post that the ranking is an evolving process and one should bear)

    So how ambitious?

    To quote, in the same paragaph: "While our previous survey polled 449 respondents... across eight cities, this year's tapped into a larger universe of 526 respondents in 11 cities."

    That's great!! - an increase of 77 respondants!!

    ...to assess the "perceptions" about (quoting from the same issue of the magazine):

  • 1257 B-Schools!

    The break-up of the sample is as below:
    Image and video hosting by TinyPic

    [Please note that the HR recruiters represented "a mix of managers from companies with annual revenues less than Rs 500 crore and more than Rs 500 crore"... I am sure this must be meaning something significant, but, frankly, "what?" escapes my comprehension]

    ...This will remain one of the unsolved mysteries of sampling techniques - i.e.,the perception these chosen 526 is supposed to reflect the views of a population consisting of:

  • 130,000 aspirants for CAT (and some 60,000 for XAT, and similar more thousands for MAT, etc.)!!!

    METHODOLOGY:
    About how the survey was conducted, the magazine describes:

    "In the first phase, we shortlisted the 30 B-schools that were to be ranked. How? By asking MBA wannabes and recruiters to list the B-schools they would consider applying to (in the case of former) and recruiting from (for the latter)."

    Hmm... er... Uh??

    Isn't there some circular logic here? - Or am I missing something?

    I mean, aren't these surveys meant to help the "MBA wannabes" to make a rational choice? And maybe even the "recruiters" to decide on the B-Schools to source from?

    But this survey asks them to list the B-Schools they would "consider applying to" or "recruiting from", so that it can tell them which B-Schools they should "consider applying to" or "recruiting from" !!!

    ...

    ...I guess, there is a time one should maintain a stoic calm, give up one's attempts to make sense out of glossy charts and bar-diagram... and feel grateful that in any decent B-School, such a survey will get an 'F' grade.

    To Read More:
  • "B-School Surveys" Misguide....Period!!!
  • Cont... Whither B-School Rankings!

  • 9 comments:

    ommachi said...

    But this survey asks them to list the B-Schools they would "consider applying to" or "recruiting from", so that it can tell them which B-Schools they should "consider applying to" or "recruiting from" !!!

    THAT is UNBELIEVABLE!!!!
    Such excess of stupidity is not in Nature. These guys should have worked a lot to have earned this mental state!

    On the other hand, I wonder what these guys from the magazine actually tell these 526 odd people when they do the survey. Wouldn't somebody from the 526 have asked them back some sensible questions about this rigmarole?

    ...I guess, there is a time one should maintain a stoic calm, give up one's attempts to make sense out of glossy charts and bar-diagram...

    Even if one could come out of the useless crap passed as objective information, I think one would still feel for the money one had wasted on the glossy sheets...

    and feel grateful that in any decent B-School, such a survey will get an 'F' grade.

    I sincerely hope so!

    Nice debunking. Keep up the good work.

    I am waiting to get a chance to do the same, when something like this is done for the Engineering schools of India...;)

    Arunn

    gaddeswarup said...

    I wonder whether there is any study about which sort of polls work well for which issues. Paul Seabright (in "The Company of Strangers: A Natural History of Economic Life". page 141) says that Iowa Electronic Markets so far worked better than any pollsters in predicting the American Presidential Election winners ( in terms of the number of votes). His theory is that generally people may give random replies but since people are actually betting on the election results, they have more incentive to gauge the opinions of others.

    madhukar said...

    Swarup
    I guess Paul Seabright maybe correct that when people have a stake in the outcome, they will make more informed choice. But that would be true in the case of popularity ranking or political elections. In case of B-School surveys, simply because more people vote for it, may or may not make it the "best" or better than others...

    The fallacy of the methodology is apparent, when one finds that one "Asia Business Academy, Bangalore" is listed among the top-30. To the best of my (and google's ;0) knowledge, this is the name of a department in some Finnish Univ - this said B-School does not even exit in India!!!

    Dilip D'Souza said...

    in any decent B-School, such a survey will get an 'F' grade.

    Are you sure of this, Madhukar?

    madhukar said...

    ;0)... Get the implied point, Dilip.

    that's why the qualifier, "decent":)

    Dilip D'Souza said...

    Well, further to my question Madhukar, I note that Symbiosis Institute of Business Mgmt, Pune, has put out ads (signed by their President and Principal Director) touting their ranking in this Business Today survey. So much for the survey getting an "F" grade.

    madhukar said...

    Yes, saw that ad... also noted that the "4th ank'is in considerable smaller print than the rest...

    personally, also know SIBM as a "decent" school... so note the contadiction you are pointing out... as for the reason for my post, it was this very distillation of a number - based on a faulty and misleading methodology - into an ad ("branding"), which may or may not reflect the product

    But, Dilip, there is some other oint that you are making, aren't you?

    ommachi said...

    My wait is over. There is a recent E-school ranking survey in Outlook. I have made a post on this survey here.

    Dilip D'Souza said...

    Madhukar, actually no, I'm not making any other point but this one: once these results are out, every institute named in it is going to make capital of the rankings. As Symbiosis has done. Few will have the guts to pick apart the process as you did.

    I find this whole ranking business nutso anyway. This Business Today ranking is particularly so, and I'm glad you pointed it out.