When Russia invaded Ukraine, it committed one of the most serious crimes a nation can commit, right up there with genocide against its own people. Russia will hopefully pay dearly for all the death and destruction they have brought to the Ukrainian people.
In order to hold Russia accountable, the world needs to produce impeccable evidence on the war crimes in Ukraine, but also on what effects the war has had on Russian society itself. The latter is important, as it can help sway many who currently shrug their shoulders at the suffering in Ukraine. If it can be proven that the Russian economy is collapsing as a result of the war, then maybe the last holdouts can be won over and the Russian aggression be brought to an end.
It was for this reason that I excitedly started reading “Business Retreats and Sanctions Are Crippling the Russian Economy,” a report published by the Chief Executive Leadership Institute, CELI, at Yale University. The authors are an impressive bunch, including Jeffrey A. Sonnenfeld, Founder and President of CELI, and their research director, Steven Tian. Three more members of the CELI research staff participated, and a number of others contributed to the report.
The intent of the report is as impressive as its conclusions (p. 3):
Our team of experts, using Russian language and unconventional data sources including high frequency consumer data, cross-channel checks, releases from Russia’s international trade partners, and data mining of complex shipping data, have released one of the first comprehensive economic analyses measuring Russian current economic activity five months into the invasion, and assessing Russia’s economic outlook.
At the heart of their efforts is a database of businesses that have chosen to either stay in Russia or leave the country since the war started. This database, the epicenter of the “crippling Russian economy” report, is carefully designed and appears to be solid from every angle—except for the methodological note, which is insufficiently brief. If it had contained at least one sample showing how they compiled the data, they would have elevated the methodological standard of their report.
With that said, the database is an impressive product in itself.
The report is not. It promises an in-depth analysis of a Russian economy in collapse, but fails in every way imaginable. I hate to say this, but this publication looks more like an academic propaganda piece than a research report.
I did not want to reach this conclusion. I wanted to find irrefutable evidence that the Russian economy really has suffered deeply and perhaps irreversibly from the war. Alas, as I kept reading the report, my hopes fell flat to the ground.
The promises of exceptional data mining and production of what was going to be original statistical information, are apparently limited to the database over how many companies have left Russia, and how many have stayed. Beyond that, the report does not provide a single statistical source that would get a passing grade in a college student’s term paper (unless they attend Yale).
What stands out the most, at least initially, is the persistent reliance on proprietary sources for information. The report is practically peppered with links to newspaper articles, and an untold number of them are hidden behind paywalls. This makes it impossible for an independent analyst to reproduce their findings.
The reproducibility of research is the most basic criterion of scholarly integrity. If a purported piece of research cannot be independently reproduced, then by default we should assume that the conclusions drawn by the authors of that research are false.
This carelessness is even more glaring given that this report sets out to contribute meaningfully to the information needed to bring an end to the war in Ukraine.
On page 53, the Yale scholars try to explain that there is a massive flight of businesses from Russia. As part of their evidence, they explain:
The authors of this paper are most intimately familiar with the business flight from Russia—and are in a unique position to assess and quantify the impact of these business retreats on the Russian economy.
It is always reassuring when a first-rate university puts out a research report where the authors ask the readers to “trust us.”
The report provides a link to a paper which opens with a reference to an article from the New York Times—which is hidden behind a paywall.
And we are all very surprised.
Later, it provides a link to the database on businesses who have or have not left Russia. Again, this database is impressive and appears to be solid from a methodological viewpoint. Unfortunately, the authors of the “crippling Russian economy” report do not seem to have examined it carefully enough. As of July 29th, the database reported that 568 businesses have remained in Russia, partly or entirely, while 806 businesses had left.
The report, on the other hand, promises “the exodus of over 1,000 global companies.” That is a 25% margin of error.
In addition to the addictive use of proprietary news outlets as scholarly reference, the authors go to great lengths to make their report look as if it is empirically founded. At the same time, they consistently fail to deliver any information that the general public would be able to examine independently.
As a case in point, the authors open every section with a chart that allegedly provides data on some specific variable in the Russian economy. One example is their section on sectorial versions of the Russian consumer price index, CPI. To the left of the chart is a blue banner where one can find the word “Source.”
If it had been followed by “Eurostat” or a similar clear, singular reference, it would be a cinch for independent analysts—or “fact checkers” as they are known over at Yale—to find that data and verify or falsify it. Instead, the report obfuscates its sources by lumping them together: “Yale Chief Executive Leadership Institute, Russian Federal Service of State Statistics, Association of European Businesses, Bloomberg.”
It is impossible to determine what part of the CPI chart is derived from where, and even more so, how the authors accounted for methodological inconsistencies in the way their various sources collect and produce their data.
The report leaves a lot of other questions unanswered. For example: are all the references original producers of Russian consumer-price data? For some reason I doubt that this is the case with Bloomberg.com. But I could be wrong.
Sometimes (see, e.g., p. 51) there is no data source mentioned at all.
On page 18 we come across this shining jewel of scholarly excellence:
The share of trade in goods and services in the GDP of Russia reaching approximately 46.1 percent in 2020.
Suppose I take $80 out of my bank account to buy groceries, then add that money and the value of the groceries ($160, obviously) and then divide that sum with my monthly income.
This makes no sense, does it? Yet this is exactly what the Yale scholars just did.
This flaw in their report may seem like a mere technicality, but it is significant for two reasons. First, it fails to contribute one iota to their thesis that the Russian economy is imploding; did you know that the same ratio for the Belgian economy is 170%?
Secondly, it exposes their ignorance of the very core concept needed in order to analyze whether or not an economy is imploding. Since GDP is the very bedrock of our statistical understanding of an economy, their glaring error calls the entire report into question.
To see just how significant this error is, let us take a look at how the concept of GDP works, and how exports and imports relate to it.
There are three ways to define GDP: by spending, by income and by production value. By definition, these three measurements always add up to the exact same number. Therefore, we know that all the value produced in the economy (formally known as “value added”) is exactly equal to all the money spent.
Suppose we produce €1000 worth of goods and services in the economy. Suppose €600 worth of those goods and services are bought by households—also known as private consumption—while government buys €200, and businesses invest for €200.
The value added in production and total spending are exactly equal.
Suppose, now, that our producers export €100 worth of their production, and that this comes out of the €600 worth of consumer spending. The spending side of GDP now looks as follows: €500 worth of private consumption, €200 worth of government spending, €200 worth of business investments, and €100 worth of exports.
But suppose now that consumers still want to spend €600. There is only €500 worth of domestic production for them, so where do they go?
Imports, of course. They simply replace the spending “lost” to exports by buying foreign-made goods and services worth €100.
Imported products are booked in national accounts as produced resources right next to those produced domestically. It is like a marketplace where all the things produced are lined up on the east side, by both domestic producers and importers, and on the west side are all those who want to spend money: consumers, businesses, government agencies and foreign buyers.
We can formalize this market place as follows. On the east side we have Y, i.e., domestic production or GDP, and imports, or Z. On the west side we have consumer spending, C, business investments or I, government spending or G, foreign buyers, or exports, symbolized by X:
Y + Z = C + I + G + X
What this means, plainly, is that exports and imports belong on different sides of the GDP equation. It is downright illogical to add them together and then divide them by GDP.
How can we trust a group of researchers, whose thesis it is to prove that the Russian economy is imploding, when they do not appear to understand even the most basic concepts in macroeconomic analysis?
There is more. On page 28 they claim: “In 2021, revenue from oil exports totaled 45 percent of Russia’s budget revenue.” The source, a fact sheet from the International Energy Agency, does indeed report that number, but it is for one month only, namely January 2022, not the entire prior year, as the Yale scholars claim.
This is nothing short of intellectual fraud.
On page 34, they again try to prove a long-term trend with a one-month figure for government revenue. This time, they actually get the time reference correct—drum roll—though once again they quote a proprietary article at Bloomberg.com.
Should we even bother to point out that they cherry pick one datapoint instead of providing an independently verifiable time series, as a college student would learn to do in his principles-of-economics class?
When they take a break from statistical malpractice, the Yale scholars provide long, droning accounts of assorted trends and events in the last three decades of Russian history. While well written, not one of those odysseys adds anything of substance to their claim that the Russian economy is imploding.
Their forecasts about the future are even less favorable to their thesis. At the bottom of page 30, we learn that the Yale scholars know for a fact that “western energy companies are not substituted with Chinese or Indian partners” in Russian exports. To solidly prove that this won’t happen in the coming years, they refer to a paywall-protected article from Science Direct.
The article was published 13 years ago.
Do they teach retroactive forecasting at Yale?
I honestly can’t stomach any more of this. I really wanted these guys to show that the Russian economy is imploding. All I am left with now is one question:
What will the Russian government do with this report?
Yale Russia Report Gets an F
When Russia invaded Ukraine, it committed one of the most serious crimes a nation can commit, right up there with genocide against its own people. Russia will hopefully pay dearly for all the death and destruction they have brought to the Ukrainian people.
In order to hold Russia accountable, the world needs to produce impeccable evidence on the war crimes in Ukraine, but also on what effects the war has had on Russian society itself. The latter is important, as it can help sway many who currently shrug their shoulders at the suffering in Ukraine. If it can be proven that the Russian economy is collapsing as a result of the war, then maybe the last holdouts can be won over and the Russian aggression be brought to an end.
It was for this reason that I excitedly started reading “Business Retreats and Sanctions Are Crippling the Russian Economy,” a report published by the Chief Executive Leadership Institute, CELI, at Yale University. The authors are an impressive bunch, including Jeffrey A. Sonnenfeld, Founder and President of CELI, and their research director, Steven Tian. Three more members of the CELI research staff participated, and a number of others contributed to the report.
The intent of the report is as impressive as its conclusions (p. 3):
At the heart of their efforts is a database of businesses that have chosen to either stay in Russia or leave the country since the war started. This database, the epicenter of the “crippling Russian economy” report, is carefully designed and appears to be solid from every angle—except for the methodological note, which is insufficiently brief. If it had contained at least one sample showing how they compiled the data, they would have elevated the methodological standard of their report.
With that said, the database is an impressive product in itself.
The report is not. It promises an in-depth analysis of a Russian economy in collapse, but fails in every way imaginable. I hate to say this, but this publication looks more like an academic propaganda piece than a research report.
I did not want to reach this conclusion. I wanted to find irrefutable evidence that the Russian economy really has suffered deeply and perhaps irreversibly from the war. Alas, as I kept reading the report, my hopes fell flat to the ground.
The promises of exceptional data mining and production of what was going to be original statistical information, are apparently limited to the database over how many companies have left Russia, and how many have stayed. Beyond that, the report does not provide a single statistical source that would get a passing grade in a college student’s term paper (unless they attend Yale).
What stands out the most, at least initially, is the persistent reliance on proprietary sources for information. The report is practically peppered with links to newspaper articles, and an untold number of them are hidden behind paywalls. This makes it impossible for an independent analyst to reproduce their findings.
The reproducibility of research is the most basic criterion of scholarly integrity. If a purported piece of research cannot be independently reproduced, then by default we should assume that the conclusions drawn by the authors of that research are false.
This carelessness is even more glaring given that this report sets out to contribute meaningfully to the information needed to bring an end to the war in Ukraine.
On page 53, the Yale scholars try to explain that there is a massive flight of businesses from Russia. As part of their evidence, they explain:
It is always reassuring when a first-rate university puts out a research report where the authors ask the readers to “trust us.”
The report provides a link to a paper which opens with a reference to an article from the New York Times—which is hidden behind a paywall.
And we are all very surprised.
Later, it provides a link to the database on businesses who have or have not left Russia. Again, this database is impressive and appears to be solid from a methodological viewpoint. Unfortunately, the authors of the “crippling Russian economy” report do not seem to have examined it carefully enough. As of July 29th, the database reported that 568 businesses have remained in Russia, partly or entirely, while 806 businesses had left.
The report, on the other hand, promises “the exodus of over 1,000 global companies.” That is a 25% margin of error.
In addition to the addictive use of proprietary news outlets as scholarly reference, the authors go to great lengths to make their report look as if it is empirically founded. At the same time, they consistently fail to deliver any information that the general public would be able to examine independently.
As a case in point, the authors open every section with a chart that allegedly provides data on some specific variable in the Russian economy. One example is their section on sectorial versions of the Russian consumer price index, CPI. To the left of the chart is a blue banner where one can find the word “Source.”
If it had been followed by “Eurostat” or a similar clear, singular reference, it would be a cinch for independent analysts—or “fact checkers” as they are known over at Yale—to find that data and verify or falsify it. Instead, the report obfuscates its sources by lumping them together: “Yale Chief Executive Leadership Institute, Russian Federal Service of State Statistics, Association of European Businesses, Bloomberg.”
It is impossible to determine what part of the CPI chart is derived from where, and even more so, how the authors accounted for methodological inconsistencies in the way their various sources collect and produce their data.
The report leaves a lot of other questions unanswered. For example: are all the references original producers of Russian consumer-price data? For some reason I doubt that this is the case with Bloomberg.com. But I could be wrong.
Sometimes (see, e.g., p. 51) there is no data source mentioned at all.
On page 18 we come across this shining jewel of scholarly excellence:
Suppose I take $80 out of my bank account to buy groceries, then add that money and the value of the groceries ($160, obviously) and then divide that sum with my monthly income.
This makes no sense, does it? Yet this is exactly what the Yale scholars just did.
This flaw in their report may seem like a mere technicality, but it is significant for two reasons. First, it fails to contribute one iota to their thesis that the Russian economy is imploding; did you know that the same ratio for the Belgian economy is 170%?
Secondly, it exposes their ignorance of the very core concept needed in order to analyze whether or not an economy is imploding. Since GDP is the very bedrock of our statistical understanding of an economy, their glaring error calls the entire report into question.
To see just how significant this error is, let us take a look at how the concept of GDP works, and how exports and imports relate to it.
There are three ways to define GDP: by spending, by income and by production value. By definition, these three measurements always add up to the exact same number. Therefore, we know that all the value produced in the economy (formally known as “value added”) is exactly equal to all the money spent.
Suppose we produce €1000 worth of goods and services in the economy. Suppose €600 worth of those goods and services are bought by households—also known as private consumption—while government buys €200, and businesses invest for €200.
The value added in production and total spending are exactly equal.
Suppose, now, that our producers export €100 worth of their production, and that this comes out of the €600 worth of consumer spending. The spending side of GDP now looks as follows: €500 worth of private consumption, €200 worth of government spending, €200 worth of business investments, and €100 worth of exports.
But suppose now that consumers still want to spend €600. There is only €500 worth of domestic production for them, so where do they go?
Imports, of course. They simply replace the spending “lost” to exports by buying foreign-made goods and services worth €100.
Imported products are booked in national accounts as produced resources right next to those produced domestically. It is like a marketplace where all the things produced are lined up on the east side, by both domestic producers and importers, and on the west side are all those who want to spend money: consumers, businesses, government agencies and foreign buyers.
We can formalize this market place as follows. On the east side we have Y, i.e., domestic production or GDP, and imports, or Z. On the west side we have consumer spending, C, business investments or I, government spending or G, foreign buyers, or exports, symbolized by X:
Y + Z = C + I + G + X
What this means, plainly, is that exports and imports belong on different sides of the GDP equation. It is downright illogical to add them together and then divide them by GDP.
How can we trust a group of researchers, whose thesis it is to prove that the Russian economy is imploding, when they do not appear to understand even the most basic concepts in macroeconomic analysis?
There is more. On page 28 they claim: “In 2021, revenue from oil exports totaled 45 percent of Russia’s budget revenue.” The source, a fact sheet from the International Energy Agency, does indeed report that number, but it is for one month only, namely January 2022, not the entire prior year, as the Yale scholars claim.
This is nothing short of intellectual fraud.
On page 34, they again try to prove a long-term trend with a one-month figure for government revenue. This time, they actually get the time reference correct—drum roll—though once again they quote a proprietary article at Bloomberg.com.
Should we even bother to point out that they cherry pick one datapoint instead of providing an independently verifiable time series, as a college student would learn to do in his principles-of-economics class?
When they take a break from statistical malpractice, the Yale scholars provide long, droning accounts of assorted trends and events in the last three decades of Russian history. While well written, not one of those odysseys adds anything of substance to their claim that the Russian economy is imploding.
Their forecasts about the future are even less favorable to their thesis. At the bottom of page 30, we learn that the Yale scholars know for a fact that “western energy companies are not substituted with Chinese or Indian partners” in Russian exports. To solidly prove that this won’t happen in the coming years, they refer to a paywall-protected article from Science Direct.
The article was published 13 years ago.
Do they teach retroactive forecasting at Yale?
I honestly can’t stomach any more of this. I really wanted these guys to show that the Russian economy is imploding. All I am left with now is one question:
What will the Russian government do with this report?
READ NEXT
Guarantee of Unhappiness
Are Net Zero’s Days Numbered?
Erdogan’s Hour of Triumph