Remarks Prepared for Delivery by Special Inspector General John F. Sopko at Georgetown University, Washington, DC

"Ground Truths: Honestly Assessing Reconstruction in Afghanistan"

Good afternoon. I am John Sopko, and I am thrilled to be here once again to speak at Georgetown. Accordingly, I would like to thank Dr. McNaugher, the Center for Security Studies, and the School of Foreign Service for the invitation. It is always a pleasure to be here, and I am really looking forward to discussing what I believe may be one of the most important issues facing our foreign policy and military experts, as well as many of you for generations to come.

No, it is not the Iranian nuclear deal, nor is it the diplomatic opening to Cuba. Rather, it relates to how we as a nation conduct development and reconstruction in conflict and post-conflict environments-a subject I know a little about from my last three years working on Afghanistan. This issue, I would argue, is one that will become increasingly important as our involvement in hot spots like Iraq, Syria, and Yemen escalates. Perhaps there won't be a reconstruction mission for all of those places, but I would be willing to bet that somewhere down the line, we will be involved in rebuilding places steeped in insecurity, suffering from widespread corruption, and with extremely low levels of development. If we are to have any chance for success in those endeavors, we must seek to learn and improve upon our almost decade-and-a-half spent rebuilding Afghanistan.

As Special Inspector General for Afghanistan Reconstruction-SIGAR for short-my business focuses solely on the U.S. effort to rebuild Afghanistan and basically involves checking alleged facts, digging out actual facts, and identifying weaknesses, failures, uncertainties, corrupt acts, and delusions. So you can just imagine how popular I am in this town.

Congress created SIGAR in 2008-six long years into the reconstruction effort-and endowed our small agency with a unique set of authorities. We are the only U.S. oversight body focused solely on Afghanistan, and with the authority to audit, inspect, investigate, and otherwise examine any and all aspects of reconstruction, regardless of departmental ownership. We are not a component of any other federal entity, and unlike most other IG shops around town, we are mandated to take a "whole-of-government" approach to reconstruction in Afghanistan, including U.S. coordination with foreign partners.

While independent, we are subject to Congressional scrutiny. Congressional dissatisfaction with SIGAR's early results led to my predecessor's resignation. President Obama appointed me to serve as Special IG, and I started work in July 2012. I've spent decades as a federal prosecutor, as a Capitol Hill staffer, and an attorney in a private firm. But for the breadth, challenge, and importance of the work, none of my prior jobs quite compares to SIGAR.

To carry out its work, SIGAR employs about 200 auditors, investigators, inspectors, attorneys, technicians, researchers, writers, and other staff. Most of these professionals work in our Arlington, Virginia, headquarters, but we also have around 40 staff in Afghanistan. So far, we have published 28 quarterly reports to Congress, plus hundreds of financial and performance audits, inspections, testimonies, and other reports. The end result of all this work has been well over $2 billion in identified savings for the U.S. taxpayer.

And what are we looking at, exactly? In short, the $110 billion that Congress has appropriated for Afghanistan reconstruction since 2002. To give that number some perspective, after adjusting for inflation, Afghanistan reconstruction exceeds the value of the entire Marshall Plan effort to rebuild Western Europe after World War II.[1] Even now, 13 years on and with the U.S. military presence much diminished, nearly $12 billion more in reconstruction funds wait in the financial pipeline, appropriated but not yet spent.

Right now and for years to come, SIGAR acts as America's insurance policy for the combined thirteen year war-fighting and reconstruction investment in Afghanistan, which comes close to $1 trillion. We ensure, to the best of our ability, that the entirety of that taxpayer investment does not fall victim to waste, fraud, and abuse.

The Need for Evidence-Based Policymaking

As students of foreign affairs, international security, and public policy, many of you will no doubt go on to become policy makers, or at the very least, advise policy makers one day. It is with that in mind that I want to speak with you about a concept known as "evidence-based policy-making." While this may seem like wonky, inside-the-beltway jargon, I assure you it is a really simple idea. Evidence-based policy-making is the general notion that "policymakers can achieve substantially better results by using rigorous evidence to inform decisions."[2] That's common sense, right? That decisions based upon the best possible, most scrutinized information might have a leg-up on decisions that are not.

When dealing with the massive Afghanistan reconstruction effort, one might rightly surmise that questions such as "Does the project have measurable metrics for determining outcomes?", "Do the Afghans actually want the project?" and "Can they sustain what we are giving them?" are all important in determining the success or failure of the U.S. mission to build a secure and stable Afghanistan. These are just a few of what I call "the seven questions" that I have asked agency decision makers to take into account when authorizing and implementing reconstruction projects. I ask them to consider those questions because, simply put, knowing the success or failure of a given intervention is vital to future strategic planning, allocation of resources, and, in short, policy-making. The bottom line is that if you don't have a means of knowing whether or not your programs are succeeding, the policymaker's job just became that much harder.

Now I'm not saying that the major players in the reconstruction mission-namely the Departments of Defense and State, as well as the U.S. Agency for International Development-never set baselines or other performance metrics to assess their programs. What I am saying is that greater scrutiny needs to be exercised along every step of the project management cycle: from selecting appropriate baselines, to measuring and monitoring data against those starting points, and finally in the interpretation of so-called "results."

Chances are, if I have issued a report on a project, there was a fundamental breakdown in one of those steps along the way. So, perhaps it becomes incumbent upon me to explain how to avoid such occurrences in the future.

First, before even considering where to set a baseline for a given project, it is a worthwhile exercise to stop and think about what it is you are trying to achieve. What is your end goal? Your "theory of change," to use a USAID term?

One Congressional Research Service report written on the subject of evaluating foreign aid notes that a key problem in measuring aid effectiveness is that "aid provided for development objectives is often conflated with aid provided for political and security purposes." [3] I see this all the time in Afghanistan. The general assumption, no matter which agency is providing the assistance, is that the bar is so low that any infusion of funding or technical expertise will improve the status quo.

In other words, the initial objective doesn't need to be precise, because the intervention will surely do some good in some area. While that may be true in many cases, especially in the early days just after the Taliban was removed from power, we must hold ourselves to a higher standard when billions of taxpayer dollars and the sacrifice of so many hang in the balance. We must demand precision, and we must never, ever confuse outputs for outcomes.

I am often called a naysayer, and I get complaints from agencies that I don't publish enough good news stories about their work. Besides reminding them that I'm not a cheerleader, but in the business of detecting fraud, waste, and abuse, I also find it necessary to point out that I am not even the one setting the standards for their work. They do that on their own. I am merely the person responsible for holding the agencies accountable to their own goals and standards.

Much like an umpire at a ball game, I don't make the rules, I just enforce them.

That brings me to my next point: the baselines that you choose inevitably become the standards by which you (and eventually others) will judge your work; so set realistic, measurable standards. In social science, this can be tricky. There is often little to go on in terms of determining reasonable standards, and in many cases it is often an educated guess based upon imperfect information. You are seeking to study and understand something, after all.

In a conflict-affected environment such as Afghanistan, this challenge is amplified. That said, perhaps constructing buildings to U.S. standards across the board in such an environment might be unwise, especially if we expect the Afghans to maintain and sustain what we give them. Don't mistake me here - I am not making a judgement call about what is "good enough" for the Afghans, and I am certainly not saying that the infrastructure we build for them should be inferior in any way. What I am saying is that standards vary widely from country to country, region to region, and when setting a baseline standard, one should be cognizant of the operating environment.

Lastly, policy decisions, far from existing in a vacuum, must be made based upon honestly-measured results. This is where that wonky "evidence-based policymaking" term comes in.

As President Obama said in a speech last month, the lesson from over a decade of war should be "on the front end, ask tough questions. Subject our own assumptions to evidence and analysis. Resist the conventional wisdom…Worry less about being labeled weak; worry more about getting it right." [4]

He was speaking about the need to give diplomacy a chance before hastily resorting to everyone's favorite hammer, the U.S. military. But the same words could, and should, be used to describe reconstruction policy in Afghanistan, and national security policy-making more broadly. If after 13 years and so much blood and treasure invested in Afghanistan, we cannot be honest with ourselves about our successes and failures, we are not only leaving the Afghans in a precarious position, but also putting our entire mission there at risk.

Start Off with Clear Objectives

I want to talk about one particularly important program in Afghanistan that highlights the issues I just discussed: the Department of Defense's Commander's Emergency Response Program, commonly referred to as "CERP."

Between November 2003 and June 2014, Congress appropriated about $3.7 billion for CERP, which was designed to help U.S. commanders in Afghanistan respond to urgent humanitarian relief and reconstruction requirements. Congress gave DOD broad authority to spend appropriated funds, which were not bound by procurement laws or federal procurement regulations.

CERP funding was intended for small projects costing less than $500,000 each. Projects with costs exceeding $1 million are permitted, but they require approval from the Commander of U.S. Central Command. A member of Congress once described the program to me as, "well-intentioned walking around money." [5]

In Afghanistan, CERP funds were used to implement projects in all 34 provinces, with a significant portion of these funds used in Kandahar and Helmand provinces, where the insurgency was strongest. Projects ran the gamut from infrastructure and agriculture to healthcare, education, and economic development.

According to CERP's authorizing legislation, as well as DOD's standard operating procedures for the program, CERP's purpose was to "enable U.S. military commanders to respond to urgent humanitarian relief and reconstructions requirements" in order to "immediately assist the Afghan people." [6] But if you think about it, "humanitarian relief and reconstruction" is not really an objective in and of itself. Would USAID say that its objective is to provide foreign aid? No, providing aid is what USAID does, but it does so to further U.S. foreign policy and national security objectives.

Incredibly, for the first nine years of CERP's existence, I have not been able to find a single, clearly articulated mention of the program's true objectives in any official document beyond the generic inputs of "humanitarian relief and reconstruction." In 2012, CERP guidance finally explicitly stated "desired effects" of the program in two main thematic categories: counterinsurgency and economic development. Prior to this, CERP guidance listed only categories of projects such as education, health, agriculture, and others, but did not identify explicit goals of the program itself.

Later guidance managed to almost completely eliminate the counterinsurgency objectives altogether, pulling them out of the body text, and exiling them to the annex of the 270 page document. The introduction of this same guidance states that "commanders should be executing projects resulting in measurable effects that support counterinsurgency objectives."[7] So, according to DOD, commanders should execute projects which can be measured according to objectives that are not openly stated.

And we wonder why there is confusion about CERP's effectiveness. When you have such a diversity of goals-some implicit, some explicit, and some which blatantly contradict each other-it becomes impossible to tell which goal dominates, and for oversight bodies, the goal for which people should be held accountable.

Part of the problem is the baseline assumption-without any actual grounding in data-that the program is helping to begin with. To commanders, the degree to which it is helping doesn't really matter as long as it is providing some utility for their troops. The military is not really in the business of calculating return on investment for taxpayer funding. It is in the business of winning engagements, and doing so with the fewest number of casualties.

Because this program was initially designed without explicitly-stated objectives and stringent performance metrics, it largely lends itself to anecdotal "evidence" by small-unit commanders who view the program as another tool in their tactical kit - much like a weapons system or other force-multiplier. Therefore, it should surprise no one that the sustainability of CERP-funded schools have been singled out because, say, they are literally crumbling and falling apart.[8] Or that CERP-funded hospitals actually became unsafe because their electrical and water supply systems failed.[9] The overarching objective was never to build a school or hospital, staff it, supply it, and otherwise maintain it in order to improve education or health in Afghanistan. The goal was to win a war, and to do so at the lowest human and materiel cost possible.

So what about that anecdotal 'evidence' about CERP's successes?-how it has saved the lives of our troops, built trust where it was lacking, and furthered the security mission in Afghanistan. It probably has. But I can't verify that. No one can. More importantly, I can't quantify how many lives CERP has saved, and to what extent the program furthered our security objectives. That type of attribution would be difficult to ascertain under great conditions. But when I can't even tell what the program's basic objectives were, I certainly can't tell you whether it's succeeding or not.

Set Realistic Standards

I am often criticized for focusing too much on the doom and gloom of Afghanistan reconstruction at the expense of the so-called "good news." In this regard, my critics are correct: I don't spend my time looking for good news stories to tell. That is simply not my job. My job is to find out what is going wrong, and to try and get massive government agencies-all of which have been around for decades, sometimes centuries, before my small organization even existed, and which are deeply set in their ways-to do better by the taxpayer.

As I have said before, I don't make the rules, I just enforce them. It is the agencies who are responsible for defining the criteria by which their programs will be evaluated.

As we have seen, it becomes really difficult for SIGAR to assess reconstruction projects and programs if agencies don't set clear criteria or project management standards. An equally frustrating, and unbelievably baffling, occurrence is when the agencies do not take the time to think about appropriate standards for their projects. I'll give you a few examples.

A few years back, USAID spent almost $15 million to build a hospital in Gardez, the provincial capital of Paktiya province. The effort was just one small part of a $57 million dollar contract awarded to build health and education facilities throughout Afghanistan, which would then be turned over to the Afghans to operate. The hospital hit a few snags in the building process, and it was delayed for several months, but other than that, it was a solidly-built structure that ended up being 12 times larger than the previous hospital in Gardez, a headline that should have meant a huge increase in local healthcare capacity. There was just one problem: USAID did not fully assess the Afghan Ministry of Public Health's ability to operate and maintain the hospital once completed. In the end, USAID managed to increase the cost for the Afghans to operate a hospital in Gardez by a factor of five.

I wish I could say that this is an isolated incident, but this sort of thing happens in Afghanistan all the time. You would think after 13 years of these types of occurrences, and hundreds of cases of SIGAR pointing these problems out, that someone would wake up, look around, and say "You know what folks? Maybe we're going about this all wrong." It seems that time and again, people have to be reminded that Afghanistan is not Kansas.

I want to give people the benefit of the doubt on this kind of thing. Part of me is saying that perhaps these officials just want the Afghans to have high-quality infrastructure. I try to remind myself of that when we build multi-billion dollar roads to U.S. weight standards in a country that has no ability to enforce weight limitations.[10] Of course the roads are crumbling and of course the Afghans don't have the funds and capabilities to maintain them. Or when a military official suggested that we spend millions building high-tech bus stops in Afghanistan, complete with solar-powered lighting, as if we were in Bethesda. I want to believe that these folks just want the Afghans to benefit from the same high quality infrastructure that we enjoy in the U.S. But, if I am doing my job and being the good umpire that I was appointed to be, all I am seeing is a modus operandi that is woefully out of touch at best, and delusional at worst.

We simply must be smarter. We have to be, if our country is going to play its critical role building peace and stabilizing war-torn societies for years to come.

Honesty is, quite literally, the best policy

Two and a half years ago, I sent the main players in the reconstruction game-the Departments of State and Defense, as well as USAID-a letter requesting that they identify, by their own judgement, their ten most and least successful reconstruction programs, and why they selected those programs. I still have not received a straight answer from any of them.

The agencies were not shy about pointing out what they thought they were doing right in Afghanistan, although even this information suffered from measurement and attribution problems. However, none of the agencies identified their top programmatic shortcomings. Instead, they identified Afghanistan's shortcomings, problems, and challenges-like corruption, poor rule of law, and lack of capacity-and chose to reiterate all the great things they were doing to address those issues. This is akin to saying that you "work too hard" or you "care too much" when a prospective employer asks you to describe your greatest weakness in a job interview. For those of you preparing to enter the job market, trust me, don't do that. It's disingenuous and at the very least it shows a lack of self-awareness.

As the old adage goes, "the first step to solving a problem is admitting that you have one." I would honestly fault these agencies less if they gave me a straight response and said, "John, we think that Program X is not doing so hot because of Y, and we believe that because we put in place a system to measure its progress." At least then, I know they have a basic level of recognition that something isn't quite right, and that they are looking to do something about it. By sending me a letter that is little more than a PR exercise, I can logically only assume one of two things: they are intentionally dodging my questions, which is unsettling, or they are not being honest with themselves, which is worrying, given the vital task with which they have been charged.

A USAID official even told me later that asking him to identify his agency's top successes and failures was like asking him to choose which of his children he loved more. Who knew that Afghanistan reconstruction is the Sophie's Choice of international development? I was not impressed with this answer, to say the least, and I'll tell you why.

Program evaluation inevitably entails or at least facilitates comparisons of projects. If not, what basis would agency managers have for deciding-say, in the face of budget cuts, sequestrations, or new mission directives-which projects to prioritize, expand, contract, terminate, transfer, or redesign? How do they decide which project managers deserve greater responsibility or career advancement, or the obverse, without comparing outcomes? How do they capture lessons learned to improve agency performance without making comparisons? Even if a formal process of comparing program or project outcomes does not exist within these agencies, I do not think it is unreasonable to ask for a limited, judgmental comparison of successes and failures as a general sanity check in what has become quite possibly the largest reconstruction effort in human history.

Comparing outcomes is, in addition to being good practice for managers and part of the job for inspectors general, the subject of formal guidance for executive branch departments and agencies. In May 2012, the Office of Management and Budget issued guidance on the "Use of Evidence and Evaluation in the 2014 Budget." That document said, in part:

Agencies are encouraged to include measurement of costs and costs per outcome as part of the routine reporting of funded programs to allow for useful comparison of cost effectiveness across programs. … Once evidence-based programs have been identified, such a [return-on-investment] analysis can improve agency resource allocation and inform public understanding.[11]

Once again, evidence-based decision-making rears its head. It's a relatively simple concept, and one that we really must employ if we expect improvement in an exceedingly complex and increasingly difficult environment such as Afghanistan. Getting things right there will largely set the tone for future stabilization and post-conflict reconstruction missions.

Looking to the Future

So, here we are. Almost fourteen years into our trillion dollar effort, with over 2,000 American lives sacrificed. At the risk of sounding dramatic, if we can't honestly point to some actual, measurable accomplishments from that massive investment, we will miss out on a crucial learning opportunity that will affect U.S. foreign policy for generations to come. In short, we risk failing to understand the conditions necessary not only to produce peace and prosperity, but to sustain them. That is the bigger picture in play for the United States in the conflict-affected environments of tomorrow.

We must not kid ourselves about Afghanistan. It will be a long struggle. Defeating a determined insurgency, improving health and education, altering attitudes toward women, reducing corruption, and building governmental competence are not casual, short-term undertakings. Indeed, in the realm of international security today, hardly anything is as clear-cut and easy as it initially appears.

In a radio broadcast just one year before the Nazis invaded Poland and World War II began, Neville Chamberlain famously opined, "How horrible, fantastic, incredible it is that we should be digging trenches and trying on gas-masks here because of a quarrel in a far-away country between people of whom we know nothing."[12] Today, Chamberlain's folly at Munich is a world famous cautionary tale for would-be appeasers, and the events that set in motion the deadliest war in modern history have informed international relations theory and leadership decisions to this day.

From Rwanda and Kosovo, to Libya, Iraq, Syria, and Afghanistan, world leaders have recognized the dangers of underestimating crises in little-known places far away. What has been less apparent, unfortunately, is the need for sustained interest in rebuilding the political and economic structures of war-torn nations. Still rarer is the recognition that this rebuilding process requires honest assessment and reassessment of successes and failures, follies and triumphs.

General William Westmoreland, the American military commander during the Vietnam War famously said after the war, "militarily, you must remember that we succeeded in Vietnam. We won every engagement we were involved in."[13] Today, I doubt many people, including U.S. military commanders who served in Vietnam, would call the Vietnam War a "success" for the United States. But Westmoreland wasn't wrong, based upon the criteria he was using to judge success. He just chose to highlight progress against baselines that would tell his story the way he wanted it told.

Without such candid introspection, we will surely be doomed to repeat our mistakes in the far afield places of tomorrow. It is for that reason that SIGAR continues its work in Afghanistan: poking and prodding U.S. agencies to set smart goals, collect better data, and make informed decisions-in short, to do better. We welcome your interest and support in that mission, as I welcome your comments and questions. Thank you.