Social security expenditure in Japan is increasing, and the budget that can be freely allocated by the government is declining. Against the background of an increasing necessity for the establishment of priorities in budget allocation, EBPM possesses considerable significance in helping us to identify effective policies based on evidence and focus resources on those policies as a matter of priority. In order to make EBPM effective, it will be essential to create mechanisms allowing policy formation on the basis of evidence. Nevertheless, it can also be seen that EBPM – the concept of policymaking based on statistical analysis – is not functioning in practice in Japan. Improvements must be made in order to enable evidence to have a greater impact on policymaking.

For example, in the tripartite councils (government, labor and employers) that decide policy in the Ministry of Health, Labour and Welfare, the ministry maintains a neutral stance as the Secretariat and attempts to find points of common ground between the respective claims of the labor side and the employers’ groups. The traditional culture of the Ministry is to consider the consistency of these points of common ground with the law, and to formulate policy on this basis. I believe that the effectiveness of policy has been maintained through the formulation of policies that satisfy the three parties to the process, but this policymaking mechanism tends to have a low level of regard for evidence. A survey is made of the status of wage revisions in order to provide evidence when determining the degree of increase in the minimum wage. The survey method is decided on by the council that determines the minimum wage, but the issue is that the evidence is neglected in decisions on policy and changes in the industrial structure are not reflected. The problem is that the generation of evidence is subordinated to the decision on policy.

It will be necessary to make the transition to a system in which the departments within Japan’s government ministries and agencies responsible for the generation and analysis of statistics conduct independent analyses and compile evidence, and that evidence is reflected that evidence in the policymaking process. The utilization of evidence in policy decisions would provide an incentive for the further promotion of EBPM, and attract superior human resources to ministerial departments that oversee statistical analysis. If such an incentive did exist, it would be necessary to give serious consideration to how to make use of administrative data while protecting personal information. If we do not design incentives for the use of evidence, even if we cultivate human resources and prepare data, efforts will simply end without bearing fruit. The fact that despite the enormous amount of administrative data that have been collected, their use as statistics has not advanced, is the result of systems that make it difficult to provide incentives for the use of these data by the responsible ministries and agencies.

EBPM represents an effort to select better policy measures while examining the causal relationship between measures and outcomes. For example, let us assume that the government is conducting a remedial education program to help children in poor households. In this case, the outcomes consist of academic achievement and enrollment rates; the basis for the causal relationship that indicates whether or not the program has improved the outcomes is evidence, and it is EBPM that allows us to refer to evidence in order to select better measures. In many cases, there are a number of potential measures that might be implemented in order to achieve specific outcomes. The significance of EBPM is that it enables us to clarify the causal effects of the measures, and to redistribute resources such as taxes and human resources to cost-effective measures that can be expected to produce outcomes.

One method of identifying the causal effects of policy measures is to compare outcomes for groups for which the policy has been implemented with those for groups for which the policy has not been implemented. Here, the important thing is how to assign the groups. In order to evaluate the outcomes of the remedial education program mentioned above, it would be necessary to eliminate factors other than the measures implemented, such as the degree of parental enthusiasm for education and the children’s own degree of motivation. In the case of a policy that provided such a program to families who apply directly, there is a risk that there may, for example, be a bias in the subjects of the policy towards households that are passionate about education. In such cases, “randomized controlled trials” are a very useful tool. This method randomly assigns those to whom the policy is applied and those to whom it is not, in order to prevent a bias towards specific attributes of the subjects, making it possible to identify the causal effects of the policy by calculating the difference in outcomes between the two groups.

Because Japan’s initiatives in the area of EBPM commenced with improving the accuracy of statistics, there is a tendency towards the mistaken view that this is equivalent to EBPM. The essence of EBPM, however, is making decisions while examining the causal effects of policy measures. Elements such as the formulation of statistics, obtaining a grasp of the facts, and creating logic models are important preparatory procedures for promoting EBPM, but in Japan excessive effort tends to be focused on these areas. It will be necessary to return to the original meaning of EBPM and examine the causal effects of policies.

There are limitations to the promotion of EBPM within the government alone. It would be wise to link resources possessed by the government with external resources and accumulate concrete case studies, beginning from small-scale and simple cases.

Dai Tamesue, a former track and field athlete who has represented Japan in three Olympics, once wrote an essay entitled “A Leading Athlete’s Flat Feet.” He refers in this essay to a famous track and field athlete who is flatfooted. Hearing about such cases, many people would believe that people with flat feet are perhaps faster runners. However, as Tamesue indicates, saying that people who run fast are flatfooted is different to saying that if a person is flatfooted, that person is a fast runner. He also adds that there is a tendency to develop superstitions in such areas, relating the result for example to the specific characteristics of a prodigiously talented athlete or to the specific sequence of actions prior to the good outcome. In fact, it appears that the majority of athletes taking part in the Olympics are not flatfooted.

This helps us to understand why EBPM is important. Just because people who run fast are flatfooted (correlation) does not imply that because a person is flatfooted that person will be a fast runner (causation). If excessive confidence is placed in cases for which only correlation has been verified and measures are implemented throughout society on this basis, the expected outcomes may not be achieved, or opposite effects may even occur. As an example, the lesson offered by the documentary “Scared Straight” is a significant one. Filmed in 1978 and the recipient of an Academy Award, Scared Straight showed a group of juvenile delinquents experiencing a three-hour session with convicted criminals in jail. Following this, the same group was filmed again in 1980, 1987 and 1999. It was found that all of the individuals who had participated in the session with the convicts had been rehabilitated, and there was no evidence that they had been involved in serious crimes. Following this, the number of state administrations in the US adopting Scared Straight as a program for the rehabilitation of juvenile delinquents increased.

However, researchers later conducted studies in which youths were randomly assigned to participate or not participate in a similar rehabilitation program, and their follow-up research showed that the youths who had participated in the program were more likely than those who had not participated to commit crimes and be arrested. Policy evaluations of this type are called “randomized controlled trials.” Such trials are well known as methods of determining whether a relationship is causal rather than correlational. It is desirable for EBPM to adopt rigorous methods of this type in order to clarify the nature of causal relationships. Young people of school age change quickly in terms of development and emotional state, and in such cases it is an easy matter to mistake changes that would have occurred even if the policy had not been implemented for outcomes of the policy. Rigorous policy evaluation is necessary in order to prevent the implementation of the policy from having the opposite effect to the intended effect, as in the case of Scared Straight.

（Submitted manuscript）

It is sometimes believed that Evidence-Based Policymaking (EBPM) will produce a dramatic improvement in the quality of public policymaking. This expectation is unrealistic. Research evidence can certainly improve policymaking, but cannot guarantee to do this.

Public policymaking should, of course, be informed by evidence. But the concept of EBPM restricts evidence to research evidence, and often to that from systematic reviews of randomised controlled trials (RCTs). Furthermore, this evidence is taken to show ‘what works’, and what does not; and thereby to tell policymakers what to do. There are several problems with this:

1. It involves misleading assumptions about the policymaking process. Likely effectiveness of policies is not the only relevant consideration that policymakers must take into account. Also relevant, for example, are whether a policy will be politically defensible; whether it entails significant costs compared to alternatives; and whether it will have undesirable longer-term effects on the attitudes and behaviour of those who must implement or respond to it.

2. It underestimates the value of practical experience, and the need for practical judgment in policymaking.

3. It overestimates the likely validity of evidence from RCTs. While certainly a powerful research method, an RCT cannot control all potential threats to validity. For instance, in drug trials it is often possible to standardise the treatment, and measure outcomes accurately, but these goals are much more difficult, if not impossible, to achieve in testing social and educational policies. Also, RCTs provide little information about contextual variation in the effectiveness of policies; but this is crucial because the implementation context of a policy will often be significantly different from the context(s) in which the evidence was produced.

4. It underestimates the value and validity of evidence produced by other kinds of research, including that using qualitative methods. This can provide in-depth understanding of the assumptions on which a policy is based, some of which may be open to question; and it can offer essential knowledge about the perspectives and practices of those who are to implement the policy, and those affected by it.

Over time, the concept of EBPM has been liberalised, for example by requiring only that policy be informed by, rather than based on, evidence, and by broadening the kinds of evidence deemed acceptable. This liberalisation is beneficial: in particular, it acknowledges the limits to the role that research evidence can play in policymaking. However, it undercuts the novelty of EBPM, and many of the claims often made on its behalf.

（Submitted manuscript）

The Japanese government decided on a policy of promoting EBPM at a Cabinet meeting in 2017, and we are now at the stage at which each of the nation’s ministries and agencies has commenced putting EBPM into practice. The establishment of systems is progressing, but EBPM has not yet penetrated to the site of policymaking. The main reason for this is that the government is not making use of the power of the private sector. This has two meanings. One of these is that the government is not utilizing the data that the private sector possesses. A diverse range of types of data is indispensable for scientific evaluations. Today, it is overwhelmingly the private sector that possesses data. Unless the government is able to make more efficient use of private sector data, it is likely that the progress of EBPM will be slow. The other meaning is the fact that the government has little concept of leaving the analysis of data and the formulation of statistics to the private sector in those areas in which the private sector is most capable of these operations. The private sector is far in advance of the government not only from the perspective of possessing data, but also from the perspective of possessing the human resources required to analyze those data. The government has no option but to make use of these resources.

The utilization of data has been slower in Japan than in other advanced nations. In the US, the government and the central bank have begun to draw on the power of the private sector to enable data-driven policy formulation. The Federal Reserve Board operates a system by means of which it obtains information from credit card processing companies, allowing it to rapidly gain an accurate understanding of consumer trends. The FRB might analyze, for example, how exchange rate changes affect the number of inbound tourists and the amount that each spends. This is a world that traditional government statistics have not comprehended. There is a deft sharing of roles with the private sector – for example, detailed and troublesome procedures such as processing the noise in raw data are entrusted to the private sector. This process, by means of which both the public and private sectors make use of the data and expertise possessed by the private sector in order to accelerate data utilization, is also taking place in Europe.

In Japan, some financial institutions have begun to utilize data in this way, but at present there is no sense of urgency with regard to the transition to data-driven decision-making in either the public or private sectors. It is the universities that will be the key to the success of EBPM in Japan. Universities have collective experience in the causal reasoning that is essential to EBPM, and abundant human resources able to put it into practice. The creation of mechanisms enabling these resources to be put to best use has already begun. It is to be hoped that this movement will gain pace in the near future.

When making policy decisions, it is important to conduct discussions based on scientific evidence rather than mere impressions and assumptions. As Professor Daiji Kawaguchi of The University of Tokyo emphasizes in this issue of My Vision, ours is an era in which the necessity for establishing an order of priority in budget allocation is increasing, and this in turn makes it necessary to establish mechanisms for the formulation of policy on the basis of solid evidence. Evidence-based policymaking (EBPM) is based on precisely this idea, and its importance has come to be recognized in recent years. However, in order to realize policymaking of this type, it is also essential to examine how to collect and analyze solid evidence, and how to utilize that evidence in policy formation. In this issue of My Vision, experts in the field discuss these points from a variety of perspectives.

Professor Kawaguchi also points out the importance of government agencies making greater use of data and creating the conditions for evidence to actively influence policymaking. He indicates that in the absence of active efforts in this area, even if ministries and agencies formulate statistics and prepare data, little progress will be made. This is an important point against the background of the occurrence of problems in government statistics, the implementation of reviews, and discussion concerning the best direction for the future.

But it is not enough to simply conduct discussions using numbers. It is a common pattern that numbers and data that suit one’s own preconceptions are found and presented as evidence – but such discussions cannot be considered to be solidly evidence-based. This point is emphasized in this issue by Dr. Yohei Kobayashi of Mitsubishi UFJ Research & Consulting and Professor Makiko Nakamuro of Keio University. Dr. Kobayashi indicates that EBPM is not simply a matter of using numbers in the discussion of policy, defining the method instead as “an effort to select better policy measures while examining the causal relationship between measures and outcomes.”

Lack of appropriate consideration of causal relationships is a major mistake that is often observed in the utilization of data. In this issue, Professor Nakamuro explains the difference between relationships of correlation and relationships of causation using an easily understandable example. However, in reality, it is rather difficult to verify causal relationships. This makes it necessary to pursue the method of “randomized controlled trials,” or RCTs, as specified by both Dr. Kobayashi and Professor Nakamuro. RCTs are a method that makes it possible to rigorously define causal relationships, something that has attracted renewed attention with the winning of this year’s Nobel Prize in Economic Sciences by Abhijit Banerjee, Esther Duflo and Michael Kremer as a result of research using the method.

As Dr. Kobayashi tells us, because the introduction of EBPM in Japan commenced with discussion of reform of the method of formulating statistics, Japanese EBPM has tended to focus on the preparation of data, but the method is inherently one that verifies causal relationships in relation to policy measures. Professor Nakamuro also indicates the importance of the correct identification of causal relationships, telling us that there are cases in which the failure to accurately identify causal relationships leads to adverse effects.

Emeritus Professor Martin Hammersley of the UK’s Open University points out to us that RCTs are not the be-all and end-all in policy-making. Indicating that RCTs are frequently only able to demonstrate the causal relationships assumed by the researchers, while other variables might actually be in play, and insisting on the importance of political factors such as the inclinations of the implementers of the policy and the people affected by the policy, Professor Hammersley warns that it is not possible to base policy directly on the results of experiments. It is clear that while RCTs enable us to rigorously demonstrate causal relationships, this does not mean that policy can be formulated directly on this basis. This is a point that needs to be adequately understood.

No matter what the case, the key to enabling rigorous policy discussion based on evidence will be whether or not it is possible to collect a sufficient amount of evidence. In Japan, there is still a dearth of evidence that can be used effectively in the formulation of policy, and how to collect this evidence will be the major issue. In relation to this point, Professor Tsutomu Watanabe of The University of Tokyo indicates the importance of making use of the private sector. Clearly, this is the age of data, and data is considered to be a goldmine; it is an age in which the private sector is consequently expending considerable effort on accumulating a wide variety of data. Given this, it will be necessary to give up the concept that the government must collect all of the data that forms the basis of policy, and embrace the concept of using private sector data. Professor Watanabe also suggests that we should not only use private sector data, but also make use of the power of the private sector for data analysis and evaluation.

The contemporary era is one in which governments no longer act alone in many areas, but rather actively make use of public-private partnerships and the creative ingenuity of the private sector. EBPM is another area in which we should make greater use of the intelligence and abilities of the private sector. In attempting to determine policy in a more scientific fashion, it will also be necessary to promote greater understanding among the public as a whole as to what constitutes evidence and what significance evidence-based policy decisions will have for policy formation. It is my hope that this edition of My Vision will help in this.