How do we know if it is working? Practical approaches to evaluating pro bono work
Claire Donse, DLA Piper
Nicolas Patrick, DLA Piper
We continue to confront an access to justice crisis around the world. The demand for pro bono services is increasing and we need to ensure that we are investing our resources in the most impactful way. As our pro bono practice matures, we have been making a concerted effort to evaluate the impact of our work.
Assessing and evaluating impact is a perennial topic for discussion in the pro bono community and much of the discussion focuses on the difficulty of such analysis. There are some metrics we can easily collect we can count the hours we spend on a project, the number of individuals that we assist or the number of downloads that a report receives but how do we go beyond that? How do we know that our work is having the intended impact?
In the past 18 months or so, we have conducted evaluations on five selected projects, ranging from an internal review of the impact of a multi-jurisdictional research project to an external review of a non-governmental organisation (NGO) partnership. In this article, we outline the approaches used in these evaluations, the rationale for the evaluation, the methodology and resources used and identified lessons. The models outlined in this article are not meant to represent a perfect approach, they are rather the start of a process that we intend to continuously improve and refine over the coming years.
Internal review of multijurisdictional research projects
The first category of projects reviewed were multijurisdictional research projects aimed at delivering systemic change. These projects are often significant undertakings for law firms and can each require up to 1,000 hours of pro bono time. We were concerned that there may have been a discrepancy between the amount of time required to produce the research and the value of the research to the client.
In recognition of the fact that our clients law reform objectives could only be realised after a significant amount of time had passed, we considered that it was important to wait at least two years between the delivery of the work product to the client and the time of the review. We therefore set about conducting a retrospective impact assessment of all multijurisdictional research projects conducted by DLA Piper lawyers over the past eight years. The review was conducted on a rolling basis by an associate in the pro bono team. The associate conducted telephone interviews with client contacts using a set questionnaire comprised of ten open questions,
 producing a report that was submitted to the pro bono team. The evaluation process did not require a significant investment of time or resources, necessitating only an hour or two of the associates time in conducting the interview and writing up the summary.
Contrary to our expectations, in every case that we have been able to evaluate to date, the research led to systemic change. In one instance, the research had led to a change in the law in six countries. The key learning from this exercise was that a properly selected research project is indeed worth the investment. We have since formalised a set of criteria to be applied before accepting instructions on a large research project, including an up-front assessment of the clients capacity and plan to utilise the research. As a result of the review, we also established new guidelines for our team in managing large research projects to address client feedback relating to the variable quality of research between jurisdictions. We also introduced a consistent policy of kick-off meetings with the client and team members at the start of a project, to provide context, connection and ensure that everyone is on the same page. An additional benefit of the review was that it allowed us to reconnect with clients, some of whom we had not worked for since delivering the research. In many instances, it also provided us with an opportunity for follow up work.
Internal review of community-run clinics
The second category of projects reviewed was community-based clinics. These longstanding projects had been running for over five years without a wholesale review. Throughout the life of the various clinics, over 60 fee earners had been involved and the firm had invested around 400,000 of billable time.
As with the multijurisdictional research, the evaluation of these projects was conducted internally on a rolling basis. DLA Piper lawyers involved in the clinics conducted site-visits to the clinic to review all the files active over the course of a three-month period. The ambit of the review included the scope and the quality of legal advice, as well as the management and administration of the clinics. The lawyers sent their reports to the local pro bono team, who prepared a summary report. We then reviewed the report and considered whether the clinics were meeting their objectives. Compared to the multijurisdictional research projects, this review process required both more time and more people, but did not require any significant expenses.
The review found that the clinics were not efficiently delivering legal support. The administrative set-up of the clinics and the system of rotating partner firms resulted in delays for clients, some of whom saw multiple lawyers for the same issue. The pro bono team therefore terminated DLA Piper's participation in the clinics and the local reviewers identified alternative avenues to provide support for the communities. This also led to the establishment of a number of new end-to-end advice clinics on specific issues affecting the local community.
Internal review of long-term pro bono secondment programme
The third project reviewed was a long-term secondment of DLA Piper lawyers to the government of a fragile, post-conflict nation. Over an eight-year period, the firm provided pro bono legal support to various arms of government on a range of matters through a regular secondment programme, in which lawyers spent between six months to a year working in the government offices of the client. Over an eight-year period, eight DLA Piper lawyers rotated through the seat and a virtual client team of more than 100 lawyers from across the firm supported the secondees. This was a significant investment of resources, estimated to be approximately 8.5m at the time of the evaluation.
A retired firm partner with no prior connection to the programme handled the internal review, conducting telephone interviews and exchanging emails with five of the secondees, as well as with representatives of the client and the two relationship partners. The interviews were supported by documentation of the matters conducted for the client to date and a schedule of hours. The partner prepared a report capturing the feedback and recommendations, which was then delivered to the pro bono team. Given the size of the investment, the firms Executive and Board was also provided with a report.
The findings in the report highlighted the importance of introducing senior lawyers at the inception of the project and confirmed the need to clearly define the goals of the project and confirm these with the client. It also flagged the need to provide robust human resources support for the secondees. These recommendations have been implemented for similar international secondment programmes.
External impact report for charity partnership
The fourth project reviewed was a project to promote childrens rights in Bangladesh. Our support included both financial and pro bono contributions over a multi-year timeframe. The agreed outputs of the programme included drafting legislation, providing support for adolescents at risk or coming into contact or conflict with the law, as well as training key professionals to strengthen child protection mechanisms. The review was conducted two years into the project. The reporting schedule was pre-agreed during the partnership negotiations to allow for clear identification of the impact of contributions. At the time of the review, DLA Piper had invested more than 340,000 and provided over 9,800 hours of pro bono support.
Recognising that we did not have the capacity, expertise or local access to conduct an evaluation of the impact of this work, we gave responsibility for the evaluation to the client and built an allowance for the cost of the evaluation into our financial contribution. The charitys account manager produced a report based on the monitoring and evaluation conducted by their in-country management and programme team. The report was submitted to DLA Piper's responsible business team.
The report confirmed that since the start of the project, 140 children arrested by the police had their cases handled in the community, rather than in the courts, and that 224 professionals, including lawyers and police, received training in child-friendly justice. These figures allowed us to report back to the firm that the partnership was fulfilling its objectives, and provided measurable impact that could be communicated both internally and externally.
External review of a NGO partnership
The final project reviewed was a multi-year fundraising and pro bono partnership. Since the inception of the partnership, DLA Piper and its staff had donated around 850,000 and performed 4m worth of pro bono legal work for the NGO. The review was conducted at the end of a three-year period, as activity planning was moving forward for the next three years.
Given the size and scope of the project, we commissioned a specialist external evaluator from I.G. Advisors to conduct the review. The consultant conducted fourteen interviews with 24 individuals from across both DLA Piper and the NGO. The consultant reviewed communications and materials produced by both organisations. The consultant then prepared a report and delivered it to both parties. Within the firm, the review was shared with the DLA Piper Executive and Board.
The reports recognition of the impact of the partnership and its alignment with the firms aims resulted in an extension of the partnership. Following the evaluation, we restructured the relationship and DLA Piper substantially reduced its expectations of the NGO. We included the NGO in our key client programme and recruited a full-time relationship manager. We also worked to build wider relationships within the NGO to source more pro bono opportunities.
We have experimented with a range of different methodologies for evaluations, from simple internal reviews to full-scale external evaluations. We have sought feedback from the beneficiaries, the people directly involved in performing the pro bono work and from people one step removed. The results of the evaluations were not always as expected, but they all confirmed the value of feedback and evaluation. They also allowed us to obtain a better insight into the needs of our clients and strengthened the relationship. We intend to continue to evaluate the impact of our work and to improve our approaches to evaluation.
 We are happy to share the questions with anyone who is interested in undertaking a similar review.