Connecting
the Dots:
Data Use in
Afterschool Systems

Chapin Hall at the
University of Chicago
April 2016

Julie Spielberger
Jennifer Axelrod
Denali Dasgupta
Christine Cerven
Angeline Spain
Amelia Kohm
Nicholas Mader

Commissioned by:

Connecting
the Dots:
Data Use in
Afterschool Systems
ACKNOWLEDGMENTS
We would like to acknowledge contributions to this study and report of current and former
Chapin Hall colleagues Cheryl Smithgall, Stephen Baker, Lisa Walker, Natalya Gnedko-Berry,
and Carolyn Winje, as well as research assistants Sarah Bryce, Callie Frietag, and Ji Yoon Noh.
At Chapin Hall, Deborah Daro, Robert Goerge, and Ada Skyles provided guidance on the study
design and/or feedback on draft reports. We also acknowledge Matt Brenner for his assistance
in editing and producing this report. In addition, we thank Cynthia Coburn, Amy Gerstein, and
Robert Halpern for their review and thoughtful comments.
We are grateful to The Wallace Foundation for its support and ongoing guidance in the
development of this study. We would like to thank the city stakeholders for partnering with
Chapin Hall throughout this project to share their knowledge and experiences and to describe
the evolution of their afterschool systems.
Citation: Spielberger, Julie, Jennifer Axelrod, Denali Dasgupta, Christine Cerven, Angeline Spain, Amelia Kohm, and Nicholas Mader.
Connecting the Dots: Data Use in Afterschool Systems. Chicago, IL: Chapin Hall at the University of Chicago, 2016.
© 2016 by Chapin Hall at the University of Chicago
1313 East 60th Street
Chicago, IL 60637
ISSN 1097-3125

Table of Contents
Executive Summary	
Key Findings	

vi
vii

Conclusion	ix

1. Introduction	

1

The Next Generation Afterschool System-Building Initiative	

3

Research Questions and Study Methods	

3

Conceptual Framework: Aspects of a Data System	

4

Overview of the Report	

7

2. The Context and Goals for Data Use	

9

Contexts for Afterschool System Building	

9

Goals and Purposes for Investing in Data Use	

15

Summary	16

3. Building Capacity to Collect and Organize Data	

19

Assessing and Engaging the Network	

21

Selecting Data Elements	

21

Establishing a Management Information System (MIS)	

24

Establishing Formal Agreements Among Network Partners	

33

Preparing and Training the Network	

35

Establishing and Maintaining Data Quality	

37

Getting Data Flowing	

39

Summary	42

4. Using Data and Building Capacity for Data Use	

43

Data Use for Learning and Improvement	

45

Strategies for Building Capacity for Data Use	

50

Summary	61

5. Summary and Emerging Themes	

63

Summary of Findings	

64

Connecting the Dots and Looking Ahead	

68

Bibliography	69

Appendix: Interview Guide (Phase 1)	

iv | Chapin Hall at the University of Chicago

71

List of Figures
Figure 1. 	 Aspects of an Afterschool Data System	

6

Figure 2. 	 Foundational Partners in Next Generation Afterschool
	
Systems, 2014	

10

Figure 3. 	 Types of Data Selected for System-wide Collection, 2014	

22

Figure 4. 	 Types of System-wide MIS Selected by the Next Generation Cities	 25
Figure 5. 	 MIS Status of the Next Generation Afterschool Systems
	
at the End of 2014	

28

Figure 6. 	 Common Objectives for Data Use Trainings Reported by
	
Afterschool Systems	

36

Figure 7. 	 Strategies Using People for Overseeing Data Quality	

37

Figure 8. 	 Sample Data Dashboard from Nashville (Anonymous) 	

41

Figure 9. 	 Types of Systematic Data Use for Learning and Improvement
	
in Five Afterschool Systems, 2014	

45

Figure 10. Afterschool Systems’ Strategies for Building Providers’
	
Capacity to Use Data	

50

List of Tables
Table 1.	Aspects of a Data System Framework	

8

Table 2.	Technology in the Next Generation Afterschool Data Systems, 2014	 12
Table 3.	The Next Generation Afterschool Systems’ Coordinating Entities
	
and Governance Structures, 2014	

13

Table 4.	Overview of Next Generation Afterschool System Partners
	
and Scope, 2014	

17

Table 5.	Processes and People in the Development of Data Systems	

20

Table 6.	Factors in Choosing an MIS: Themes in Afterschool
	
Data System Development, 2014	

32

Table 7.	 Processes and People in the Development of Data Use for
	
Compliance, Learning, and Improvement	

44

Table 8.	Opportunities and Challenges for Building Capacity for Data Use	

56

Connecting the Dots: Data Use in Afterschool Systems | v

EXECUTIVE SUMMARY

Afterschool programs—programs for children and youth that happen regularly after
school on weekdays, weekends, and during the summer—aim to keep young people safe
and to foster skills needed to succeed in school and life. As concerns about children’s wellbeing and the quality of education in low-income communities have mounted, cities are
increasingly implementing strategies to ensure that afterschool programs are high quality
and widely available. To this end, a growing number of cities have invested in afterschool
systems to coordinate disparate programs and funding streams. A primary function of
most afterschool systems is to develop and maintain a data system, which allows the
afterschool system to collect, analyze, and apply data to accomplish its goals.

With support from The Wallace Foundation, nine cities across the
country are participating in the Next Generation Afterschool
System-Building Initiative, a multi-year effort to strengthen the
systems that support access to and participation in high-quality
afterschool programs for low-income youth.
The nine cities were selected in part because they already had a solid foundation for
an afterschool system that included strong city leadership and mayoral commitment.
This interim report documents how these cities used data to inform and improve their
afterschool systems over a two-year period from 2012 through 2014. It is important to
note that these system-building and data system development efforts are continuing,
and the findings presented here are current as of the end of 2014. A second report will be
produced at the end of the study in 2017.
Information for the report was drawn from an analysis of city documents, site visits,
and interviews with key stakeholders, observations of selected trainings and meetings
related to data use, and attendance at three, cross-city grantee meetings in fall 2013,
spring 2014, and fall 2014. Over the first two years of the study, the nine cities engaged
in a wide variety of activities to develop their capacity to use data. They did this primarily
through investments in three components that make up a data system: people, processes,
and technology. The current study is guided by the overarching idea that data use is a
meaning-making activity that involves people and processes as well as technology.

vi | Chapin Hall at the University of Chicago

To date, research on data use in afterschool systems has focused more on implementation
of technology; that is, management information systems (MIS)1 and the purposes for
which those systems were used, rather than on what it takes to develop and sustain
effective data use. The current study focuses especially on the people and processes that
intersect with technology in a data system and the nature of the relationships among
these three aspects in the development of the capacity for data use.

This report discusses the current capacity of city afterschool
systems to collect and use data to inform their decisions and
system-building activities.
It also describes the common strategies system stakeholders have used to develop that
capacity, the contextual factors that affect data use, and perceptions of the value of data use
to improve afterschool program quality, access, systems, and youth outcomes. The report
highlights some of the challenges the cities have encountered as well as the progress they
have made in choosing outcomes for their data systems; designing the technology to help
manage, collect, and analyze data; and engaging stakeholders in the process.

Key Findings
Developing the capacity to use data depends as much on people and processes as it does
on technology. Consistent with prior research, the nine cities in this study recognize the
value of system building as a way to improve program quality and increase access to and
participation in high-quality programs among low-income youth. They also recognize the
need for information about whom they are serving and whether specific strategies are
resulting in better program quality and more equitable access to high-quality programs.
To address that need, the cities in this study have invested significant resources in data
systems to support the improvement of their afterschool systems and programs.
At the time of this report, these cities were in various stages of collecting and using data
to inform their afterschool system building, improve program quality, and make decisions
about the allocation of resources. They viewed the development of the capacity for data
use as an important part of their system building, but they were still developing their
capacity. Efforts to identify what data are needed, how to collect and organize these data,
how to build or obtain the human capital or expertise to develop the system, and how to
develop and sustain effective ways to apply these data to real world challenges, have been
complex and challenging.
1.	 MIS refers to both a management information system and systems.

Connecting the Dots: Data Use in Afterschool Systems | vii

KEY FINDINGS INCLUDE:

•	 The purposes for investing in these data systems

are similar across cities. City afterschool systems
particularly seek data about program supply and
demand, characteristics of participating youth, program
quality, and program effects on youth. They use data
to manage programs, to fulfill accountability and
contract requirements, to make decisions about how to
improve program quality, to improve the allocation of
resources and availability of services in underserved
communities, and to identify children and youth
who might need additional services that afterschool
programs can provide.
•	 Although the purposes for data are similar to

those in earlier research, the context for data use is
constantly changing. City afterschool data systems are
continuing to evolve in response to new knowledge
and technology, new priorities for cities and their
afterschool systems, renewed focus on performance
management and the use of data to drive decisions,
and transitions in city and school leaders who support
the afterschool system.
•	 In selecting indicators and outcome measures for

their data systems, city afterschool leaders wanted
them to be useful and actionable for multiple
stakeholders, including providers, school staff, and
city leaders. Some of the nine cities were making
progress towards developing and supporting a “culture
of data use” through training in data visualizations,
data interpretation, and quality improvement cycles.
However, even cities further along in the development
of their data systems were still learning how to
present the data in formats that can be understood
broadly and applied to improve program quality and
youth outcomes.

viii | Chapin Hall at the University of Chicago

•	 Stakeholders in all nine cities have invested in processes

to improve program quality. At the end of 2014, six of the
nine were collecting data about program quality using
formal assessments designed for that purpose. Five of
the nine afterschool systems were beginning to use data
as part of a continuous improvement cycle by helping
providers interpret the data to support change at the
program level. However, there was little evidence of the
use of data as part of a continuous cycle of inquiry to
improve the afterschool system as a whole.
•	 City afterschool systems are still developing and refining

their measurement strategies to accurately reflect their
effects and outcomes. Stakeholders in more than half of
the cities expressed interest in collecting social-emotional
indicators as well as, or instead of, academic measures,
and most viewed such indicators as more appropriate
to the expected outcomes of afterschool programs. In
some instances, because measures of social-emotional
competencies were still being developed and did not easily
align with the outcomes sought by school districts or city
leaders, it was challenging to incorporate these indicators
into their systems.
•	 Building the capacity to use data is not an easy or a

straightforward process and requires time, patience,
persistence, and flexibility. Most of the study cities found
that they were shifting course several times during the
processes of selecting data elements and technology for
collecting and managing the data. They also experienced
common challenges of outdated technology that was
inadequate to meet their data needs or too costly to
replace or upgrade. Other challenges came with changes
in city or system leadership and goals, the availability
of new technology, or turnover in key personnel.
These challenges indicate that a data system, much
like the afterschool system itself, needs the ability to
accommodate and adapt to change.

Emerging Strategies
Although it is premature to specify strategies that have
worked especially well across this group of afterschool
systems, there were similar themes in how they
approached the development of their data systems.
•	 Earlier stages of development focused on

identifying purposes for data use and designing and
implementing management information systems
(MIS) technology, followed by the development of
training and other processes around that technology.
From there, at least in five cities, evidence showed
that the use of data for learning and improvement as
part of a cycle of inquiry was starting to deepen the
commitment to a data-driven approach and guide the
growth of the data system.
•	 One theme in the activities of a number of cities was

to intentionally start small with a limited set of goals
for data collection and use, and/or a limited set of

providers piloting a new data system, with plans to
scale up gradually.
•	 Research and data expertise was essential to

developing data systems. Expertise came from within
as well as outside the intermediary organization
coordinating the initiative. Three cities contracted with
outside research partners through all phases of the
development of their data systems, from establishing
an MIS to producing reports. Three other cities did so
for more limited purposes, primarily the analysis and
reporting of data collected by providers. There was
some indication that cities in which research partners
played more wide-ranging roles in establishing their
data systems and assisting in the analysis and use of
data made more progress than cities in which research
partners had more limited roles. We will be following
this theme in the next phase of the study.

Conclusion
The Next Generation cities have many purposes for their
data systems and varied strategies for achieving them
based on available people, processes, and technology
resources of their afterschool system. They assessed
needs for data based on the goals of system partners and
a broader city agenda. In selecting data to be collected,
they were attentive to the goals and information needs of
a diverse group of providers and other stakeholder groups,
and they complied with the requirements of federal privacy
laws governing the use of data when partnering with
school districts, while choosing measures and indicators to
capture the potential outcomes of a heterogeneous group
of afterschool programs in their cities. They evaluated their
existing data systems and identified limitations, costs, and
tradeoffs to make decisions about technology that would
give users of various levels access to meet their needs.
They established policies, practices, and structures such

as working groups and committees to support access to
data and effective data handling. The cities sought to hire
qualified staff with both technical expertise and the ability
to communicate the value of—and expectations for—data
use to various audiences, including but not limited to, front
line staff, agency directors, and funders. They invested in
training and professional development to improve the
capacity to analyze and interpret data. Based on these
findings, we hypothesize that as important as technology
is to system building, most of the factors that appear to
facilitate or inhibit data use in city afterschool systems—
norms and routines, partner relationships, leadership and
coordination, and technical knowledge—hinge on the
people and process components of a data system. As this
study continues, we look forward to learning more about
what it takes to develop these aspects of a data system
and the value of the resulting data use for system building.

Connecting the Dots: Data Use in Afterschool Systems | ix

1. INTRODUCTION
“Collecting and having the ability to analyze data citywide will allow us as a city, as a group of stakeholders,
to make informed decisions, set priorities, solve problems and reach a consensus on a citywide strategic plan
for out-of-school time. Data drives everything.” —Fort Worth stakeholder

KEY POINTS

•	 Research has shown that high-quality
afterschool programs have a positive influence
on youth development.
•	 Nine cities invested in efforts to build
afterschool systems to improve quality,
impacts, and access to programming for youth.
•	 To inform their system building, the cities
invested in people, processes, and technology
to develop their capacity to use data.
•	 This interim report highlights the initial
accomplishments and challenges of the
cities in developing the capacity to use data
throughout the afterschool system.

Afterschool programs for children and youth that happen
regularly after school on weekdays, weekends, and
during the summer, aim to foster personal, social, and
occupational skills young people will need to succeed in
school and life. Interest in these goals, as well as in keeping
children safe during out-of-school time, has increased
dramatically over the past two decades as concerns
mounted about children’s well-being and the quality
of education, particularly in low-income communities.
Research on the impact of afterschool programs also has
increased. Studies indicate that afterschool programs
can have a positive influence on young people’s selfconfidence, social behaviors, grades, and test scores if they
are of high quality and engage young people on a regular
basis.2 Research has also added to our understanding of
what makes for a quality program that will positively affect
the academic, physical, social, and emotional well-being
of participating youth. Generally speaking, the following
characteristics of afterschool programs are associated with
positive outcomes for youth who participate on a regular
basis: (1) settings that provide opportunities to develop
supportive relationships with adults and peers,
(2) engaging and varied hands-on learning experiences, and
(3) appropriate structures to facilitate these interactions
and learning.3

Cities are increasingly implementing strategies to coordinate afterschool programs,
although the degree of coordination varies widely.4 Ensuring that afterschool programs
are high quality and widely available requires a high level of coordination among disparate
programs and funding streams. These efforts are often led by coordinating bodies that
bring together diverse perspectives, advocate for financial and political support for
the afterschool field, coordinate funding, develop common quality standards, and link
programs with supports and services.5 When we refer to “system building” in this report,
we mean efforts to develop, implement, and maintain an afterschool system.
2.	 Bodilly et al. 2010. Gardner, Roth, and Brooks-Gunn 2009. Halpern 2003a, b. Little et al. 2008. Larson 2000. Russell, Mielke, and Reisner 2009.
Vandell, Reisner, and Pierce 2007. Larson et al. 2004. Durlak and Weissberg 2011. SAFE (http://www.expandinglearning.org). Mahoney, Parente, and
Zigler 2009. Halpern 2002. Little, Wimer, and Weiss 2008.
3.	 Durlak et al. 2010. Eccles and Gootman 2002. Larson 2000. Little, Wimer, and Weiss 2008. Mahoney, Parente, and Zigler 2009. Smith 2013.
4.	 Simkin et al. 2013.
5.	 See, for example: Bodilly et al. 2010. Halpern, Spielberger, and Robb 2001. Little, Wimer, and Weiss 2008. Johnson, Rothstein, and Gajdosik 2004.
The Wallace Foundation 2013. Yohalem, Wilson-Ahlstrom, and Yu 2005.

1 | Chapin Hall at the University of Chicago

There is no template or model for building effective collectives such as afterschool
systems. However, we are learning from cities engaged in system building that key
elements for developing and sustaining an afterschool system include strong leadership,
coordination, effective use of data, and a comprehensive approach to quality.6 Effective
afterschool systems use data to identify community needs, advocate for funding,
distribute limited resources, develop and monitor quality standards, and create shared
accountability. 7
We also are learning that cities face particular challenges in their efforts to collect
and use data to their fullest potential.8 Indeed, a recent study found that of cities
with populations above 100,000 that were implementing some type of afterschool
coordinating structure, only 34 percent had common data systems.9 It is challenging to
gain access to data from multiple organizations, overcome privacy concerns, learn to use
and interpret data, and select the right data to assess quality and impact. Moreover, many
organizations and institutions in afterschool systems tend to view data mainly as a tool for
monitoring and compliance rather than as part of a cycle of inquiry to support continuous
improvement (see Box 1).
This report presents interim findings from a study of how afterschool systems develop the
capacity to use data to its fullest potential and how the people, processes, and technology
that form the data system affect the development of this capacity. The study, which began
in the summer of 2013 and will conclude in the spring of 2017, is taking place in nine cities
participating in the Next Generation Afterschool System-Building Initiative, funded by The
Wallace Foundation.
Box 1.

The Purpose of Data in Afterschool Systems
Kingsley describes three main frameworks for data use in
afterschool systems: compliance, accountability, and continuous
improvement.10 He argues that data use for compliance is the
weakest way to make use of data because the data may never be
analyzed or connected to decision making about programming.
In an accountability framework, systems analyze and use
data to measure performance on specific outcomes for which
providers are held accountable. Further, Smith notes that data
use for accountability can have “high” or “low” stakes, in terms
of whether information on performance is the basis for funding
decisions.11 But this performance information is still not explicitly
integrated into decision making about programming. Data use
for continuous improvement starts to happen when data are used
as part of a cycle of learning and improvement, regardless of
whether they are also used for compliance or accountability.
Gerstein suggests that the compliance framework currently
dominates the use of data in youth-serving organizations, as
opposed to using data to ask questions, measure performance,

solve problems, develop strategy, and make decisions.12 She
views data use as a continuum of three phases that span from
what might be considered the simplest form, reporting data for
compliance (phase 1), to connecting data to strategy (phase 2),
to what might be the most complex form, using data for learning
and improvement (phase 3). These phases are not necessarily
distinct or linear, and organizations tend to move back and forth
along the continuum.
This study is particularly interested in how afterschool systems
build the capacity to use data for learning and improvement—or
what the National League of Cities terms “high-value” uses of
data. These include: (1) assessing youth outcomes and system
impact; (2) promoting accountability, demonstrating return on
investment, and allocating scarce resources; (3) empowering
afterschool providers and reducing paperwork; and (4) facilitating
peer benchmarking and professional development.13 As we will
see in Chapter 4, many of the ways in which cities had begun to
use data by the end of 2014 overlap with these four categories.

6.	 Browne 2015.
7.	 Bodilly et al. 2010. National League of Cities 2012.
8.	 Bodilly et al. 2010. McCombs et al. 2011. National League of Cities 2012.
9.	 Simkin et al. 2013.
10.	Kingsley 2014.
11.	 Smith 2013.
12.	 Gerstein 2015.
13.	 Bodilly et al. 2010. McCombs et al. 2011. National League of Cities 2012.

Connecting the Dots: Data Use in Afterschool Systems | 2

The Next Generation Afterschool System-Building Initiative
Selected for their solid foundation for system building that includes strong mayoral
leadership, the Next Generation Afterschool System-Building Initiative (Next Generation)
cities are Baltimore, Denver, Fort Worth, Grand Rapids, Jacksonville, Louisville, Nashville,
Philadelphia, and Saint Paul. Consistent with the goals of the initiative, these nine cities
are working on system building in several areas, including expanding youth participation in
afterschool programs, improving the quality of programs, improving policies, governance,
and coordination within the system, and developing systems to collect and use data to
inform their system-building activities. To support these efforts, technical assistance in the
areas of organizational development, quality assessment and improvement, data systems,
and governance was provided through support from The Wallace Foundation. The cities
also engaged in cross-site meetings, conferences, and other learning opportunities to
share successes and challenges across the sites and support the development of collective
knowledge about afterschool system-building efforts.
This study builds and expands on previous studies in this area. The 2010 Hours of
Opportunity study of afterschool system building focused its research about data
use on the implementation of management information systems (MIS) and the
purposes for which the MIS were used in the five cities that participated in the first
generation of afterschool systems supported by The Wallace Foundation.14 In the
present study, we continue to look at the development of MIS, but expand the focus
to explore all components of a data system, including the people, processes, and
technology more broadly.

Research Questions and Study Methods
The overarching goal of this four-year research study is to understand how city
afterschool systems develop the capacity to use data, what facilitates and hinders their
progress, and what value the data provide to the afterschool systems.
This interim report seeks to address the following questions:
•	 How did contextual factors affect how the cities approached the development of their

capacity for data use?
•	 How did the cities plan to use data?
•	 How were systems collecting, analyzing, and using data as of 2014 and what were their

strategies for building data systems?
•	 What challenges and successes did the cities experience in developing the

capacity to use data?
Our primary sources of data for this report are the activities of the nine cities participating
in The Wallace Foundation’s Next Generation Initiative and the perspectives of those
participating in those activities. This study began in the summer of 2013, a little more
than a year into the Next Generation Initiative. We used similar data collection and
analysis approaches across the cities. The initial tasks were to establish a baseline
understanding of each of the afterschool systems and the context in which each system
was developing, document the individuals and organizations involved, catalog planned
and in-progress activities, and capture the goals for data use for each system. We used
multiple sources of information, including literature on data use in the afterschool field
and in education and other fields, early documents and observations from each of the
14.	 Bodilly et al. 2010. McCombs et al. 2011.

3 | Chapin Hall at the University of Chicago

participating cities, telephone and in-person interviews, and on-site observations of selected
meetings. To organize and guide our analysis, we developed a conceptual framework that
includes the people, processes, and technology involved in collecting and using data within an
afterschool system.
We developed semi structured interview guides based on our literature reviews and our initial
conceptual framework of a data system. We also included questions about the characteristics
and policy context to understand the broader setting of the afterschool systems. (See appendix
for the interview guide.)
Chapin Hall researchers conducted in-person site visits with each city once during the winter
and spring of 2014 to interview key stakeholders and observe selected meetings related to the
development of data systems and use of data. Across the cities, 104 interviews were conducted
(an average of 11–12 per city) and 13 meetings observed. Stakeholders represented a range of
organizations and roles, including system-level leadership and managers in the afterschool
community. To preserve confidentiality, we only identify our informants by city name.15

Conceptual Framework: Aspects of a Data System
In this section, we present the conceptual framework that guides this study. From the
beginning, we have viewed data use as a meaning-making activity that involves people and
processes as well as technology. Here, we provide definitions of data as used in this context
and of key factors related to people, processes, and technology that any city involved in, or
planning the development of, an afterschool data system should consider. This framework is
the lens through which we examine the way that the Next Generation afterschool systems
have developed the capacity to use data.
When we use the term data in this report, we are referring to any information collected for
a particular purpose and collected in a way that ensures accuracy and consistency. Data can
come in the form of numbers, such as the number of students in a program, or words, such
as parents’ views on program quality. Such raw data are usually not as valuable for decision
making as summarized or analyzed data are; the latter provide a sense of scope and the
relationship among variables, including possible causes and effects. An example of raw data
would be the attendance records of students in a particular school, including those who
participated in an afterschool program. Calculating the average school attendance rates of
participants in the program and then comparing them to the average attendance rates of
nonparticipants at the same school might summarize these data. The data might be further
analyzed to assess whether those who participate have higher school attendance rates than
nonparticipants while parceling out the effect of other factors, such as special education status,
mobility, and ethnicity, which also might affect students’ attendance.
Data can be used at different levels of the afterschool system:
•	 To help youth at the individual level by identifying individuals who need extra support and

matching them to activities that respond to their needs.
•	 To inform frontline workers and program managers at the program level by providing

information on program quality.
•	 To influence other stakeholders and the functioning of the afterschool system itself at the

system level by identifying gaps in access to afterschool programs across a city.

15.	 Additional information on study methods is available upon request from the authors of the report.

Connecting the Dots: Data Use in Afterschool Systems | 4

To manage and facilitate the use of data, afterschool systems develop data systems. The data
system is a set of functions that allows the afterschool system to collect, analyze, and apply
data to accomplish its goals. Initially, the development of a data system for an afterschool
network appears to be a set of fairly linear and concrete tasks that gradually build toward
a fixed endpoint or sustaining product or process. The reality, however, is more complex.
While there are common steps, we found that each city’s progress was affected by an array of
factors within the system. Others who have studied data systems categorized these factors
broadly as human, social, and technological.16
Data systems, even those that are mature, are never static. They evolve in ever-changing
environments. Mayors are elected with new priorities. Grants end. New ones begin. School
districts are reorganized. Data security laws are enacted. Technological advances lead to
new ways of interacting with the data. The people, processes, and technology of a data
system change to respond to these changing contexts. We found that at each stage of data
system development, factors related to people, processes, and technology affected what was
accomplished at that stage as well as the general trajectory of the data system as a whole.
What emerges from the initial findings is the understanding that the three components of a
data system shift in response to both external contexts and internal pressures.
A data system might be illustrated as the three sides of a triangle with each side representing
one of three primary components: people, processes, and technology (see Figure 1). Each of
these three aspects of a data system influences the other factors, as indicated by the arrows
in the figure. For example, the expertise of the people involved in the data system influences
the type of MIS chosen, which, in turn, affects how data can be analyzed.
People are stakeholders—individuals, organizations, and institutions—involved in the
operation of the afterschool network in general and the data system specifically. Individuals
and groups bring different types of capital to a data system:
•	 Political capital from public officials, funders, school leaders, and governmental staff.
•	 Social capital that emerges from the network of relationships among those

involved in the system.
•	 Knowledge capital from those with technological, data collection, data analysis, data

application, and advocacy skills and experience.
Based on the evidence we have collected thus far, social capital is critical to establishing
trust and working through the complexities of sharing data in a transparent manner. It is also
apparent that the turnover of people within an afterschool system can be both frequent and
disruptive. The systems that seem to be better able to mitigate the impact of turnover on
the functioning of the afterschool system in general, and on its data system in particular, are
those that have intentionally focused on building horizontal and vertical connections among
the people and groups involved in the system. As data systems mature, our preliminary
research indicates that the network of relationships often broadens. They may begin with
a few key leaders with social, political, and knowledge capital but over time, become more
intentionally inclusive with the establishment of advisory boards, councils, and/or leadership
teams that span levels and domains.
The second aspect of a data system is the set of processes involved in building and
maintaining the system. The processes side of the triangle includes the routines, norms, and
practices related to the collection, organization, analysis, interpretation, and use of data
to meet the goals and inform the operation of the afterschool system. As systems evolve,
the processes side of the triangle expands to develop structures and supports that align to
16.	 Coburn and Turner 2011. Mandinach 2012.

5 | Chapin Hall at the University of Chicago

the needs of the system. For example, initial processes are established so that people know how
to use the system and to enter data accurately. As systems develop, the focus widens to include
strategies that facilitate meaning-making with the data at all levels of the system.
Technology, the third side of the triangle, is the means by which data are organized and accessed
to inform the operation of the afterschool system. An MIS comprises both hardware and software
systems. Key questions need to be answered before and during the implementation of technology:
•	 Where should the system live? The selection of an organization or institution to house the

hardware and manage the software will have implications for how accessible data are, who is
accountable for data quality, and the amount of long-term resources to support the system.
•	 What type of MIS best fits the afterschool system? Cities can choose from packaged

software systems, hybrid systems that combine packaged and custom-built solutions, and custom
solutions built specifically for the context of the afterschool system. Each option has pros and cons
in terms of upfront and ongoing cost, adaptability, and usability.
The framework comes into sharper focus when we look at the aspects of each side of the triangle
that are critical to the development of an afterschool data system.
Table 1 presents the critical factors, gleaned from both
the literature and our first wave of data collection,
which influence data use in afterschool systems.
This report provides an early look at how
these factors vary across sites, how local
Figure 1.
contexts influence them, how they
Aspects
of an
foster or impede progress, and how
Afterschool Data System
they interact.
From our observations
and analyses thus far,
promising practices for
system building are
emerging. During our
PEOPLE
second round of data
collection, we will
continue to investigate
these factors and refine
our understanding as
the nine afterschool
systems mature.

PROCESSES

DATA
SYSTEM

TECHNOLOGY

Connecting the Dots: Data Use in Afterschool Systems | 6

Overview of the Report
To date, research on data use in afterschool systems has
focused more on implementation of technology—that
is, MIS and the purposes for which those systems were
used—than on what it takes to develop and sustain
effective data use.

The current study addresses these gaps in the
literature by focusing on the people and processes that
interconnect with technology in a data system and the
nature of the relationships among these three aspects in
the development of the capacity for data use.
This report covers activities between spring 2012, when
the initiative began, through December 2014. In presenting
our early findings, we have taken both a chronological
and a topical approach to describing the evolution of
data use in nine very different afterschool systems. For
example, the sequence of topics in Chapters 3 and 4 follow
a typical progression in which the capacity to collect and
organize data precedes the development of the capacity
to analyze and interpret data. At each stage, we also
discuss cross-cutting issues of people, processes, and
technology to show how they work together, often in a
less linear manner. This approach reflects the complexity
of system building, which is a multifaceted process that
occurs within a changing context. The report attempts to
capture the journeys of nine cities as they develop capacity
to use data. Because this is an interim report, the story is
still emerging. We will provide a more complete picture
of data system development in our final report in 2017,
which will draw on a second wave of data about the nine
systems. Below we provide a brief overview of each of the
chapters of the report.

CHAPTER 2: THE CONTEX T AND
GOALS FOR DATA USE
Chapter 2 looks at the unique contexts in which each of
the nine afterschool systems took root, with a particular
focus on the role of government agencies and school
districts in the formation of these systems. We also
examine the role of coordinating entities—the groups that
orchestrate the afterschool systems—in the development
of data systems and then look closely at the goals each
city planned to pursue.

7 | Chapin Hall at the University of Chicago

CHAPTER 3: BUILDING C APACIT Y TO
COLLECT AND ORGANIZE DATA
The types of data elements collected across the cities, the
ways in which cities selected, built, and adapted their MIS
to meet changing needs, and the ways they ensured the
quality of data in their MIS are discussed in Chapter 3. In
each section, we consider key facilitators and barriers to
progress related to people, processes, and technology.

CHAPTER 4: USING DATA AND BUILDING
C APACIT Y FOR DATA USE
In Chapter 4, we examine the ways cities in the study
engaged and trained stakeholders in data analysis and
interpretation and how they used data for decision making.
As in the preceding chapter, we consider key facilitators
and barriers to progress at this stage of data system
development related to people, processes, and technology.

CHAPTER 5: SUMMARY
AND EMERGING THEMES
The interim report concludes with a summary of findings,
including key barriers, facilitators, and lessons from our
first round of data collection. We also preview plans for the
final round of data collection.
Throughout the report, we present tables showing
variation across sites on specific aspects of data system
development and use and provide examples of how
particular cities approached challenges. These cases
highlight successes and areas for growth in system
building. The nine afterschool systems that we examined
were in various stages of development during our first wave
of data collection, and we draw more heavily on examples
from more advanced data systems in this interim report.
We will continue to document the ways in which these
city afterschool systems are developing the capacity to
use data and produce a final report in 2017 which will add
to the knowledge base of how afterschool data systems
evolve over time, as well as the factors that influence their
development, adoption, spread, and sustainability. The
next phase of the work will expand to explore the views
of afterschool stakeholders about the value, relative to
the costs, of investing in data use for improving program
quality, access, systems, and youth outcomes.

Table 1.

Aspects of a Data System Framework

People
Stakeholders (individuals, institutions,
organizations) involved in the operation of
the afterschool system and the dynamic
connections among them
l	 Staffing: Staff time dedicated to the creation
and maintenance of the data system
l	 Human capital: Skills, knowledge, or
expertise of individuals, institutions, and
organizations involved in the creation and
maintenance of the data system
l	 Roles and functions: Distribution of
responsibilities among individuals,
institutions, and organizations for the creation
and maintenance of the data system
l	 Partnerships: Formal contracted or
documented connections between individuals,
institutions, and/or organizations involved in
the data system that facilitate coordination,
collaboration, or sharing
l	 Power: Individuals, institutions, and
organizations who shape the direction, goals
or the course of events related to the creation
and maintenance of the data system
l	 Relationships: Informal or personal
connections between individuals, institutions,
and/or organizations involved in the data
system that leverage trust or historical
engagement to facilitate coordination,
collaboration, or sharing

Processes
Routines, norms, and/or practices that evolve
and are repeated over time and include the
collection, organization, analysis, interpretation,
and use of data to meet the goals and inform the
operation of the afterschool system
l	 Indicators and tools: Identifying and refining
data elements to include in the system,
instruments to collect them, ways in which
they will be reported
l	 Data governance: Formalizing and enforcing
the standards for data sharing, transfer,
reporting, and use, which are aligned with
state and federal regulations
l	 Training: Providing initial and ongoing
instruction in the collection, organization,
analyses, and application of data
l	 Data collection: Gathering data from
multiple sources (such as surveys and
other data systems)
l	 Analysis and interpretation: Transforming
data into usable knowledge

Technology
Means by which data are organized and accessed
to inform the operation of the afterschool system,
including the hardware and software systems, often
called management information systems (MIS)
l	 Database: Tool or platform where data
are integrated, stored, and accessed
and which outputs reports and other
usable forms of data
l	 Hardware/software (interface): Tools and
protocols used to access and review the data
l	 Hardware/software (infrastructure): Tools
and protocols used to store, integrate, and
manage the data
l	 Data visualization: Dashboards, reports, and
summaries from the MIS to support data use
l	 Capital investment: Funding mechanisms
to support the purchase, construction,
and maintenance of technology, including
contracting and compliance structures

l	 Compliance: Using knowledge to measure
progress towards goals for the purposes
of accountability
l	 Continuous improvement: Applying
knowledge to improve access, quality,
practices, and outcomes
l	 Communications: Sharing knowledge with
stakeholders in diverse formats that reflect
the way in which they use information

Connecting the Dots: Data Use in Afterschool Systems | 8

2. THE CONTEXT AND GOALS FOR DATA USE
“Ultimately, we seek a data system that can be used to support the best allocation of resources that we provide,
as well as program mentoring, program self-evaluations, and standardization of best practices.”
—Grand Rapids stakeholder

KEY POINTS

•	 The nine Next Generation afterschool
systems evolved in different contexts that
influenced the goals of the system and
populations served.
•	 Afterschool systems with strong partnerships
and/or existing coordinating entities were able
to advance their data use more quickly than
those who had to build partnerships and/or
establish a coordinating structure.
•	 Entering the initiative with an existing data
system was not necessarily an advantage,
as some cities struggled with adapting old
systems to new purposes.
•	 The majority of the Next Generation
cities were focused on using data to
address community goals and improve
program quality.

This chapter provides a foundational understanding of the
nine city afterschool systems participating in the Next
Generation Afterschool System-Building Initiative. We
begin with the contexts in which the afterschool systems,
including their data systems, grew. None of the Next
Generation cities began developing the capacity to use
data with a blank slate. They had to consider the people,
processes, and technology already in place and decide
what to capitalize on and what to change to realize their
larger system-building goals. We then describe the size
and scope of the Next Generation afterschool systems, the
coordinating entities charged with orchestrating them, and
the purposes that afterschool stakeholders in the cities
identified for their data system in 2014. The variation across
the nine cities, both prior to and during the Next Generation
initiative, affected both the successes and challenges
encountered in the trajectory of their use of data.

Contexts for Afterschool System Building
In the following discussion, we use the Aspects of a
Data System framework outlined in Chapter 1 to review
the context in which the Next Generation afterschool
systems developed.

PEOPLE

The afterschool systems in the Next Generation cities
generally include schools, parks and recreation programs,
libraries, and community-based programs, which may be
operated by a local organization including small, faithbased, and immigrant organizations, and larger, wellestablished multisite national organizations. Differences in
these organizations’ goals, structures, resources, funding,
and quality emerged as both facilitators and barriers
to coordination across the cities. As many of our informants reminded us, they often
contend with frequent staff turnover and lack of motivation for professional development
due to low salaries, few opportunities for career advancement, and limited program
resources—even within the large governmental settings.

•	 The systems faced similar challenges such as
changing city leadership, data privacy laws and
regulations, and diversity of goals and interests
among partners.

9 | Chapin Hall at the University of Chicago

Figure 2 shows the groups involved and their prevalence across the cities, while Table 4 at
the end of this chapter provides the names of the organizations identified by leadership
as important partners in 2014 system-building work. As shown in both the figure and
the table, city government departments, nonprofits, and school districts were important
partners in all of the cities. Below, we briefly discuss the impact of the interests and
capacities that some of these participants brought to the afterschool system. United Way
and community foundations were significant partners in six of the cities. These public and
private partners come with their own institutional missions, capabilities, and philosophies,
which may influence the structure and goals of the afterschool system. Across the cities,
there was variation in the roles of school districts, libraries, and other city institutions
in the afterschool system. These partnerships, in turn, influenced the nature of the data
systems that were developed.
The Wallace Foundation prioritized funding for participation in the initiative to cities
with strong mayoral leadership and investment in afterschool programs. This resulted in
governmental departments and agencies having a significant role in establishing the early
priorities for system building in all nine cities. As a provider, funder, and/or coordinating
entity, local government departments and agencies, informed by city leadership,
established the high-level priorities for youth and, in turn, were critical in shaping the
overarching vision for these systems.

Figure 2.

Foundational Partners in Next Generation Afterschool Systems, 2014
City Government
Nonprofit Agencies

PARTNER ENTITIES

School District
Park District
Police/Crime Prevention
Community Foundations
Research/Data Center
Public Library
United Way
0	1	2	3	4	5	6	7	8	9
NUMBER OF AFTERSCHOOL SYSTEMS

Connecting the Dots: Data Use in Afterschool Systems | 10

The cities with established partnerships between the afterschool system and public
sector, including the school district, appear to have had an easier time with their
initial system building than other Next Generation cities. Such strong partnerships
are particularly helpful given both the importance and the challenges of afterschool
system-school district partnerships. In many of the study cities, afterschool services
were provided in school settings and leveraged school resources, including classroom
space, access to students, utilities, and meals/snacks. Strong relationships and data
sharing agreements with local districts also expand afterschool systems’ access to data
sharing that can inform understanding of the afterschool systems’ performance and the
alignment of resources to the needs of students within the systems. Districts regularly
collect individual-level data on their students that can help answer afterschool providers’
questions about the youth they serve.
Relationships with the provider communities were critical to the design and implementation
of the efforts. Voluntary networks that were able to build on longstanding relationships also
seemed more likely to engage providers in their system-building activities. Stakeholders
in three cities spoke about the value of prior relationships for the new Next Generation
initiative. Grand Rapids’ Expanded Learning Opportunities (ELO) Network has engaged
providers, funders, government, and evaluators for almost a decade. Therefore, it benefits
from healthy partnerships with the provider community through professional development
and network meetings as well as solid relationships with local philanthropic funders and
the United Way. Sprockets in Saint Paul draws its influence from a history of engagement
with afterschool work and key partners. In contrast to Grand Rapids and Saint Paul, Fort
Worth faced challenges with engagement as they decided to establish a new nonprofit
intermediary, Strengthening Programs through Advocacy, Resources, and Collaboration
(SPARC), which required significant time and effort to form and then to engage the
provider community.

PROCESSES
Along with the existing partnerships and technology that predated the Next Generation
initiative, the nine city afterschool systems had prior or existing management,
communication, and training processes that provided the groundwork for establishing
practices and routines necessary to build and sustain a data system. In developing
processes for this effort, the cities also were affected by federal regulations governing the
privacy of data collected by schools and other entities, like afterschool programs, that
are “dedicated to enhancing the academic achievement of its enrollees,” as required by
the Federal Educational Rights and Privacy Act (FERPA).17 Local interpretations of these
privacy regulations had implications for the Next Generation cities in terms of many of
the processes involved in the development of data systems, including the time spent
negotiating or renegotiating data sharing agreements along with the trust and willingness
to find acceptable ways to facilitate data sharing.
At the start of the initiative, all but one of the city afterschool systems established a small
number of committees and work groups to attend to different processes of their system
building. Most often, one group was dedicated to the area of data or reliable information,
which included making decisions about the MIS or system database. A second group was
focused on implementing a plan for improving program quality. In many cities there were
leadership and coordinating councils, and in some cities, smaller executive committees
with responsibility for overseeing system building in all the domains. Our informants
valued the opportunity to connect with other partners and influence the partnership
through these meetings.
17.	 United States 2011.

11 | Chapin Hall at the University of Chicago

“They’re all buying in. The data committee is open to anybody who wants to come…
Basically all of the different committees that make up the work groups attend the
coordinating council and share and discuss, so if there are issues then all the players
are there to say, ‘I can’t agree,’ or, ‘I don’t know how you’re going to do that.’ The more
they understand, the more they buy in.” —Louisville stakeholder
TECHNOLOGY
Each of the Next Generation afterschool systems had prior
experience with technology and had developed or was in
the process of developing an MIS when they became part
of the initiative. As previously defined, an MIS includes the
technology hardware and software that stores, organizes,
and facilitates the use of data. Table 2 reflects the variation
of the technology in use by the cities at the end of 2014.
Specifically, three of the afterschool systems were using
and/or adapting existing or “legacy” MIS that predated
the initiative. The other cities were building new database
systems, either through an external software developer
or a partnership with a local research organization. As the
evolution of these databases was not straightforward, they
resulted in different types of MIS; for example, some cities
used a single database and others used multiple databases
that allowed for data to be linked.
In most of the cities, earlier attempts to build data
systems had been limited to the efforts of individual
afterschool organizations and providers to collect data—
often to comply with reporting requirements of funders—
with little intentional coordination of data collection
among providers. The technological systems for collecting
data were still evolving during the time period covered
by this report.

Table 2.

Technology in the Next Generation Afterschool
Data Systems, 2014
MIS (TECHNOLOGY)

NEXT
GENERATION CITIES

Existing database
(“in house” or proprietary)

Baltimore, Jacksonville,
Louisville, Nashville

New proprietary/packaged
database (Efforts to Outcomes
[ETO], CitySpan)

Philadelphia, Saint Paul

New database built “in house”
in collaboration with the
coordinating entity

Denver, Grand Rapids

System-level database not yet
developed

Fort Worth

COORDINATION STRUCTURES IN AF TERSCHOOL SYSTEMS
Each city identified a public or private intermediary or coordinating entity to manage
their system-building activities. In describing the governing structure of afterschool
systems, Deich, Neary, and Padgette list the following key functions of these coordinating
entities: engaging key community leaders and diverse stakeholder groups, providing
strategic direction, measuring progress and outcomes, and managing resources.18 The
Next Generation cities often pursued these goals by convening partners in workgroups
and committees. At the outset of the initiative, the nine cities had coordinating structures
that varied in their location, committee structure, and funding capacity. We explore these
variations and their implications on the following pages.

18.	Deich, Neary, and Padgette. Manuscript in preparation.

Connecting the Dots: Data Use in Afterschool Systems | 12

Table 3.

The Next Generation Afterschool Systems’ Coordinating Entities and Governance Structures, 2014

CITY

COORDINATING
ENTITY

TYPE OF COORDINATING
ENTITY

FUNDER

AGE RANGE SERVED

Baltimore

Family League of Baltimore

Nonprofit intermediary

YES

K–12

Denver

Denver Afterschool Alliance

Locally coordinated network

NO

K–8

Fort Worth

Park and Recreation
Department (to be transferred
to Fort Worth SPARC)

Local government (in transition to
nonprofit intermediary)

NO

K–12

Grand Rapids

Our Community’s Children
facilitating the Expanded
Learning Opportunities
(ELO) Network

Local government

NO

K–12

Jacksonville

Jacksonville Children’s
Commission (JCC)

Local government

YES

K–8

Louisville

Building Louisville’s Outof-School Time Coordinated
System (BLOCS; facilitated by
Metro United Way)

Locally coordinated network

NO

K–12

Nashville

Nashville After Zone Alliance
(NAZA)

Local government

YES

6–8

Philadelphia

PhillyBOOST

Locally coordinated network

NO

K–12

Saint Paul

Sprockets

Locally coordinated network

NO

K–12

13 | Chapin Hall at the University of Chicago

The coordinating structures of the Next Generation afterschool systems were housed in
nonprofit organizations, public offices or departments, or in locally coordinated networks
(as shown in Table 3). In a few cities, the coordinating entity involved representatives from
several public institutions in either a formal or informal structure. Formal structures imply
an agreed-upon partnership between governmental agencies to sustain the effort and
include official meeting schedules, decision-making processes, bylaws, and memoranda of
understanding to stipulate roles and responsibilities.19
Relying on informal structures early in the development of afterschool systems allowed
the leaders to test processes with a small group of partners or to take time, as was the case
in Philadelphia, to decide what kind of governance structure would best meet the needs of
their provider community. As systems grew, they appeared to benefit from the transition
to more formal structures, which clarified roles and processes for all partners, particularly
those just entering the system, allowing them to quickly engage in the work of the system.
Each system had an individual or a small team of people who oversaw the day-to-day
operations of the system. In some cases, this person or group was supported by full-time
staff; in others, this work was divided among staff with other responsibilities. For example,
in Denver, the three foundational partners—the school district, the Office of Children’s
Affairs, and the Boys & Girls Clubs of Metropolitan Denver—have staff time allocated to
the afterschool intermediary, the Denver Afterschool Alliance. These staff members work
together to facilitate meetings, fundraise, and share responsibility for communications.
Next Generation cities that housed afterschool coordinating structures worked to mitigate
the impact of mayoral and other leadership changes in agency partners such as the school
district and ensure sustainability of their afterschool systems. As mentioned, Fort Worth
stakeholders created a new nonprofit, SPARC, to ensure that the afterschool system would
get the priority and focus it required. Although at the end of 2014 the administration
and coordination of their system-building efforts were still with the Park and Recreation
Department, they are expected to transition to SPARC in the future. In Grand Rapids, the
Office of Our Community’s Children (OCC), a designated city government office for children
and youth initiatives, was created as a joint partnership of the City Commission and the
Board of Education. OCC works in partnership with the ELO Network, a voluntary provider
network, to provide oversight and daily management of the initiative. By building leadership
within the ELO Network, current leaders hope that future changes in city government will
not reduce the strength of the provider network. As another example, Nashville’s NAZA
in mid-2014 moved from the mayor’s office to the public library because city and system
leaders believed that as part of the library, the afterschool network would be more likely to
weather changes in city leadership.
Whether or not a coordinating entity funds programs was a critical determinant of the
strategies used to implement policies and practices that affected the cities’ system-building
efforts. In three of the nine cities, the coordinating entity was also the major funder of its
afterschool programs. In these cities, funding and accountability were viewed as effective
levers to encourage programs to adopt rules and standards. A Nashville stakeholder stressed
that the financial incentives and the "clout" that funding brings can "escalate the timeframe
[and] scale this thing up."
As stakeholders in these cities explained, funding leverage was particularly helpful when
it came time to obtain partners’ buy-in to the data system. In contrast, coordinating
structures without funding capacity and afterschool systems without a formal coordinating
structure, such as that in Philadelphia, could mandate that their own providers use the new
database but did not have the authority to get providers across the system to participate.
19.	Deich, Neary, and Padgette. Manuscript in preparation.

Connecting the Dots: Data Use in Afterschool Systems | 14

Another concern was sustainability. City stakeholders in systems in which the
coordinating entity is not a funder, such as in Fort Worth, Grand Rapids, and Saint Paul,
expressed concern whether providers in a voluntary network would remain involved if
they had to pay to participate in a system-wide MIS designed to collect and report data
on all participants. Without the levers of funding and accountability, these coordinating
entities had to develop alternative strategies to engage providers and partners. Grand
Rapids and Saint Paul involved partners in the design of the afterschool system so that
the system addressed their needs and concerns. Although these efforts could be time
consuming, some city afterschool leaders argued that consensus building helped to fortify
the data system. One informant described the process as “creating our own system in
a way that we know will be sustained within the community.” A number of informants
mentioned the value of data systems for bringing diverse partners together and fostering
collaboration instead of competition.

Goals and Purposes for Investing in Data Use
Given the overarching goals of the Next Generation initiative, the nine cities focused
their afterschool system-building activities on building quality, expanding participation,
strengthening coordination structures, and using reliable information. These systembuilding activities informed how cities defined their specific goals and purposes for
developing their capacity to use data.
Identifying appropriate indicators for afterschool programs and using them for
improvement rather than for compliance or accountability were two issues raised by a
number of stakeholders in the Next Generation cities. Many providers in the afterschool
systems were required to report data to funders for accountability purposes that are
often associated with “high stakes” that can influence funding for programs. However,
system leaders were intent on using data for “low stakes” to inform decision making for
the purpose of improving the system. In this regard, our informants talked about the use
of data to (1) improve outcomes for participating children and youth as a use that affects
the individual level; (2) support program staff and managers in improving program quality
as a use that impacts the program and organizational level; and (3) inform resource
allocation, engage public and private funders in supporting the system, and/or improve
the functioning and coordination of the system, as uses that impact the system level. The
most common purpose for initially investing in data systems, according to informants in
the Next Generation Cities, was to improve program quality and to respond appropriately
to student and community needs.
Aspirations at the system level included using data for (1) improving the system’s
alignment between the youth they serve and the youth they want to serve, (2)
expanding the overall number of youth who participate in afterschool programs, (3)
improving program quality across the network, and (4) uniting partners in the system
and reinforcing the value of system building. The uses of data informed the design of
professional development, the evolution of the MIS (both collection and representation of
data), and the strategies for engaging partners and communicating about the impacts of
the system-building efforts.

15 | Chapin Hall at the University of Chicago

Summary
The Next Generation afterschool systems arose in different contexts, in different
forms, and with varying goals. It is not surprising that the cities that already had strong
partnerships and existing coordinating entities were able to advance in their data use
more quickly than those who had to build or fortify partnerships. However, entering the
Next Generation initiative with an existing data system was not necessarily an advantage,
as some cities struggled with adapting old systems to new purposes—a challenge that will
be discussed further in the next chapter.
Although the purposes for developing data systems to inform and guide their system
building were influenced by stakeholders’ goals for the afterschool system, these purposes
were also influenced by a number of other factors, including their local political contexts,
the organizational structure of the system, quality of relationships among lead agency
staff, and the priorities of the large public organizations that were part of the afterschool
systems. The cities also faced common challenges and constraints that influenced their
purposes for investing in people, processes, and technology to develop their capacity to
use data. Some of these challenges were inherent in the nature of afterschool systems,
which typically encompass a heterogeneous group of providers. Other challenges
were created by federal regulations developed to protect student level data, increased
emphasis by funders to hold programs accountable for their quality and ability to engage
youth, staffing issues, and challenges in identifying the “best” measures for assessing
program outcomes, including academic or social and emotional competencies.

Connecting the Dots: Data Use in Afterschool Systems | 16

17 | Chapin Hall at the University of Chicago

Seeking an external research
partner; RFP completed
November 2014

Community Research
Institute (CRI)

•	 Parks & Recreation Department Fort
Worth Independent School Districts
•	 Mayor’s Office

•	
•	
•	
•	
•	
•	

Park and Recreation
Department (in
transition to Fort
Worth SPARC)

Our Community’s
Children
facilitating the
Expanded Learning
Opportunities (ELO)
Network

Jacksonville
Children’s
Commission (JCC)

Fort Worth
812,200

Grand Rapids
193,800

Jacksonville
853,400

Duval County Public Schools
Chamber of Commerce
Jacksonville Public Education Fund
United Way of NE Florida
Jacksonville Department of Parks,
Recreation and Community Services
•	 Community Foundations

University of North FL Center
for Community Initiatives (CCI)

Civic Canopy (manager of
Community Partnership
System (CPS) database
developed by CiviCore)

•	 Denver Public Schools (DPS)
•	 City Office of Children’s Affairs
•	 Boys & Girls Clubs of Metro Denver

Denver Afterschool
Alliance

Denver
663,900

•	
•	
•	
•	
•	

Baltimore Education Research
Consortium (BERC)

•	 Baltimore Public Schools
•	 Mayor’s Office

Family League of
Baltimore

Baltimore
622,800

Grand Rapids Public Schools
Mayor’s Office
Heart of Western Michigan United Way
Kent School Services Network
Community Research Institute
Doug & Maria DeVos Foundation

EXTERNAL DATA
AND RESEARCH
EXPERTISE

FOUNDATIONAL
PARTNERS

COORDINATING
ENTITY

CITY AND
POPULATION
(2014 EST.)21

Overview of Next Generation Afterschool System Partners and Scope, 2014 20

Table 4.

Subset

All

All

All

Subset

NETWORK SCOPE (ALL
OR SUBSET OF PUBLICLY
FUNDED PROVIDERS)

27 organizations;
69 JCC-funded programs
entering data into the MIS

The ELO Network includes
60 organizations serving
more than 21,000
children at 180 sites; 20
organizations with 41
programs entering data

40 organizations with 240
programs or sites, none of
which were entering data
by the end of 2014

Demonstration project
cohort: nine sites (11 orgs.);
Capacity-building cohort:
32 sites (nine orgs.);
CPS data users: 80 orgs.

29 organizations with 51
Community School sites
and 45 out-of-school time
programs; 88 programs
entering data

NUMBER OF NETWORK
PROVIDERS AS OF 2014*

Connecting the Dots: Data Use in Afterschool Systems | 18

Nashville After Zone
Alliance (NAZA)

PhillyBOOST

Sprockets

Nashville
644,000

Philadelphia
1,560,300

Saint Paul
297, 600

Metro United Way
Jefferson County Public Schools
Louisville Metro Government
Louisville Metro Alliance for Youth

Metro Nashville Public Schools (MNPS)
Mayor’s Office of Children and Youth
Metro Council
Community Foundation of Middle
Tennessee
School District of Philadelphia
Department of Human Services (DHS)
Parks and Recreation
Free Library of Philadelphia
After School Activities Partnership (ASAP)
The Police Athletic League (PAL)
Catholic Archdiocese Mission Schools
Saint Paul Public Schools
Mayor’s Office
Parks and Recreation
Saint Paul Public Library
Augsburg College
YWCA of Saint Paul

•	
•	
•	
•	

•	
•	
•	
•	

•	
•	
•	
•	
•	
•	
•	
•	
•	
•	
•	
•	
•	

FOUNDATIONAL
PARTNERS

Amherst H. Wilder
Foundation

Metro Information
Technology Services (ITS)
Pilot project – American
Institute for Research (AIR)

EXTERNAL DATA
AND RESEARCH
EXPERTISE

All

All

Subset

All

NETWORK SCOPE (ALL
OR SUBSET OF PUBLICLY
FUNDED PROVIDERS)

44 organizations with
135 programs or sites; 38
organizations participate
in data system

72 organizations with
328 programs entering
data (131 of these
entering data into DHS'
Provider, Contract,
Attendace, Performance,
and Payment System)

15 organizations with
43 sites in five zones
entering data

54 school-based
providers are in school
district database; 48
entering data

NUMBER OF NETWORK
PROVIDERS AS OF 2014*

21.	 United States Census Bureau. American Fact Finder. Last Revised: February 23, 2015. http://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml.

20.	All of the Next Generation cities’ system-building efforts benefited from partnerships with the provider community. Due to the number and evolving nature of those partnerships, the names of the providers are not listed; however, the relationships and partnerships with the
providers were critical to the system-building efforts.

*Network as defined for the purposes of the Next Generation initiative. Source: City Year 3 progress reports (December 2014). Most cities expanded the scope of their systems between 2012 and 2014 and sought to
grow in subsequent years; these numbers reflect providers most actively involved in Next Generation system-building activities. Several cities began implementation with pilot projects involving a small number
of programs, but intended to expand to include all publicly funded providers in the city. Systems only engaging a subset of providers were also the ones in which the coordinating entity was a funder. Data on the
number of programs were provided by system leadership.

Building Louisville’s
Out-of-School Time
Coordinated System
(BLOCS; facilitated by
Metro United Way)

COORDINATING
ENTITY

Louisville
612,800

CITY AND
POPULATION
(2014 EST.)21

Table 4. continued

3. BUILDING CAPACITY TO COLLECT AND ORGANIZE DATA
“The mayoral task force looked at what afterschool systems in other cities were doing, and it was really clear
that you’ve got to have leadership, you’ve got to have data, and you’ve got to focus on quality to translate into
benefits for young people. We invested hours and hours into thinking about program quality and data sharing,
and it’s been hard and frustrating and takes more time than anyone has, but I don’t regret it one bit because if
our system survives it will only be because we grounded it in those things.” —Nashville stakeholder

KEY POINTS

•	 Selection of data to be collected in the MIS
was influenced by afterschool system goals,
available data, capacity of the MIS, and
feasibility for partners to collect.
•	 Cities were able to access protected
academic data by developing data sharing
agreements that met their city’s terms for
data privacy or by housing data with external
research partners.
•	 Identification of the data elements required
the development of common definitions
of measurement.
•	 Identifying strategies to minimize burden
of data entry on providers was critical to
engagement and buy-in.
•	 Ensuring the quality of data in the MIS
required people and processes, including
ongoing training and monitoring.
•	 Cities well along in the development of
their data systems were still refining ways
to present data for learning, improvement,
and new trainings to build those more
advanced skills.

19 | Chapin Hall at the University of Chicago

Although cities varied in their purposes for data
use within their individual community and system
contexts, they all made investments in people,
processes, and technology to develop their capacity
to collect and use data to enhance their programs
and systems. In this chapter, we describe the
Next Generation afterschool systems’ strategies
for collecting, organizing, and managing data.
Common processes involved in the identification,
collection, and management of data in afterschool
data systems, which emerged from our 2014
interviews with system-level stakeholders, are
presented in Table 5.
Stakeholders often had to attend to multiple
tasks concurrently as one choice could affect
many others. For instance, decisions about
technological components and processes involved
in the collection, analysis, and uses of data had
implications for designing and implementing
other components of the system. In the cases
where cities were building an MIS, the selection
of data elements informed the construction of
the technology infrastructure. In another instance,
establishing a formal data sharing agreement was
predicated on identifying specific data elements in
addition to resolving the regulatory issues.
Recognizing that most cities were simultaneously
addressing multiple components to advance their
data collection and management systems, for
ease of discussion we talk about each of these key
elements and their associated tasks individually.

Table 5.

Processes and People in the Development of Data Systems
DATA COLLECTION PROCESSES

KEY ACTIONS

PEOPLE INVOLVED*

Assessing and engaging the network

Engage key stakeholders to define data
needs for the system

System-level leadership

Assess the needs and goals of providers
in the system

System-level leadership

Selecting data elements

Identify data elements

System leadership; afterschool providers

Implementing a management
information system (MIS)

Build or buy an MIS or adapt the
existing MIS

System leadership; technical staff

Establishing formal agreements
among network partners

Develop data access governance
through agreements and memoranda of
understanding (MOU)

System leadership with legal consultation

Preparing and training the network

Provide training on data collection and
the MIS

System leadership and staff; technical staff;
research partners

Engage provider staff on the data use to
inform practice

System leadership and staff; afterschool
providers; research partners

Establishing and maintaining
data quality

Define data collection and monitoring
processes

System leadership and staff; technical staff;
research partners

Getting data flowing

Develop data dashboards and
report format

System leadership; technical support;
research partners

Populate dashboards and reports
with data

System leadership; technical support;
research partners

*Examples of system leaders are initiative managers, leaders of coordinating entities, partners with decision-making authority representing mayors’ offices,
school districts, and city institutions and agencies. Technical leaders include both internal knowledge and data managers and external research partners or
contracted entities.

Connecting the Dots: Data Use in Afterschool Systems | 20

Assessing and Engaging the Network
Engaging the network was the first step in determining the overarching goals of the
system, the purposes for data, and the needs of potential provider participants. An
afterschool system stakeholder in Baltimore discussed the need to create a “common
vision” for the system and an understanding of how each aspect of the system informs
one another, stating:

“[F]irst, they would say, ‘What are the goals [and] why do we want to share our data?’…
You have to have those conversations and come to some agreement about what you’re
sharing data for because once you have a common vision for what you’re trying to
get to and what data you need to get there, then there might be costs around training
[and] capacity development. [Once we agree] these are the data points we want, we
need to know how to collect them so you have to have an infrastructure around that;
the protocols and practices for collecting the data. And folks need to be trained; there
needs to be follow up and support for some people, maybe some coaching.”
More than half of the Next Generation cities talked about conducting needs assessments,
surveys, and focus groups, or hosting community forums as part of their process for
engaging partners and reaching consensus on providers’ data needs. In a few cities, the
coordinating entities worked with a research partner to survey providers to obtain
this information. For example, Fort Worth worked with staff at the Forum for Youth
Investment to conduct a “partnership characteristics survey” that surveyed both providers
and system-level stakeholders to define the goals and scope of the system. “[It] will be
an online survey [for] the providers and others like the advisory committee and folks like
that, the board, etc. [It will be] a way to figure out what they think Fort Worth SPARC
should look like moving forward and then help us establish some agreement,” explained a
stakeholder. Surveying the network also helped to assess the current capacity of providers
to collect data and the additional support they might need to do so. According to another
Fort Worth informant, in planning for an integrated system for data use, it was important
to find out “what people are collecting and what they would be willing to collect, what
they have access to and what they need, and the ability of their staff to actually collect
it, enter it.”
Cities engaged diverse and representative groups of providers, including existing
organizations with a shared fund such as United Way, small community providers, and
large, well-established providers such as the YMCA or Boys and Girls Club. The aim was
to make the system and associated processes relevant for all providers, not just a select
few. When Jacksonville held focus groups with providers to get feedback on its current
database, staff tried to include unfunded as well as funded providers to get their input. As
a stakeholder explained, “We didn’t want to pick a standard that works great for the big
guys but just doesn’t help the little guys at all.”

Selecting Data Elements
A critical task in developing a data system is identifying data elements. City stakeholders
considered data that were already being collected as well as elements not currently in the
system that were viewed as essential for addressing critical questions. Study cities varied
in how they ultimately balanced between using existing data elements and introducing
new ones. Across all of the cities, there was particular interest in including existing school
data, such as information on demographics (e.g., free or reduced-price lunch eligibility,
which served as a proxy for family income), academic progress (grades, test scores,
special education placements, etc.), attendance, and behavioral infractions. A majority
of the cities also planned to continue to collect data on youth program participation or

21 | Chapin Hall at the University of Chicago

attendance and program quality—or to implement new measures of program quality if
they were not already collecting these data. Figure 3 describes the types of data cities
reported they had prioritized for collection at the end of 2014.
In each of the Next Generation cities, we spoke with stakeholders who believed that
social-emotional learning (SEL), well-being, and development were important outcomes
to track in the afterschool field, although only some of the cities incorporated socialemotional indicators in their data systems at the end of 2014. These stakeholders, in most
cases, considered these indicators to be more appropriate for the outcomes that could
be expected from afterschool programs and more easily demonstrated. As an informant
in Louisville observed, “We’d like to be able to say that the test scores are going up and
[school] attendance is going up for our kids that attend out-of-school time programs, but
that’s really hard to do sometimes.”

Figure 3.

Types of Data Selected for System-wide Collection, 2014*

TYPES OF DATA

Program Attendance

8

Program Quality

7

School Data
Youth Development (SEL)

6

5

0	1	2	3	4	5	6	7	8	9
NUMBER OF AFTERSCHOOL SYSTEMS

*Data elements that were being collected system wide or by a subset of providers in 2014 and likely to be part of system-wide data collection.
One city, Fort Worth, had no MIS at that time.

Across the nine cities there was variability in the sequencing that cities engaged in to
identify their data elements and to select their technology solution. In some cases, city
stakeholders discussed what data elements to collect while simultaneously considering
their options for the technology that would be used to collect and process them. In other
cases, decisions were made about the data elements before selecting the MIS. Two cities,
Nashville and Louisville, started with data that were already being collected through
existing data systems (i.e., school districts) with the potential for collecting other data
elements later. Because the selection of data elements was connected to the selection of
technology, we return to this topic in a later section.

Connecting the Dots: Data Use in Afterschool Systems | 22

THE INFLUENCE OF KEY STAKEHOLDERS IN
SELECTING DATA ELEMENTS
While cities often invoked strategies to engage a diverse set of partners in systembuilding efforts, it is undeniable that some partners exerted greater influence on these
efforts. Given the prioritization by The Wallace Foundation to select cities with strong
mayoral leadership and support, the leadership agenda of those mayors influenced the
original philosophy of the system and the types of data collected.
For example, Saint Paul’s long-standing positive orientation to supporting nonacademic
domains in afterschool programs was noted by several afterschool informants. According
to one stakeholder, “The mayor has a really deep interest in social-emotional learning. He
really believes that one of the strong functions of afterschool is helping kids develop other
capacities beyond academics. So, the more ways that the Sprockets afterschool network
can help organizations support the full development of kids, the more effective it’s going
to be. That’s a really a strong value of ours.” Similarly, when the Nashville Afterschool
Zone Alliance (NAZA) was first established and operated out of the mayor’s office, the
mayor’s interest in reducing the rates of high school dropout influenced the network’s
decision to focus resources on middle school students in order to provide early support to
students at risk of dropping out later in their school careers.
As school districts traditionally have been the primary organization collecting data on
youth in a systematic and standardized manner, they also were central figures in many
afterschool networks’ data systems and exerted a good deal of influence on the system as
a whole. Because of this influence, city afterschool systems often adopted the priorities
of the district (or combined them with that of the afterschool system) and selected
data elements that were closely aligned with that of the school district. A Nashville
stakeholder said:

“The system leaders were interested in aligning NAZA with the outcomes of the school
district…The mayor really wanted to put something together for the middle school
students that would keep them engaged, improve performance, and, ultimately, improve
the graduation rate. So we took the dropout indicators…[and] set the thresholds for
attendance, grades, and discipline…The indicators for how NAZA is doing, who they
should gear their programs toward, and what kind of outcomes we wanted to see were
wrapped up in the three indicators the school district is looking at.”
NAZA leaders intentionally aligned its data collection and outcomes with the school
district’s priorities. Initial data goals of the system focused on three early warning
indicators that research suggests can flag youth at risk for dropping out of school.22
According to an informant, “We looked to [the school district] to kind of set some of
the measures that they were tracking, and said, ‘You know what? If we’re going to be
in this together, we’re going to adopt those same measures.’ And so, for now, they
have identified school behavior, attendance, and academic performance as their three
primary indicators.”
Some cities prioritized data elements by identifying a core outcome of interest. Like
Nashville, afterschool leaders in Baltimore selected chronic absenteeism, an issue
of central concern to the school district, as the target outcome for the afterschool
system. They held multiple meetings to determine the threshold at which data would
become “actionable” for program staff, given that “chronic absence” was defined in the
school system as a cumulative number of absences of 20 or more over the course of a
full year. They then designed the afterschool data system to flag students with multiple
22.	Balfanz et al. 2014.

23 | Chapin Hall at the University of Chicago

consecutive absences (e.g., three days in a row), creating a trigger for outreach to an at-risk
student before his or her absenteeism became chronic. This focus also aligned to a city plan to
streamline publicly funded youth services by reallocating resources and strengthening connections
between schools and afterschool. Both the city and the afterschool system benefited from this
connection. The city and school district increased their support of the afterschool system, and
the afterschool system helped city-level stakeholders understand “what was happening all day
long for our kids, what was going on in the schools, and what the needs were for the community,”
according to a city stakeholder.
Several of the Next Generation cities argued that investing in quality afterschool programs
is a way to prevent delinquent behavior. To support this goal, Fort Worth, Grand Rapids, and
Louisville involved police departments and organizations focused on preventing or reducing youth
involvement in the justice system. The involvement of these partners, in turn, influenced their data
plans. For example, the ELO Network in Grand Rapids has been working for almost a decade with
the police department to develop a juvenile offense indicator to determine whether afterschool
investments were having an impact on juvenile crime. In the network’s deliberations about what
data to collect in its new MIS, some stakeholders argued against its use, concerned that it would
not be possible to attribute change in this indicator to participation in afterschool programs with
existing measures. Additionally, some felt it would not accurately reflect the success of their
programming. Others, including the city’s Our Community’s Children (OCC) office, favored using
the indicator because it aligned with the city’s interest in reducing juvenile delinquency and would
have broad-based appeal for funders as well as the community. Because of the city’s apparent
influence, the decision was made to collect and report on these data.
Regardless of how they selected data elements, system leaders reported that they needed to
establish common definitions. Both Baltimore and Grand Rapids informants described lengthy
discussions on how to define and calculate “attendance” for their data systems. Grand Rapids
appears to have gone through the most intensive process with its partner, the Community
Research Institute (CRI), to develop data dictionaries to help its technical staff catalog and
reconcile differences among providers in the way they defined and collected data elements. To
make it possible to include each of the measurement strategies in the system, CRI developed a
way to convert the provider data into a common “dosage” measure.

Establishing a Management Information System (MIS)
“Ultimately we seek a data system that can be used to support the best allocation of resources
that we provide, as well as program mentoring, program self-evaluations, and standardization
of best practices.” —Grand Rapids stakeholder
In order to use the data elements prioritized by the afterschool systems, cities had to develop
and implement an MIS. Consisting of both hardware and software, the MIS is the part of the
data system most readily identifiable with the technology aspect of afterschool systems’ data
use. However, a close examination of the experiences of the nine cities revealed that skills and
expertise (people) and rules, norms, and standards (processes) were necessary to integrate the
technological functions of the MIS with the larger goals and functions of the data system.

T YPES OF MIS
Two major types of MIS emerged from the interviews with the sites. One type is self-contained,
in which the technology, tools, and skills for data access and use are located mainly within the
afterschool system or an entity directly contracted by the afterschool system, such as a research
partner. A second type is a composite, in which the database, tools, and skills for data access and
use are located mostly in an adjacent system, typically the school district, with other data housed
in external data management tools such as Excel.

Connecting the Dots: Data Use in Afterschool Systems | 24

Self-contained MIS
Self-contained MIS are systems with a single, primary database that is managed and
coordinated by the afterschool system, sometimes with the support of an external
research partner or other contracted assistance. Data collected by providers are
merged and may be integrated with data from other systems. Self-contained systems
were most likely to be located in cities where the afterschool system had its own
distinct goals or where a system accommodated a wide range of provider goals
and potential outcomes that did not lend themselves to coordinating with other
systems. Self-contained MIS also were more common in cities with strong data and
technical skills that could design and manage changes and updates to the system.
Cities with a self-contained MIS and limited internal supports (e.g., technical skills,
staffing) frequently contracted with research partners or another outside entity to
manage their system.
Four of the Next Generation cities (Baltimore, Jacksonville, Grand Rapids, and Saint
Paul) adopted a self-contained MIS and had a functioning system at the end of 2014.
The experiences of these cities highlight key considerations in managing the design
of self-contained MIS. One city sought to systematically contract out the tasks to
different entities and centrally coordinate the effort. Another used expertise within
the system to customize an existing system to a new set of needs and purpose with
assistance from a research partner who coordinated or advised most of the major MIS
development and operation tasks.
The Jacksonville Children’s Commission (JCC) used an existing system,
SAMIS, in part because the JCC was still under contract with the
developer and making a change had cost implications and
because it was familiar to its afterschool providers. In
this instance, the choice of SAMIS was intended to
build on the technical skills that already existed
Figure 4.
in the afterschool system as well as the efforts
Types of System-wide MIS Selected by
to engage and train providers. The challenge
the Next Generation Cities*
for Jacksonville was aligning the existing
structures of the MIS with the goals of the
afterschool system as a whole. SAMIS
was designed as a financial management
and compliance system and not for
the particular needs of the afterschool
Composite:
Self-contained:
system, which includes the capacity to
3
4
measure participation and outcomes.
Thus, although JCC system leaders
planned to continue to use SAMIS
for the foreseeable future, they also
began exploring other options to either
significantly upgrade the database or
replace it with a proprietary MIS.
*Two cities (Fort Worth and Louisville) had not
established their MIS at the end of 2014.

25 | Chapin Hall at the University of Chicago

Grand Rapids and Saint Paul viewed a self-contained system as one that would be
both low cost to providers and “low stakes,” meaning that data would not be used for
funding decisions. Both cities leveraged existing relationships with a research partner to
specifically define their goals and contract the entire process, from design to training to
implementation and support, to a single entity connected to the system. To illustrate,
contracting with a research partner allowed Grand Rapids to balance a number of their
conditions for the MIS that initially seemed in opposition: low-cost technology, minimal
burden on providers, adaptable to a wide variety of providers, and capacity to collect and
coordinate data rigorous enough for a meaningful evaluation of the impact of afterschool
participation. In selecting a system supported and managed by CRI, the ELO Network
was able to provide tailored support to the providers. As described by one stakeholder,
the ELO Network is a voluntary network that includes “everything from multimilliondollar operations with IT departments to three people that roll out the basketballs every
night who are keeping attendance records on pieces of paper.” To accommodate this
heterogeneity, afterschool leaders worked with CRI to build a customized system that
allows providers to submit various data formats and measures. According to another
informant, “Whatever data collection methods that the programs use, we’re going to try
to accommodate that. It’s an ambitious goal, but that way they don’t have to change or
buy software to accommodate the needs of the project.” This decision required significant
technical capacity to align the systems, but it met the needs of a diverse range of partners
and encouraged their participation in the network.
Although self-contained systems allowed cities to customize the MIS to their particular
needs, they were not without their challenges. The biggest challenge for cities designing
a self-contained system was managing the initial phase of developing the MIS and
deciding what technology would best suit their needs. City stakeholders had to clearly
define and prioritize their goals, identify the needed skills and expertise, and then decide
what technology would best meet their needs. In some instances, cities that chose a
self-contained system were starting with a blank slate when considering the options
for system-wide technology. Stakeholders reported that this type of startup required a
tremendous amount of energy to coordinate efforts and upfront resources to hire skilled
individuals to manage the process, align technology, and develop the necessary processes
to ensure that the MIS captured and produced high-quality information. Of the four cities
that chose to pursue a self-contained MIS without a prior data system in place, only one
had data flowing by the end of 2014.
Another significant challenge in building a self-contained system was cost. Systems
that opted for a self-contained MIS had to manage the full cost of establishing and
maintaining the system themselves. In the case of Grand Rapids and Saint Paul, the use
of a research partner was a major facilitator in acquiring technology and developing
processes that were well tailored to a set of shared system goals. However, the use of a
research partner was a significant resource investment.

Connecting the Dots: Data Use in Afterschool Systems | 26

Composite MIS
Composite MIS are most likely to be housed in afterschool systems where the necessary
skills and expertise are located in a partner system. Other databases complement the
information stored by the partner system. Because composite systems have the benefit
of working with existing and functioning infrastructure, they can often get data flowing
through the system more quickly than other types of systems. Additionally, adapting
technology currently in use reduces startup costs.
Three of the Next Generation cities (Denver, Nashville, and Philadelphia) chose a composite
MIS.23 Denver and Nashville both worked with MIS in their local school districts. At the end
of 2014, Nashville’s MIS was functioning well, with all NAZA providers entering data into
the database. Metro Information Technology Services helped build a web form that created
access to the Metro Nashville Public Schools data warehouse for NAZA providers. Providers
were able to access reports from the school district database, although they had to do so in
the presence of a designated school district employee.
A Nashville stakeholder cited the ability to use existing resources as “a primary
consideration” and “one of the driving forces” for selecting a composite MIS, adding, “We
wanted to make sure that most of our money went into providing afterschool seats for kids,
and not spend too much on infrastructure if we didn’t need to.” The Denver Afterschool
Alliance built an afterschool data structure that includes the Community Partnership
System (CPS) that is also used by the school district, and associated MIS that house quality
measures (e.g., the Youth Program Quality Assessment) and youth engagement indicators
(e.g., the Survey of Academic and Youth Outcomes).24
In Philadelphia, the MIS managed by the city’s Department of Human Services (DHS), the
largest provider in the afterschool system, continued to be the database that DHS-funded
providers had to use. However, this system was not flexible enough to easily add users
outside of the DHS network of providers. According to an informant, “It took us months just
to add one library branch.” Thus, system leaders decided to transition to a self-contained
system, which may be more adaptable to the needs of a wide range of afterschool
providers. Towards that end, they engaged in an almost yearlong process to select Efforts
to Outcomes (ETO), a software vendor working with other city agencies, as the MIS for the
PhillyBOOST afterschool system. As of the end of 2014, they had completed a pilot of the
database with a range of publicly funded agencies and were trying to build commitment to
continue using the system among its key partners.
Composite systems seem to be more successful when the afterschool system goals and
indicators were well aligned with the goals of the organization that maintains the MIS.
When they function well, as in Nashville and Denver, composite MIS also appear to have
the potential to strengthen and support relationships between systems. There is strong
potential for shared reporting, collective meaning-making, and coordination around
planning programming for youth. However, an afterschool system cannot control or dictate
the priorities of the agency housing the MIS and may have limited control over features and
processes of the MIS or their goals and outcomes.
23.	As noted in Figure 4, the afterschool systems in Fort Worth and Louisville did not have an established functioning MIS at the end of 2014. Although
Fort Worth had not yet selected an MIS at this time, Louisville had, but was still trying to negotiate access to the data system for all of their providers.
Afterschool system leaders in Louisville had reviewed a number of different options and selected the MIS used by the school district, named CASCADE.
Using this existing system was appealing because it had demonstrated success, was adaptable to a variety of data collection needs, and had the
potential to provide relevant individual-level data to providers that might assist with instructional programming. However, school district policies
limit which afterschool providers were permitted to access CASCADE. In order for providers to access CASCADE (and, by extension, individual-level
student data), providers had to demonstrate that they met the school district’s criteria as a provider of individual-level academic support that aligned
with the school district’s curriculum and instructional strategies. The school district did not consider all of the providers in the afterschool system as
meeting these standards, which meant that some providers were not able to enter their program attendance data into CASCADE.
24.	For additional information on the Youth Program Quality Assessment (YPQA), visit http://www.cypq.org/assessment. For additional information on the
Survey of Academic and Youth Outcomes (SAYO), go to http://www.niost.org/Training-Descriptions/survey-of-afterschool-youth-outcomes-for-staffand-teachers-sayo-s-and-sayo-t.

27 | Chapin Hall at the University of Chicago

The most significant challenge facing composite systems appears to be the extra level
of coordination required to ensure that data elements can be integrated into actionable
information aligned to common goals. Without a larger partnership or governance
structure, partners within the system can drift away from common tasks and goals. This
challenge was best illustrated by the experience of Philadelphia. Leaders of the effort
readily acknowledged that the PhillyBOOST initiative did not yet have, but needed,
a dedicated intermediary to promote and coordinate use of a central MIS. Without
appropriate coordination, composite systems may also experience duplication of effort,
which can limit system efficiency and frustrate stakeholders, although in the case of
Philadelphia, DHS providers were not required to enter data into both the DHS and ETO
databases. Another potential challenge with a composite MIS was ensuring the users of
more than one system were trained in the use of both systems. To address this concern,
Philadelphia developed distinct training and guidance for DHS providers. In contrast,
Denver Afterschool Alliance developed a highly structured and formalized process of data
integration across the systems that aligned to its core mission. They have not experienced
the same challenges in engaging nonprofit providers in the use of the CPS after resolving
initial technical issues.
At the end of 2014, seven of the nine cities had a working MIS that was collecting,
processing, and reporting data at a system level. Some cities expressed satisfaction with
their plans, while others were beginning to experience a need to “reset” their MIS. Figure
5 captures the status of the MIS in each of the nine afterschool systems at the end of
2014. Fort Worth had yet to select an MIS and Louisville was working to expand access
for all of its providers for its MIS. Jacksonville was exploring and negotiating with new
vendors to replace its MIS or acquire database technology to extend their current capacity,
while Philadelphia was working to pilot, promote, and expand use of ETO among its
major partners while DHS providers continued to use its database. At least three cities—
Baltimore, Denver, and Grand Rapids—were actively piloting or working on updates or
improvements to an existing MIS. Finally, Nashville and Saint Paul had established MIS
and were not prioritizing change or improvement.

Figure 5.

MIS Status of the Next Generation Afterschool Systems at the End of 2014*

STATUS OF MIS

Piloting and improving existing MIS
Actively looking for new or extended MIS capacity
Established MIS w/out immediate plans for change
In the process of contracting or selecting an MIS
0	1	2	3	4	5	6
NUMBER OF AFTERSCHOOL SYSTEMS

*Two cities, Philadelphia and Jacksonville, are represented twice in this figure. Both systems were actively looking for new MIS capacity
and working to improve an existing MIS.
Connecting the Dots: Data Use in Afterschool Systems | 28

None of the afterschool systems were completely satisfied with its MIS, and all saw
ways in which they could be improved. Moreover, in every city, system stakeholders
reported that establishing an MIS was a different and more complex task than they
imagined it to be at the proposal stage. In addition to technology, there were several
critical aspects related to the processes and people that are part of a data system, as
described in Chapter 1, which factored into the cities’ selection and establishment
of the MIS for their afterschool systems. These included assessing their context and
resources, locating appropriate expertise, defining their data use goals, and identifying key
considerations such as cost.

COMMON PROCESSES ASSOCIATED WITH BUILDING
AND SUSTAINING AN MIS
Several common processes—referred to as “tasks and activities”—emerged from our
analysis of the 2014 interviews. Although they varied relative to where they occurred in
the flow of data through an MIS, the following processes were commonly associated with
managing a database:
•	

Hardware/Software (infrastructure) (technology): Managing the technical aspects of
the hardware and software, i.e., the database and security.

•	

Hardware/Software (interface)(technology): Managing the user experience and
access to data.

•	

Data Visualization (technology): Designing and programming of queries, reports, and
other data access tools.

•	

Data Collection and Analysis (process): Performing a wide range of tasks that include
the transfer of data across servers, the management of extracts, and the system-level
queries and reporting about the contents of the database. Includes matching or linking,
where data from different sources describing the same program or individual are
combined in a single record.

•	

Data Quality Control (process): Standardizing identifiers and fields, tracking the number and
origin of records.

•	

Technical Support and Training (process): Instructing and assisting users.

Although stakeholders across the cities identified these as common tasks, how they were
accomplished and who was responsible for them varied. Many informants reported two
challenges: getting data flowing across systems and finding people with the right skills
to run and implement the systems. To develop and maintain their technical capacity,
afterschool system leaders worked with key partners, including school districts and city
agencies, external research consultants, and contracted MIS vendors. Many cities relied
on multiple sources to accomplish these tasks. For example, in three cities that partnered
with research organizations, individuals from those organizations provided a number
of different supports. They frequently were contracted to provide technical resources
(e.g., MIS management and infrastructure), support with processes (e.g., professional
development, access to data, and systems for managing data quality), and people (e.g.,
researchers and technology support).

29 | Chapin Hall at the University of Chicago

Challenges and Considerations in Selecting an MIS
Informants in all nine cities spoke of the complexities and challenges of selecting the “right”
MIS. Across and within the city afterschool systems, there were numerous parameters
and considerations that made different types of technology more or less attractive. Issues
of functionality and adaptability figured prominently in each city’s MIS deliberations. As
afterschool leaders explored different possibilities, some came to realize that the technology
they had originally selected was no longer the best option. Grand Rapids decided to build its
own in-house systems after concluding that their commercial system, for its cost, would not be
flexible enough to accommodate a wide range of providers using the system over the long term.
Philadelphia and Jacksonville both considered a new commercial system, ETO, to build out
new functions for their existing data system. Jacksonville formally assessed the capacities of its
existing MIS, determined it was lacking key functions such as reporting, and began searching
for alternatives. When Denver prepared to build its data system, afterschool system leaders
discovered that a local nonprofit organization, Civic Canopy, was already in discussions with
Denver Public Schools about a database platform that could link school data to data from a
range of social service providers. The platform was called the Community Partnership System
(CPS) and was designed to allow for a “data exchange” between DPS and social service
organizations providing services to DPS students (e.g., mental health). The school district could
learn which services individual students were receiving, and social service agencies could learn
how the students they serve were faring academically. The database, however, did not prioritize
the needs of any particular type of provider, so some data of interest to afterschool providers or
system leaders such as attendance were not included in the data system.
The factors mentioned most frequently as major considerations while in the process of selecting
and establishing an MIS are listed below. Table 6 provides an overview of how each of these
factors was experienced by the cities using each kind of MIS in 2014.
•	

Cost: Stakeholders in every city spoke of the need to manage cost as a primary consideration
in the task of selecting and establishing the MIS. Each kind of MIS presented unique challenges
and advantages with regard to cost effectiveness and depended on a city’s goals.

•	

Burden/duplication of efforts: Having the time and resources needed to engage with, manage,
or support the MIS were key issues for the cities. Most frequently this type of cost was associated
with duplication of effort. System stakeholders noted that providers with their own MIS were
sensitive to being asked to enter the same information in another system-level MIS. Provider
staff often felt overwhelmed by different requirements for data collection and cities used various
strategies to reduce the burden placed on staff whose time was in high demand. For example, a
number of cities structured their MIS to reduce this burden by creating look-up functions that
would allow providers to identify and “enroll” students rather than create new records for them.

•	

Access: Access to the MIS emerged as a theme across several cities in relation to their choice
of MIS. MIS purchased from vendors typically require licenses for use, which impose limitations
on access, such as how many organizations and sites may access data, how many users may
simultaneously use the system, and how many total users may access data. MIS located wholly
or in part in other systems also may be regulated by specifications or rules over which the
afterschool system does not have control.

•	

Timing, functionality, and adaptability: Afterschool stakeholders voiced several concerns
around timing. First and foremost were considerations of how quickly a system could get up
and running. Secondly, systems discussed how startup and maintenance phases might have
different requirements and constraints. Planning and allocating resources arose in several
discussions about how to create an MIS that was responsive to evolving system goals and
sustainable going forward.

Connecting the Dots: Data Use in Afterschool Systems | 30

Putting It All Together: Build, Buy, or Adapt
The biggest choices system stakeholders faced were selecting, contracting, and
establishing an MIS. The three most common paths to a functioning MIS are:
•	

Build: An afterschool system designs and builds a system from scratch.

•	

Buy: An afterschool system contracts with a proprietary vendor to acquire technology as
well as some processes and expertise.

•	

Adapt: An afterschool system uses the infrastructure of an existing system. It may also
use an existing interface or use some combination of in-house or contracted expertise to
customize the system for afterschool users.

Although it seems possible to build, buy, or adapt technology for any type of MIS, some
seem to lend themselves more readily to one strategy or another. It depends on how
well the afterschool system has identified its needs and goals for data, the capacity and
functionality of existing MIS and databases, and resources for internal or external research
and data expertise. Thus, the process of establishing an MIS is not easily reduced to a
single set of considerations or even a single strategy.

31 | Chapin Hall at the University of Chicago

Table 6.

Factors in Choosing an MIS: Themes in Afterschool Data System Development, 2014
TYPE OF MIS
FACTORS
Self-contained

Cost

Duplication of
Efforts

•	
•	
•	
•	

Cost borne fully by system
High startup costs
Potential for staffing cost
Cost to customize MIS solution to specific needs
of the system
•	 Maximal flexibility of features means you only
pay for what you want
•	 Greatest potential to outsource technical skill
needs rather than have to identify staffing
resources

•	 Additional investment needed for coordination
•	 Need to meet compliance standards and timelines of
another system
•	 Possibility of simultaneously paying for multiple
systems
•	 Cost of staff to integrate information across multiple
systems
•	 Designed to work with existing resources and
structures
•	 Limited startup costs
•	 Cost of technology and expertise shared or supported
by another system

•	 System may not have capacity to communicate
with existing technology
•	 System collects redundant information with
other similar systems
•	 Training on data quality and data entry not
connected to any other MIS
•	 Selection of a data warehouse, MIS type already
in use by other systems
•	 MIS that is compatible with provider MIS so
providers enter data into one MIS
•	 Research partner or internal expertise can
support MIS across systems

•	 MIS may not be able to accept or to bridge provider
and external MIS technology
•	 Data stored in multiple systems may entail duplicate
entry, especially for systems in transition
•	 Using existing resources reduces duplication of
efforts if providers are already entering data into the
MIS for other related programs
•	 Trainings on data entry and data quality may be
maximized as providers might enter multiple types of
data into the system

•	 Proprietary systems frequently require licenses
•	 If cost or licenses are not factors, system can
freely control access to MIS

•	 Access may vary by database and rules of managing
organization
•	 System that owns MIS may impose additional access
conditions
•	 System has multiple options for how access can be
granted
•	 System that owns MIS can freely grant access

•	 Tends to require heavy investment of time and
money in the startup phase
•	 System has full control over MIS and can design
it to be maximally responsive to needs

•	 Priority of afterschool system requests subject to
competing demands within the adjacent system
•	 Systems requiring coordination and/or in transition
might have more challenges in executing goals in a
timely fashion
•	 Changes in the MIS by the owner of the system may
impact timing to enter or access data
•	 Working with existing resources means systems are
up and running faster

Access

Timing

Composite

Connecting the Dots: Data Use in Afterschool Systems | 32

Establishing Formal Agreements Among Network Partners
Formal agreements outlining participation in the afterschool network and the associated
MIS were a critical component in the development and application of the MIS. In the
words of a stakeholder in Jacksonville, data sharing agreements “set the parameters, the
boundaries of everything we’re going to do.” Such agreements take the form of detailed
data sharing agreements and memoranda of agreement or understanding (MOA or MOU)
between network participants that outline the parameters of participation in the system.
Writing, developing, and establishing formalized data sharing agreements often took a
significant amount of resources.
Data sharing agreements often indicate levels of permission to access data. In the Next
Generation cities, these agreements typically specified the level of aggregation of the data
that providers can access, including individual-, program-, agency-, system- or city-level
data. Every afterschool system envisioned that providers have the ability to see aggregate
data for their youth participants, but most limited provider access to data that would
allow them to only make direct comparisons of their own youth to the broader system
and not directly to other programs or providers.
In creating data sharing agreements with providers in Saint Paul’s Sprockets network,
large multisite agencies specified the data site-level staff could access, resulting in
differences in levels of access among the agencies within the network. According to a
stakeholder, “Parks and Rec chose to say, ‘We don’t want someone at this rec center to be
able to see [information about] youth from this rec center.’ The YMCA has another model.
Any YMCA staff person can log in and see anybody who goes to any YMCA program.” In
Grand Rapids, the agreement was designed to provide access to aggregate data about all
of the programs in the ELO Network. According to a stakeholder, “If a program is nested
within another program, like the Boy Scouts of Harrison Park, they will get their individual
information and how they compare with all Boy Scouts programs in [the public schools].
Then they get their program compared to all ELO programs in the aggregate level.”

Box 2.

FERPA and Data Privacy
School district data can be an important source of information
about students for afterschool programs. Over the past decade,
there has been increasing interest in connecting afterschool
program attendance records with school data on student
attendance and performance.25
Nearly all of the Next Generation cities tracked program
attendance in their afterschool data systems and a few were
linking afterschool and school data. As of January 2012, the
amended Federal Educational Rights and Privacy Act (FERPA)
regulations allowed schools and other “so-called educational
programs” to enter into agreements to share data, including
“afterschool programs dedicated to enhancing the academic
achievement of its enrollees.” This policy change allowed for

25.	Afterschool Alliance 2014.

33 | Chapin Hall at the University of Chicago

greater discretion by school districts to use data in collaboration
with afterschool programs when that joint work could be justified
explicitly as a support to school goals.
At the state level, however, legislative activity in 2014 suggested
increasing concern about student data privacy. According to the
Data Quality Campaign, 30 student privacy bills were signed
into law in 21 states during 2014, and many other states were
considering similar kinds of legislation. The bills ranged in
scope from reiterating protections that already exist in FERPA to
preventing the collection of certain types of data to introducing
new data governance procedures. It’s important to note that four
of the Next Generation cities are in states that passed student data
privacy bills in 2014.

Any agreements to exchange student-level educational data had to satisfy the school
districts’ interpretation of the Federal Educational Rights and Privacy Act (FERPA).26
(see Box 2). Five cities were able to negotiate varying degrees of access and exchange
of student data from the school districts by the end of 2014. A sixth city, Jacksonville,
was able to obtain school district records for students in their afterschool programs by
providing student identification numbers to the school district.
In the Next Generation cities, people, in addition to processes, were critical to negotiating
agreements and access to data in four main ways: (1) willingness of legal counsels of
school districts to grant or deny the sharing of student data with afterschool programs
within the constraints of FERPA, (2) “champions” in the afterschool/school district
who successfully advocated for the sharing of school district data with afterschool
programs, (3) leadership at the school district believing that afterschool programs help
to accomplish the broader goals of the city and/or school district, and/or (4) research
partners who were seen as neutral and trusted collaborators. All city stakeholders
endorsed using multiple strategies to negotiate these agreements. For example, the
Denver Afterschool Alliance indicated that city government, champion leaders, and
attorney interpretation were critical to their success. Cities with key stakeholders in
government agencies also appeared to have an easier time of negotiating data sharing
agreements and accessing data.
In Denver, according to several informants, the process of developing an MOU with
the school district was collaborative. Denver Public Schools (DPS) staff gathered the
perspectives of different departments in the school district to discuss the potential
benefits of acquiring and sharing data with providers. The school district’s legal team
engaged in an extensive planning process that resulted in a broad data sharing agreement
between the afterschool network and the school district. Legal counsel determined in
their interpretation of FERPA that afterschool programs were operating as “agents” of the
school district and providing a service on behalf of the district. This interpretation made it
easier for the Denver Afterschool Alliance to facilitate the establishment of data sharing
agreements between afterschool providers and DPS. To mitigate risk and manage access,
each partner established their own data sharing agreement with the school district and
permissions vary based on the nature of the partnership with the school district.
Three cities worked through an external research partner to obtain the data they needed
from the school district. Having a trusted third party, such as an external research partner,
simplified their process as it leveraged agreements between the research partner and
the district. The Family League of Baltimore established an MOU with the school district
to receive specific aggregated indicators to be used internally to improve program
operations. However, the Family League also obtained more detailed school data through
their partner agency, the Baltimore Education Research Consortium (BERC). According
to one informant, the school district prefers this arrangement, not only because BERC
has experience with data permissions, but also because they have developed a wealth of
expertise on how to use and interpret the data. BERC serves both as the “data source” for
the Family League and as technical assistance provider on the use of the data.
In Grand Rapids, the ELO Network sought help from CRI, its research partner. CRI already
had data sharing agreements developed with the Grand Rapids Public Schools (GRPS)
through a prior local initiative. The Doug and Maria DeVos Foundation, a local funder,
also assisted and provided CRI with information about the permissions and data sharing
structure they had negotiated with GRPS. Those agreements authorized data transfers
between GRPS and local afterschool providers for program management as well as
research and evaluation. Similarly, Wilder Research, the local research organization for
26.	For additional information on student privacy bills and state legislative activity, visit http://dataqualitycampaign.org/.

Connecting the Dots: Data Use in Afterschool Systems | 34

Saint Paul, houses the data system for Sprockets. As is the case in Baltimore and Grand
Rapids, the school district and other organizations in Saint Paul viewed Wilder Research
to be, in the words of one informant, a “neutral and trusted” party that can responsibly
collect and analyze confidential student data. Having a trusted third party can simplify
the process of establishing an MOU. The tradeoff is that the third party typically cannot
release individual student information, which is of interest to some providers who would
like it to guide their programming. While Wilder does have the ability to provide Sprockets
with student-level data, Sprockets has not pursued this data as they would not be able to
obtain it as often as would be needed for program planning.
It has not been easy for most of the Next Generation afterschool systems to initiate,
develop, and sustain data sharing partnerships with school districts. At the outset of
the efforts to develop data systems, district stakeholders in most of the cities appeared
to consider the afterschool systems as partners in their mission to increase academic
achievement. Addressing the actual challenges to sharing data often relied on the
strength of individual relationships—superintendents, principals, and researchers, among
others—and the alignment of the system with the priorities of city government. School
district staff in at least a few cities seemed to view collaborating with the afterschool
system and sharing data on their students as tangential rather than integral to their
goals. Thus, afterschool systems’ requests for student data were perceived as extra work
that was not directly related to the mission of schools. As one informant shared, “The
district already has a daytime job.” The interest of school districts and their willingness to
make time and resource investments in afterschool systems appeared to vary over time,
primarily due to changes in school leadership and school priorities, but also in response to
financial constraints.

Preparing and Training the Network
In addition to defining the goals of the network, engaging stakeholders, assessing their
data needs, choosing and implementing an MIS, and establishing data sharing agreements,
the afterschool system also had to put processes in place to prepare and train members
of the network to collect and share data. Program directors, administrators, and staff
needed to understand the purpose of data collection, how to identify what they need
to use the data (e.g., learn data “language”), and how to use the technology at hand to
achieve expected outcomes when using data. To accomplish these ends, cities provided
afterschool provider personnel with access to trainings designed to: (1) familiarize them
with the overall objective for the use of data, the purpose it serves, and how it can be used
to inform their own programming; (2) develop the technical expertise needed to enter
data; and (3) enhance their skills in effectively identifying and communicating data needs.
Trainings were developed for a number of objectives to build afterschool network
partners’ capacity to use data. Figure 6 presents the most common objectives cities
identified for their data use trainings. Developing providers’ capacity to understand and
interpret the data elements contained in the afterschool system’s MIS was the most
frequently mentioned purpose and served to help providers make meaning of the data.
Understanding how to combine and use various data sources was the second most
frequently mentioned purpose of data use trainings. City stakeholders expanded trainings
in Denver, Nashville, and Saint Paul to include supports for organizations to meaningfully
use the data for quality improvement, modeling the cycle of data inquiry to providers and
how the use of data supports continuous improvement.

35 | Chapin Hall at the University of Chicago

Figure 6.

DATA USE
TRAINING OBJECTIVES

Common Objectives for Data Use Trainings Reported by Afterschool Systems
How to interpret MIS data elements
Model the data inquiry cycle
How to use multiple sources of data
0	 1	2	3	4	5	6	7	8	9
NUMBER OF AFTERSCHOOL SYSTEMS

Informants in eight of the cities reported the creation of trainings and materials to
support providers and other staff in using the MIS. As they learned about the needs of the
providers, system leaders expanded the types of professional development they offered.
For example, in addition to providing group trainings, the ELO Network in Grand Rapids
customized trainings to meet the needs of individual providers. Denver began by creating
a basic instructional module called CPS 101. An informant explained that “getting the
data from CPS is not easy and so we started training about how to do that and walked
them through those steps.” Denver system leaders then designed additional trainings,
CPS 201 and 301, to provide a continuum of training experiences that include data
interpretation and use of data for communication.
In addition to offering training on the data system and how to use the data to inform
practice, Nashville created a manual to support network providers’ use of its MIS. As a
NAZA stakeholder explained, “We produced the manual because we really want our folks
to use our data system and use it to guide them in their questions without requiring that
conversation with the zone director. We really want them to be able to ask questions
of themselves on their own.” The manual was created jointly by Metro Information
Technology Services, NAZA, and the school system, ensuring that each section was
written by someone with expertise in that area, and that the instructions and tasks
reflected the perspective and needs of the provider community.
Trainings served not only to make providers more knowledgeable and savvy with
regards to collecting and uploading data, but also to instill in providers the value of data
and encourage them to make it a priority in their work. City afterschool systems also
demonstrated the priority of collecting and using data by offering ongoing trainings on a
regular basis. Training needed to be prioritized given the high turnover the vast majority
of programs experienced. The provision of ongoing trainings helped to ensure data stay at
the forefront of providers’ activities.

Connecting the Dots: Data Use in Afterschool Systems | 36

“We know that there will be ongoing training needs, so we’ve established webinars.
That’s also important because there’s high staff turnover in out-of-school time, so we
know there will be a need for ongoing basic training. Then there will be a need for more
advanced training.” —Philadelphia stakeholder
Some cities also considered training as an opportunity for system building. In this
way, training was often viewed as a potential avenue for engagement and a means for
demonstrating a return on the effort that providers were investing in collecting and
inputting the data. Systems-level leaders in Philadelphia discussed efforts to get providers
who may view training and data collection as increasing their workload to see beyond the
work at hand and, in the words of a stakeholder, see that “it’s worthwhile, and it’s going
to be valuable not only to us as a city but also to you as an agency.” Another stakeholder
in Philadelphia asserted, “The more our providers are now seeing that the vision for the
network is to be data driven, the more I think we’ll see providers now make a connection
between the data that they’re collecting and then using that data.”

Establishing and Maintaining Data Quality
Ensuring the quality of the data collection required cities to develop a set of ongoing
monitoring processes for both the people and technology. Some systems, like the
commercial system used by Baltimore and Philadelphia, incorporate data checks into
the technology of the data system. Others use dedicated staff to monitor data quality.
Still others use both strategies. In addition, some cities supported these dedicated
staff positions with data committees, data work groups, or, in the case of Baltimore, an
attendance committee. Figure 7 depicts these strategies, which included (1) developing
data-centered, formal professional positions in a coordinating entity where one person’s
primary responsibility was to oversee the upload of data; (2) establishing formal
committees or workgroups to oversee the process; or (3) relying on an external research
partner or contractor. Some cities used a combination of these strategies to address
different aspects of the process. To ensure the quality of data, cities monitored both
the people who upload the data and the technology and software used to upload and
store the data.

Figure 7.

DATA
QUALITY STRATEGIES

Strategies Using People for Overseeing Data Quality*
Develop staff position in coordinating entity
Establish committee or workgroup
Rely on external partner or contractor
0	1	2	3	4	5	6
NUMBER OF AFTERSCHOOL SYSTEMS

*Some systems use more than one strategy. Fort Worth is not represented in the chart.

37 | Chapin Hall at the University of Chicago

Three cities designated at least one person at their research partner organization to
be responsible for checking and ensuring the accuracy of the data. Wilder research
staff in Saint Paul conduct daily checks, by hand, of the small number of youth
afterschool participants whose names cannot be matched with school district lists.
They also do monthly checks where both programming scripts and human judgment
identify duplication errors. NAZA in Nashville worked with the school district using a
similar approach.
Recognizing that its existing MIS, SAMIS, will be in operation for at least another year,
the JCC in Jacksonville planned to have its data manager run newly developed custom
queries of its existing data to remove incorrect and outdated data on a regular basis.
Denver designated a staff position to specifically include review of the quality of data in
the Community Partnership System (CPS) “to make sure that when school principals are
looking at CPS, that it is accurate.” A Philadelphia informant stressed the importance of
the role of the knowledge manager within the Department of Human Services, who has
been largely responsible for the development of the data system and training for staff
on data quality issues, and who continues to monitor the quality of data coming from
each partner using the data system. According to the informant, “Down the road, if we’re
building a budget for conditions for success, we need somebody like him that is devoted
to quality and making sure our quality objectives are happening.” Denver also relied on its
external database manager, Civic Canopy, to monitor providers’ upload of data to ensure
it gets done in a timely and accurate fashion. A Denver informant said:

“The onus [is on] the providers to make sure that their information is up and current
but a big piece of Civic Canopy’s role is making sure that they are [doing it on a regular
basis]…You still need somebody to be on point and communicating with providers about
making sure that they’re keeping their information up to date and [understand] why
that’s valuable. Even though it’s up to the providers to keep it current, you still have to
have a team working to help make sure that it is current. The system has to be able to
send reminders out because the data is only as good as the inputting of it.”
As another strategy to ensure the timely and accurate upload of provider data, several
city afterschool systems established a committee to oversee the data upload process
and ensure its quality. For example, in Baltimore, where data quality had been identified
as a challenge, the afterschool system developed an “attendance committee” to oversee
quality control checkpoints to ensure, according to an informant, that “data are entered
properly and well.” As a result, the system reported that the revised data collection and
upload process resulted in much improved data quality.
Ensuring data quality involves not only processes for supporting and monitoring
the people collecting and entering data, but also the technology used in the upload
and storage process. Many cities, such as Jacksonville and Nashville, developed staff
positions to monitor the quality to ensure the integrity of the data within the system.
In Jacksonville, systems-level staff were tasked with following up with providers to
confirm that the attendance data they entered in the system matched the data the
system reports out.

Connecting the Dots: Data Use in Afterschool Systems | 38

Getting Data Flowing
DEVELOPING DATA DASHBOARDS AND REPORTS
Collecting data and getting data entered into the MIS has been a major focus of the
Next Generation city afterschool systems, supported by considerable investment
in the people, processes, and technology aspects of a data system. In order for that
information to be used, it needs to flow out of the system and into the hands of decision
makers. System leaders were well aware of the importance of communication to their
stakeholders, regardless of the status of their work around collecting data and developing
their database. They described the need to craft messages specific to targeted audiences,
including their providers. A majority of the cities were presenting data in various ways for
providers and other stakeholders in their afterschool networks by the end of 2014.
Across the cities, system leaders shared examples of how they were developing the
capacity to design and produce useful reporting tools. For example, Jacksonville’s MIS
provided users with dozens of reports. As described by system leaders in Jacksonville, the
initial reporting interface was cluttered and confusing for providers, and the reports built
into the MIS did not align strongly with the needs of the afterschool network. To address
this challenge, system stakeholders had begun to talk about customizing a set of four to
five standardized reports that would summarize the data that afterschool programs use
on a regular basis. In Baltimore, the challenge was somewhat different. System leaders
had assumed that providers would be able to work with the raw data that flowed out of
the MIS, perhaps by using sort functions in spreadsheets, for example. Learning that this
was not the case jump-started conversations about ways to make data more useable for
their program managers and frontline staff.
At the end of 2014, six cities—Baltimore, Denver, Grand Rapids, Jacksonville, Nashville,
and Saint Paul—had already begun to systematically generate dashboards and reports
using their MIS to make information about system and provider performance accessible.
In general, these dashboards present one or more data elements, with the goal of allowing
users to understand changes over time or relationships among data elements. These
ways of summarizing data were largely descriptive, serving as a snapshot of agreed-upon
indicators. Here we highlight how these six systems with established MIS approached the
work of building out data dashboards, what they chose to include in their dashboards, and
the concerns and priorities that informed the design of these tools.

POPUL ATING DASHBOARDS AND REPORTS WITH DATA
Afterschool systems with already well-established MIS by the end of 2014 took two
different approaches to populating the dashboards that they developed to communicate
data to providers and other system stakeholders. The first approach relied on capacity
within the MIS to generate reports. The second strategy relied on internal staff or
research partners to extract data from the MIS and synthesize the contents into
actionable information.
To illustrate, Denver and Nashville automated the generation of provider dashboards that
summarized key data elements housed within their databases. This automation, which
required that the systems agree on a small number of standardized dashboards to build
and integrate into their MIS, yielded real-time (or close to real-time) information about
programs and their participants. Providers in these systems who wanted additional data
relevant to their specific goals and that informed how they were engaging their youth
had to make separate data requests. Grand Rapids and Saint Paul saw considerable value
in customized provider dashboards that specifically addressed the needs of individual
providers and used their staff capacity or research partners to populate these program39 | Chapin Hall at the University of Chicago

specific tools. In these cities, individuals such as a knowledge manager with analytic
capabilities and authority to access the data created the dashboards for the sites, and
research partner staff were responsible for creating customized reports.
Determining the type of reporting capabilities cities prioritized for the system informed their
selection of their MIS. For systems that wanted real-time reporting or to regularly produce a
predefined set of reports or dashboards, an MIS with the capacity to generate these reports
or support standard queries was seen as a strong asset. For systems with more diverse needs
or systems whose stakeholders planned on conducting an evaluation alongside regular data
updates, relying more heavily on human capital than technology informed their MIS design.

DETERMINING THE CONTENT OF DASHBOARDS AND REPORTS
As described earlier, systems and providers often have heterogeneous goals and priorities
for the programming they deliver to children and youth. When thinking about how to share
and present data, stakeholders needed to identify data elements that both reflected their
priorities and were broadly salient to their provider community. Multiple cities noted that
giving providers information that they could report to funders and program stakeholders was
a key tool for ensuring and maintaining provider buy-in. Baltimore, Denver, and Nashville
developed standardized dashboards for their programs, focusing on a small number of
potential indicators as their highest priorities.27 Grand Rapids and Saint Paul, as described
above, invested in customized reporting that included the indicators that individual providers
requested. A major theme of these reports, as described by a system staff member in Saint
Paul, was aggregated information based on data drawn from the school district system, such
as academic performance and attendance.
Concerns about making data actionable influenced what data the afterschool systems chose
to include in their dashboards. According to a Baltimore system leader, the main question
driving their dashboard design was how to represent impact through trends in program
attendance, for example, comparing current year data with data from the prior year. In
Nashville, the decision to focus on average daily attendance as a primary indicator of youth
engagement and to include this information in program dashboards had already enabled
comparisons from one year to the next across programs at the same school site and across
programs in a particular zone of the afterschool system. With multiple years of data on this
indicator now collected, system stakeholders anticipated being able to use trend lines to
inform the interpretation of findings.
Along with requests from providers, the priorities of afterschool system partners and
suggestions from external technical assistance organizations helped to inform what should
be included in their dashboards. In Nashville, dashboard content was heavily influenced
by the measures the school district prioritized for tracking: students’ school behavior,
attendance, and academic performance. Structuring the dashboards around these three
indicators meant that providers could easily access real-time information about their
performance in ways that aligned the primary foci promoted by the system. Because
the dashboards also summarized the number of students that exceeded the thresholds
established for behavior, attendance, and academic performance, having the data also
sparked conversations about how to support students around relevant priorities for both
afterschool system providers and school partners. According to one system manager, having
these types of conversations using program dashboards was leading to more effective
conversations about students with school staff, despite year-to-year turnover in staffing on
both the afterschool provider and school sides.
27.	 It should be noted that Jacksonville produced annual reports that included program-level information for its key academic and behavioral
indicators. To the best of our knowledge, however, these were produced primarily for the city council and not used with providers, although JCC
monitors kept track of attendance data through the year and used them when completing their monitoring forms on individual providers.

Connecting the Dots: Data Use in Afterschool Systems | 40

MAKING DATA ACTIONABLE THROUGH DASHBOARD DESIGN
As part of getting data into the hands of providers and other system stakeholders, cities
also had begun to think about how to make their reports and dashboards useable—a task
that required making decisions about how to present and summarize key data elements.
Systems that identified youth attendance and participation in afterschool programming
as a key priority had to determine how attendance data should be summarized: as
numbers, averages, or trends over time. In Baltimore, program-level average daily
attendance was a key accountability metric and was included in the scorecard used to
communicate with providers about performance. In Nashville, program dashboards also
totaled the number of students whose attendance dropped below a certain threshold
and providers could drill down to explore individual student attendance, a capability
that was possible because of the real-time, individual-level nature of data housed in the
database. In Grand Rapids, CRI consulted individually with programs to help develop
attendance benchmarks, creating a customized dosage indicator to reflect the program’s
structure and expectations. Cities developed a number of ways to convey more complex
information other than simply presenting numbers or averages for data elements included
in the systems. Doing this facilitated the use of data to improve outcomes for participants.
In Nashville, the system’s goal with its dashboards was to “make data most
understandable” for the people using them. There, the need for quick, easy-to-understand
snapshots of program performance informed decisions about what charts and graphs
to include and how to lay data out on the page, resulting in dashboards such as the
illustration in Figure 8.

Figure 8.

Sample Data Dashboard from Nashville (Anonymous)

41 | Chapin Hall at the University of Chicago

Each provider can access an individualized report through the MIS that depicts current
status or trends in key indicators for the students they serve. Viewed from the system
perspective, the choice to display attendance over time and quality results was intended
to promote thinking and conversation about the system’s collective focus on quality.
One system manager described the benefit of structuring the dashboard in this way as
allowing system stakeholders and providers to “ask some pretty strong questions related
to why attendance may be up or down, based on what the program quality results are.”
Across multiple sites, dashboards and reports were emerging as tools for convening
partners and starting meaning-making discussions that would allow afterschool systems
to transition from building the capacity to use data to actually using it. At the end of
2014, the question of how much reports could and should be standardized, as opposed
to being customized to meet the individual information needs of providers and other
stakeholder groups, was just surfacing. This topic will be examined further in the second
half of the study.

Summary
As cities identified the goals and purposes for data use, they concurrently worked through
ways to measure those goals and to collect information. Next Generation cities had
to determine the type of MIS that best aligned to their resources and the types of data
being collected in the system. Those that decided to build their own systems were able
to customize the functionality of the system; however, they had to balance financial
costs, whether they had staffing resources to manage the MIS, and whether consensus
on data components could be reached. Other Next Generation cities opted to leverage
existing resources of other partners, primarily school districts, to develop their MIS. The
use of existing technology facilitated the access to data and individuals with expertise to
manage the MIS. Cities that elected to use existing resources often aligned their goals to
those of the partner organization. Other cities combined systems by accessing multiple
different databases, usually with the assistance of a research partner. Regardless of the
type of system, cities had to establish data sharing agreements that articulated the
parameters of the use of data, types of data to be shared, who had permissions to access
data, and how the partners would use the data. These agreements often took a significant
amount of time, especially if they involved linking with school districts to gain access to
information protected by FERPA.
As cities developed their MIS, they began to establish strategies to build the capacity of
partners to enter and use data. Specifically, cities developed trainings to help providers
understand how to enter data based on common definitions and access the data stored
in the MIS. As providers became fluent in data entry, systems focused on the quality
of the information to ensure that partners and stakeholders were making decisions
based on accurate data. Throughout the process of establishing the components for the
use of data, systems often had periods of growth and times of frustration. This forced
systems to establish and leverage the people, processes, and technology that are the key
elements of the systems. Establishing the MIS was the foundation for using data to create
meaning. In the next chapter, we explore how cities moved from collecting, storing, and
communicating about data to using the data to inform the goals, activities, and decisions
of the afterschool system.

Connecting the Dots: Data Use in Afterschool Systems | 42

4. USING DATA AND BUILDING CAPACITY FOR DATA USE
“The biggest challenge is making sure providers really understand what we’re trying to accomplish and then
how to use the data [for planning]. In our last conversation with the provider agencies, we found that providers
read the report [about their baseline data], but they didn’t use those findings as part of their goal-setting
process. They had ideas of how they were going to improve their program, which were great ideas, but we
wanted to make sure they understood the link between going through this process and how it should become
part of their overall action planning.”—Nashville stakeholder

KEY POINTS

•	 As actionable data became more accessible
from the MIS, city systems were beginning to
use the information to focus on achieving the
goals of the afterschool system.
•	 The cities with a functional MIS were able to
use data to make decisions regarding resource
allocation to prioritize areas in the city for
expansion, improve program quality, identify
at-risk youth, and facilitate service planning.
•	 Once providers became more familiar with
entering data into the system with quality,
training needs expanded to include a focus on
using data to facilitate conversations about
program practices within and across providers.
•	 Ongoing coaching and training processes
emerged as a critical need to ensure continued
use of the system to account for the
significant turnover in the field.

43 | Chapin Hall at the University of Chicago

In this chapter, we discuss how the Next Generation cities
were analyzing and using data at the end of 2014 for
learning and improvement. Developing and implementing
data use practices depended on both existing capacity and
strategies for building capacity. All of the city afterschool
systems indicated that they use or plan to use data for
compliance and monitoring purposes. However, in this
chapter we focus on data use that is for the purpose of
planning, decision making, and system improvements.
Our interviews with key stakeholders in each city indicate
that at the end of 2014, five city afterschool systems—
Baltimore, Denver, Grand Rapids, Nashville, and Saint
Paul—were showing evidence of systematic data use for
these purposes. We first provide examples of how these
cities were developing the people and processes aspects
of their data systems, while taking advantage of the
technology aspects they had established. These processes
are outlined in Table 7. We then describe key strategies of
all of the cities to support and build the capacity of both
providers and system stakeholders to use data and the
factors that influenced those efforts. We conclude with
a discussion of facilitators and barriers to developing the
capacity for data use.

Table 7.

Processes and People in the Development of Data Use for Compliance, Learning, and Improvement
DATA USE PROCESSES

KEY ACTIONS

PEOPLE INVOLVED
USING DATA

Using data for
compliance and
accountability28

Provide data to comply with funders requests

System leadership and staff; technical staff;
research partners

Generate program-specific dashboards
and reports

System leadership and staff; technical staff;
research partners

Using data for learning
and improvement

Analyze gaps between current programming
and community needs

Afterschool providers; system leadership; funders

Support providers to improve program quality
and enhance youth engagement
Develop and maintain relationships with partners

SUPPORTING SYSTEMATIC DATA USE
Building provider
capacity for data use

Provide professional development (trainings,
coaching, and/or data dives)

System leadership and staff; network work groups;
technical staff; research partners

Motivating providers to
use data

Communicate a low-stakes orientation to
data use

System leadership; funders

Provide program-level data reports in
usable formats

System leadership and staff; technical staff;
research partners

Establish providers’ accountability for
performance on selected data elements,
e.g., attendance or quality

System leadership; afterschool providers

Collect relevant data

System leadership and staff; network work
groups; technical staff; research partners;
afterschool providers

28.	All of the city afterschool systems indicated that they use or plan to use data for compliance and monitoring purposes. However, in this chapter we focus on data use that is for the purpose of
planning, decision making, and system improvements.

Connecting the Dots: Data Use in Afterschool Systems | 44

Data Use for Learning and Improvement
All of the city afterschool systems with established MIS provided evidence of using data
for compliance and accountability purposes. Like most afterschool systems, the Next
Generation cities were using data or evidence about programming—for example, youth
participation or attendance—to respond to funder requests. Although this type of data use
predates the establishment of their data systems, some stakeholders reported that they
were beginning to actively use program-specific dashboards and reports, made possible by
new databases, to address longstanding compliance and accountability requirements of
their funding streams. Afterschool stakeholders also reported that their decisions about data
were influencing funders’ interest in and use of data. Although funders have long asked for
participation or attendance information, stakeholders in the afterschool systems noted that
funders were beginning to understand the limits of these data and were asking for data on
quality and youth outcomes.
Below, we focus on four uses of data for learning and improvement supported by the
successful establishment of data collection technology and processes. At least one of the
five city afterschool systems identified above had begun to regularly and systematically
implement these uses by the end of 2014. These uses were: (1) identifying gaps between
current programming and community needs, (2) identifying and responding to the needs
of individual children and youth, (3) supporting providers to improve program quality
and enhance youth engagement, and (4) using data to broker and sustain relationships
with partners. These uses align, in many respects, with the uses of data that the National
League of Cities 201229 report identified as high value or what Gerstein30 would describe as
connecting data to strategy or data use for learning and improvement.
29.	The National League of Cities report (2012) included the following in their definition of high-value data uses: (1) assessing youth outcomes and system
impact, (2) promoting accountability, demonstrating return on investment, and allocating scarce resources, (3) empowering afterschool providers and
reducing paperwork, and (4) facilitating peer benchmarking and professional development.
30.	Gerstein 2015.

Figure 9.

Types of Systematic Data Use for Learning and Improvement in Five Afterschool Systems, 2014*

TYPES OF DATA USE

Analyzing program-level quality and youth
engagement data
Sharing reports about system trends
and performance
Mapping the distribution of current
afterschool programming
Reviewing disaggregated student data
0	1	 2	3	4	5	6	7
NUMBER OF AFTERSCHOOL SYSTEMS

*The five cities represented in the figure are Baltimore, Denver, Grand Rapids, Nashville, and Saint Paul.

45 | Chapin Hall at the University of Chicago

DATA USE TO IDENTIF Y GAPS BE T WEEN CURRENT PROGR AMMING
AND COMMUNIT Y NEEDS
Cities are increasingly using geographic information systems (GIS) and other mapping
technology to identify areas where underserved youth live. Stakeholders in four cities
(Denver, Grand Rapids, Nashville, and Saint Paul) have used the results of community gap
and needs analyses to inform schools and community-based organizations about the needs
of the youth they serve and then, if necessary, realign or expand services to address gap
areas. In Saint Paul, stakeholders used school district data to compare the current locations
of providers belonging to the Sprockets network to a map showing the percentage of
young people living in poverty across the city. One stakeholder described this work as a
first step in an effort to better support the development of afterschool programs in areas
where young people would benefit most. Similarly, Denver conducted gap analyses to
identify community needs in different parts of their city to improve how they recruited
and supported potential providers with training and technical assistance. A stakeholder in
Denver described the process of working with providers to identify gaps:

“I was able to say, ‘What parts of the city do you want to be in?’ We were able to look at
the schools within the city and say, ‘There are a lot of arts programming in these schools
already. I can see why you’re not getting traction. Have you thought about this part of
the city instead?’ ‘Well, yes, and here’s why it doesn’t really work for us.’ ‘Okay, great.
Let’s dig a little bit deeper.’”
These analyses also revealed age or grade-span gaps, leading to conversations about how
to attract providers who work with underserved youth at particular ages in low-income
areas of the city.
In Grand Rapids, the afterschool system’s work began with analyses of community needs and
gaps in services. This city continues to use the tools it developed to learn more about how
children and youth participate in afterschool programs, particularly youth participation in
multiple programs, the accessibility of current afterschool programming, and where resources
should be allocated. Nashville has used annual citywide census data to determine where
the system focuses its work. The afterschool system uses information about the unique
characteristics and needs of district schools when deciding which providers it works with in
particular areas of the city and to determine where to expand NAZA programming. Although
other cities had not yet used community needs and gap analyses to inform system decision
making at the end of 2014, system leaders viewed this type of data use as foundational to
making good decisions about where programs should be located.

DATA USE TO IDENTIF Y AND RESPOND TO
INDIVIDUAL STUDENT NEEDS
At the end of 2014, the city afterschool systems were beginning to use data to help
providers respond to individual student needs, though these uses are limited to cities
that have established MIS and data sharing agreements allowing providers access to
data on individual participants. NAZA system leaders in Nashville generate attendance,
academic, and behavior information about individual youth to help inform providers,
enabling them to make decisions about what opportunities to provide to specific
participants in their programming. As described in Chapter 3, afterschool system provider
staff can access reports through their MIS dashboards about the numbers of students
with poor attendance, with behavioral issues, or not meeting academic expectations.
They also can meet with the school’s data designee and identify specific students who
are struggling and develop strategies for supporting students based on up-to-date
information. In an effort to better meet the needs of students in Grand Rapids, provider

Connecting the Dots: Data Use in Afterschool Systems | 46

interest in knowing more about which students were having trouble led to
conversations with the school district about the possibility of sharing early
warning indicators, such as academic or behavior problems, that are associated with
dropping out of school.
Beyond using data to track the participation of youth already enrolled in afterschool
programs, Baltimore system stakeholders had begun to use these data to implement a
more targeted youth recruitment process. They create recruitment lists based on analyses
of whether youth participate in afterschool programming in high-crime areas and share
them with school principals at the beginning of the school year. These lists highlight
students who might benefit from participation based on demographic characteristics such
as eligibility for free and reduced-price lunch, identification for special education services
through the district, and school attendance patterns. The city’s coordinating entity,
Family League of Baltimore, encourages providers to enroll youth from the lists in their
afterschool programming.
Other cities with established MIS with the technical capacity to identify and track
individual student needs have chosen not to pursue the use of data to intervene with
individual students due to concerns about the accuracy and timeliness of existing
individual level data or the ability of providers to respond appropriately to student needs.
Thus, they have taken a different approach to sharing these types of information with
their providers. In Saint Paul, customized reports provide aggregate data on the youth a
provider serves, including rates of school-day attendance (e.g., number of youth who had
attendance of at least 90 percent), measures of academic performance (e.g., number of
youth performing at proficiency level on statewide academic tests or making grade-level
appropriate gains), and demographic characteristics (e.g., number of youth eligible for free
or reduced-price lunch). In Grand Rapids, student-level data are aggregated to pinpoint
patterns, enable conversations about what the data show, and engage the community in
addressing the identified needs.

DATA USE TO SUPPORT PROVIDERS IN ENGAGING YOUTH AND
IMPROVING PROGR AM QUALIT Y
Program attendance—the amount of participation in programming, or dosage—is a key data
element for all of the cities, although only some were actively using it as a tool for learning
and improvement at the end of 2014. For the afterschool systems, program attendance
provides information on compliance factors (e.g., number of students enrolled, hours of
service provision) and quality factors (e.g., turnover rates of students/staff during the course
of the program, attendance of enrolled students). Systems reported using attendance data
to have conversations with program staff about trends and as an early warning indicator
that the program was experiencing challenges either entering data into the MIS or serving
children and youth. Though afterschool providers often individually report on attendance
data to their funders, the systems struggled with how to standardize attendance data
across programs of varying intensities (e.g., programming offered daily or on selected days
of the week) and length (e.g., programming structured in alignment with the school year
or in terms of a certain number of weeks). Denver’s database, for example, was not initially
designed in a way that could answer questions about dosage, although they were working to
make modifications by the end of 2014. In Baltimore and Nashville, providers are expected
to enter program attendance data in the system’s database. Data trends are analyzed and
programs are supported in meeting individual student needs and addressing programmatic
barriers to attendance. As a core component of their role and responsibilities, NAZA zone
directors and coordinators review program enrollment, utilization data, and other MIS
reports and dashboards at regular intervals. In the words of an informant in Nashville:
47 | Chapin Hall at the University of Chicago

“[Average daily attendance] is one of our primary indicators of youth engagement, so if the
kids are really engaged and enjoying what they’re doing, they’re going to be coming regularly…
Now we’re starting to get multiple years of data so you can start to look at trend lines. Is it
getting better? Getting worse? What does that mean?”
Beyond looking at these data to identify patterns over time, system coordinators are expected
to assess providers’ success with their strategies and talk with them about performance issues
reflected in the data. As another stakeholder explained, being able to review attendance and other
data in a timely manner allows the NAZA system to determine the root causes of these patterns
with providers and identify strategies to make programming more engaging for the youth they are
intended to serve. “If kids aren’t participating, we want to know right away,” the stakeholder said.
The Baltimore and Nashville afterschool systems are using attendance data to assess demand
for services and determine the number of slots that the systems support through contracts
with their providers. Explaining the rationale for using data to inform funding choices, a system
stakeholder in Nashville explained, “It’s really clear that if they cannot engage kids, they cannot
keep them coming.” Nashville reduces the number of slots allocated to a provider if, after
implementing new strategies, it does not improve youth participation. In general, all afterschool
system providers are expected to have an average daily attendance of 70 percent or greater
in order to receive an additional stipend. The Baltimore afterschool system monitors youths’
average daily attendance rates to identify patterns of utilization and, if a provider’s average daily
attendance is low, they reduce the provider’s final quarter payment.
Beyond the five cities that are the focus of this section, other cities involved in the initiative—
Jacksonville, Louisville, and Philadelphia—reported similar uses of data at the end of 2014.
However, despite aspirations to do so, they were not yet using the data for learning and
improvement systematically in the ways that stakeholders in Baltimore, Denver, and Nashville
described data use. For example, in Louisville, an informant reported telling a provider that
the program was being paid a lot for an average daily attendance of three students in an
effort to encourage more consistent and accurate data entry. In Philadelphia, because of
differences between funding agency expectations, some providers have to report attendance
in an agency’s legacy MIS but are not required to report it in the new MIS developed for the
PhillyBOOST network, complicating the city’s ability to systematically collect and analyze
attendance across all programs. In Philadelphia, a subset of providers—those funded by DHS—
have traditionally had performance-based contracts tied to youth participation, which serves
as one model for how the broader system in that city could interpret and use data. System
stakeholders in Jacksonville also planned to use average data attendance to inform system
decision making, though challenges related to getting providers to consistently enter attendance
information were complicating this use.

Box 3.

What Are YPQA and SAYO?
The Youth Program Quality Assessment (YPQA), developed by the
David P. Weikart Center for Youth Program Quality, is a validated
instrument designed to measure the quality of youth programs
and identify staff training needs. It has been used in community
organizations, schools, camps, and other places where youth have
fun, work, and learn with adults. It measures seven domains: safe
environment, supportive environment, interaction, engagement,
youth-centered policies and practices, high expectations
for youth and staff, and access. For more information, see
www.cypq.org/assessment.

The Survey of Academic and Youth Outcomes (SAYO), developed
by the National Institute on Out-Of-School Time (NIOST),
measures eight outcome areas that research suggests are
linked to long-term positive development and academic and life
success in afterschool program youth: behavior in the program/
classroom, initiative, engagement in learning, relations with adults,
relations with peers, problem solving, communication skills,
and homework. For more information, see http://www.niost.org/
Training-Descriptions/survey-of-afterschool-youth-outcomesyouth-survey-sayo-y.
Connecting the Dots: Data Use in Afterschool Systems | 48

Program quality, as defined by the David P. Weikart Center for Youth Program Quality and by
the National Institute on Out-of-School Time (NIOST), goes beyond investing resources in
the safety and organizational elements to include relational factors and youth engagement.
Cities reported using other data sources to improve program quality and engage youth,
including assessments of program quality such as the YPQA and SAYO assessments
(see Box 3). Of the cities implementing the YPQA, all but one (Baltimore) reported their
quality assessment data to the Weikart Center, which provided analyses and reports to the
afterschool systems. Baltimore has developed the capacity to internally analyze the data
from the YPQA—although these data are not included in their MIS—and developed their own
reports to share with providers. Although cities actively used these data to support programs’
improvement planning and implementation through separate databases or spreadsheets,
collecting the data in a separate database meant that they had limited ability to use their
MIS to identify patterns or trends in program quality in concert with other data elements.
Nashville recently created the capability to store YPQA data in its MIS, and Denver and Saint
Paul were considering adding SAYO data to their MIS. This type of move could, as one Denver
stakeholder explained, emphasize the value of program quality and youth outcome data.
The MIS, as a data “hub,” would remind users about the importance of paying attention to
program quality in decision making.
By 2014, a few cities were also tracking information about participation in professional
development. In Nashville, these data were maintained in a separate MIS and could
be analyzed by individual or in aggregate. These analyses, along with provider requests
for particular trends in program quality and student-level indicators, informed system
planning for future trainings and workshops. As one system stakeholder explained, “That’s
another piece of data that we’re able to use to see actually how our programs are
performing and to be able to assist them in the areas where they’re low by offering them
professional development opportunities.” In other cities like Baltimore, data are analyzed
to assess whether programs are meeting participation benchmarks for professional
development. These data are maintained in a separate database with no plans to add these
elements to the MIS.

DATA USE TO COMMUNIC ATE AND BUILD
REL ATIONSHIPS WITH PARTNERS
Stakeholders in some cities reported leveraging data to communicate and strengthen
relationships with key partners, including city agencies and funders. In the words of a Grand
Rapids system stakeholder, “[Data] drives how you advocate.” In using data to promote and
advocate for the afterschool system, our informants underscored the need to consider the
preferences of providers and partners. Although funders are often interested in using data
to inform their investment strategies and decisions, afterschool providers and other system
stakeholders are concerned about potential high-stakes uses of data.
System stakeholders, funders, community members, and other external audiences are
interested in knowing more about the quality and outcomes of afterschool programming
delivered to children and youth. Impact reports can help city afterschool systems
communicate their progress to key partners and make the case for public investment in
afterschool systems. The specific focus of the reports to these audiences in each city has
varied. In Nashville, for example, system stakeholders share reports with a range of city
decision makers, including NAZA’s leadership council, a school district advisory group, and a
local education funder group. Prepared by the Weikart Center, these reports describe system
performance and illustrate patterns of program performance on various indicators of quality.
Denver worked with a communications organization to craft its first comprehensive impact
evaluation report focusing on afterschool programming across the city. System leaders
shared this report with city and school leadership as well as their funders.
49 | Chapin Hall at the University of Chicago

In addition to increasing interest in afterschool programming, stakeholders we interviewed
asserted that the data in these reports also help inform the funding process. As a Philadelphia
informant explained, even though many stakeholders already believed in the importance of
afterschool programming, “the dollars could be wiped away if we couldn’t justify their use.” More
data, and especially quality data, helped to make a stronger case for the funding needed to bring
the system to scale. An informant in Louisville highlighted the critical contribution data can make
in framing conversations about ongoing public investment in afterschool programming. As she put
it, being able to put data in front of city council members made it easier to reframe conversations
that generally unfolded along the lines of, “Don’t ask me to give you money or pay higher taxes
if you can’t show me that this is really making a difference and improving the outcomes in my
community.” She added that, in the current political environment, “I don’t think you can do it
without showing [data].”

Strategies for Building Capacity for Data Use
Many of our informants stressed that data must be made meaningful before they can be used. In
this section we discuss activities that some of the cities have implemented to build the capacity
of providers and system leaders to use data. Provider strategies include professional development
and efforts designed to motivate providers to use data. System strategies include starting small,
managing and improving the afterschool system, and working with external researchers. We
continue to draw mainly from examples of the five cities that were showing evidence of active use
of data for learning and improvement at the end of 2014, but also provide examples from other
Next Generation cities that have implemented strategies to foster data use.

Figure 10.

Afterschool Systems’ Strategies for Building Providers’ Capacity to Use Data
9

NUMBER OF AFTERSCHOOL SYSTEMS

8
7
6
5
4
3
2
1
0

Supporting
ongoing data
use through
individualized
support/coaching

Providing
trainings on using
data for program
improvement

Providing regular
opportunities
for groups
of providers
to interpret data

PROFESSIONAL DEVELOPMENT

Providing
program-level
data reports in
useable formats

Promoting a lowstakes orientation
around data use
within the system

Prioritizing
collection of
data relevant to
providers

Holding providers
accountable for
performance
on selected data

BUILDING PROVIDERS' BUY-IN AND MOTIVATION

STRATEGIES

Connecting the Dots: Data Use in Afterschool Systems | 50

BUILDING THE C APACIT Y OF PROVIDERS
To build the capacity of providers to use data, the city afterschool systems invested in two
main strategies: professional development and training, and activities to foster providers’
buy-in and motivation for data use. Figure 10 provides an overview of these strategies and
the number of cities that were investing in them at the end of 2014.

Professional Development Strategies
Analyzing and using data for learning and improvement involved a steep learning curve for
most providers across the nine cities. A Philadelphia informant explained:
“We have learned through this data system that our program staff did not necessarily learn
program development in the sense of it being equal parts design implementation and
evaluation…What we’re trying to do is train our program staff to be clear that if you do not
track what you are doing then you are not programming. That is a very different shift in
culture and mindset.”
Consistent with the literature on data use for decision making,31 several informants for this
study suggested that providers would be more likely to appreciate the relevance and meaning
of data if they could review program data with peers through a facilitated process in a
comfortable, open setting. Thus, many city afterschool systems organized trainings or meetings
on how to interpret data. Six of the nine cities—Baltimore, Denver, Grand Rapids, Nashville,
Philadelphia, and Saint Paul—have implemented regular trainings, and other cities described
plans to implement similar activities in the future. Intended to boost providers’ capabilities to
use data for planning and improvement, these data trainings served as low-stakes32 settings
where providers and system leaders have conversations about data that have been collected
and how that data can be made useful for providers, school personnel, and system stakeholders.
In the five cities highlighted at the beginning of this chapter as afterschool systems with
more systematic use of data for improvement, data trainings specifically focus on developing
providers’ skills and capabilities to address questions using multiple sources of data such as
YPQA, demographic, community, and school data. For example, Family League of Baltimore
staff convene workshops in which providers review data, set priorities, and develop a datainformed program improvement plan.
In three cities, data use trainings were specifically designed to model a data inquiry cycle for
providers. As mentioned earlier, the Denver Afterschool Alliance developed a series of training
sessions to meet the needs of providers with different levels of experience with data. An initial
session, called CPS 101, walks providers through the logistics of accessing program data in
the MIS and uploading their rosters; CPS 201 helps providers learn how to interpret the data
and what they can or cannot say based on the data; and CPS 301 focuses on using data as a
communications tool. These trainings also include “take-it-back” activities to help participants
engage other staff at their program sites around data analysis and use.
The decision to focus the CPS 201 and 301 trainings on the data use cycle was motivated by
feedback from providers participating in a pilot version. In order to analyze and use data for
improvement, providers first needed to consider how they would want to use the data and
reports now accessible through the MIS. Denver also provides opportunities for a small cohort
of providers in its so-called Demonstration Project to participate in additional “Planning with
Data” sessions to enhance their understanding of the data inquiry cycle, triangulation,33 and
meaningful use of data.
31.	Coburn and Turner 2011.
32.	As noted in Chapter 3, Smith 2013, among others, notes that data use in an accountability framework can be either for “high” or “low” stakes, meaning
system decisions about how to make use of information about provider performance (for example, as the basis for funding decisions or not) can create
different incentives and motivate varying data activities.
33.	Triangulation is a technique that facilitates validation of data through cross verification from two or more sources.

51 | Chapin Hall at the University of Chicago

In Nashville, providers and school staff from each NAZA zone come together to review
and discuss site-specific data reports. These “data dives” are convened twice a year after
report cards are issued. In the January 2014 session, individuals were grouped by their
program location and each received a packet that included a map of the community area
for their school or program with flags representing addresses of students eligible for free
and reduced-price lunch, a data dashboard printout with the standard school information
for NAZA participants such as attendance and discipline, and a bar chart with YPQA data.
According to an informant, these data dives have served as professional development
for providers with limited experience using data and helped to facilitate communication,
strategic planning, and “a culture of collaborative inquiry” among providers.
Saint Paul system leaders, with their research partner, Wilder, designed an annual
daylong event to help providers learn to analyze and use multiple data sources to inform
their program development efforts. In these “Making Meaning with Multiple Datasets”
sessions, also called M3, trainers work with providers to draw connections among many
diverse data reports—on attendance, program quality, program experiences, and youth
outcomes—to reflect on each individually and identify patterns, and then examine the
patterns in the context of other data sources. Trainers emphasize that each data source
provides both information and context useful to guide decision making.
Finding additional time for more regular practice in interpreting data was challenging,
but several cities implemented other processes to complement their formal trainings.
These included providing individualized coaching for provider staff and creating cohorts
to participate in professional learning communities. To supplement the data analysis
that happens at Nashville’s twice-yearly data dives, for example, zone directors conduct
monthly site visits with each of their providers. They use the strategies identified in the
provider’s improvement plan to structure discussion. Similarly, in Philadelphia, program
specialists worked with providers to look at individual youth’s participation, guiding them
through a process of interpreting and making meaning of attendance and other data.
Individualized coaching was implemented in Denver as well and was under development
in Baltimore. Grand Rapids was reported to be considering a similar strategy. As a
stakeholder explained, “Providers don’t have a lot of time to really digest and pore
through [the data], and they don’t have an objective person to ask the kind of questions
they need to, say, ‘Okay, how would you interpret this?’”

Motivational Strategies and Incentives
In the Next Generation afterschool systems, fostering a low-stakes orientation to data
use was considered an important strategy to increase providers’ buy-in and interest
in data use. System stakeholders emphasized that with the creation of the MIS, many
providers were seeing information about participating youth for the first time. As a result,
system stakeholders in several cities described the learning process as an iterative, timeintensive reflection process. A system stakeholder in Denver emphasized the importance
of building providers’ motivation and interest in data use as critical to the success of any
MIS or data use-focused initiatives, stating:
“It’s in the process of evolving and getting people comfortable with it. That’s why
I certainly haven’t been aggressively pursuing the finest use of [data]. There are
so many things we’re just working out. . . I’m more interested in maintaining
people’s commitment and trust, and participation and, overall, to see the right way
to be going forward.”

Connecting the Dots: Data Use in Afterschool Systems | 52

Similarly, an informant in Grand Rapids anticipated that provider buy-in would
increase as providers experienced their own successes from using data for decision
making, saying:

“What I hope is having some data will help them do some program planning and
program improvement and then they’ll like that they’re seeing improvements in data
and we’re going to help them understand their information. They’ll be the ones who will
want to show that to their funders and say, “Look, I can prove I’m showing outcomes.”
Likewise, system stakeholders in Saint Paul intentionally decided not to evaluate or give
feedback on providers’ improvement plans as a way to make their application of data lowstakes. In a similar vein, one Jacksonville stakeholder described an interest in fostering a
quality mentoring—as opposed to quality monitoring—approach to data use.
Beyond promoting a low-stakes improvement orientation to data use, cities described
implementing three other strategies that were helping to motivate providers to use data.
One strategy was to show providers how MIS dashboards and reports could help them to
address their specific questions and concerns. Though providers have traditionally given
some data on outputs to their funders, they rarely have had the opportunity to use these
data themselves because the systems they enter data into do not always provide easy
formats for them to extract the data. While providers have the raw data that they report
to funders, they may not have the internal capacity to compile and analyze the data to
inform programming decisions. Cities reported that providing reports was helping increase
their providers’ engagement in data use. In the words of a stakeholder in Philadelphia,
“Now that we’re beginning to generate reports, [decision makers] see the data across
systems from the enterprise level. That has really helped in generating interest and kept
the conversations going.”
A second strategy, as discussed earlier in this chapter, was to hold providers accountable
for performance on selected data elements. While cities predicted negative consequences
for using youth academic outcomes in a high-stakes manner, they also acknowledged
that in order for programs to engage and benefit youth, the youth need to be physically
present. Drawing on the idea that consistent participation (or high utilization rates) was
an indicator of a successful program, average daily attendance rates were one source
of data that system leaders of Baltimore, Jacksonville, and Nashville programs used
to inform decision making and planning. In Baltimore and Nashville, system leaders
do not renew contracts with providers who demonstrate consistent patterns of failing
to engage youth.
A third strategy was to collect more data on indicators important to providers. The
promise of expanded access to data, particularly data that providers saw as relevant to
their missions, helped to foster their motivation to use data for learning and improvement.
In Louisville, the system’s promise of access to information about students’ socialemotional learning and timely feedback that could inform program planning created, as
described by an informant, “phenomenal” interest in the data system among providers. In
Baltimore, Grand Rapids, and Saint Paul, efforts to engage providers in determining what
data elements would be collected and housed in the MIS were characterized as a key
mechanism for building buy-in for data use activities. An informant in Saint Paul observed
that engaging providers in system planning helped keep the MIS and data relevant. This
relevance was particularly important, according to a Baltimore stakeholder, because
provider frontline staff “are in youth programs because they want to be there working
with the kids. They have the passion for that. When we tell them, ‘But administratively,
you have to take this much time out of your day to do this bookkeeping work,’ that really
turns them off.” System stakeholders in these cities discussed investing in data about

53 | Chapin Hall at the University of Chicago

students’ social-emotional learning development as having considerable potential to
engage frontline workers and promote greater focus on youth outcomes.

BUILDING THE C APACIT Y OF THE SYSTEM
Turning data into reliable, useful knowledge for the system as a whole—just like for
improving individual programs—fundamentally involves people and processes. What
system leaders want to know about the entire afterschool system is similar to what
providers want to know about their own programs:
•	

What do we know about youth participation?

•	

What explains patterns of robust or limited youth participation?

•	

On what quality indicators is the system fairly strong and where are key areas
for improvement?

Data uploaded by providers into the MIS can be used to identify overall trends or
uncover patterns in progress on key indicators over time. The data also can help system
stakeholders assess the degree to which the system is meeting its goals. Analyses of the
data can be used to customize professional development for providers and guide efforts
to align programming with city goals for the afterschool experiences of children and
youth. Informants’ comments suggested that shifting data use habits was a particularly
salient system-level activity in cities that formally partnered with multiple agencies as
part of the afterschool system.
Although they may ask similar questions, the purposes for which system leaders seek to
use data (e.g., to manage and improve the system) are not necessarily the same as the
purposes for engaging providers in data use to improve program quality and outcomes
for youth. According to our informants, system leaders want to take up questions about
where and how to expand services, assess where system structures such as the MIS and
professional development opportunities are contributing to program quality and youth
outcomes, and decide how to allocate resources to achieve system priorities.
System leaders placed considerable weight on tracking system progress on key measures
like the number of students and communities served, the system’s success in reaching
the youth that can most benefit from afterschool programs, and evidence about the
relationship between youth participation and academic outcomes. Measurement and
data-driven decision making are a cornerstone of advocacy and legislative efforts to
improve programs and policies for children and youth, including current federal and state
education accountability policies. In Grand Rapids, informants described feeling increased
pressure to show outcomes for afterschool programs in order to maintain current levels
of funding for the system. One system stakeholder’s perception was that, as a result of
the way the system was responding to these pressures, providers’ data use was becoming
more compliance-focused than improvement-focused.
Despite the considerable attention that afterschool system leaders gave to supporting
providers in using data for improving program quality and youth participation, they
struggled to develop parallel processes for using data for system building and
improvement. System stakeholders valued a variety of system-level purposes for data
use such as establishing a vision of quality, developing shared thinking about high-priority
outcomes and system-level improvement, and creating comparable quality standards.
The system’s ability to make these kinds of systemic changes depended on the capacity
of the MIS to provide useful data at the system level and system leaders’ time to analyze
and use the data. A system leader in Baltimore felt the challenge of finding time to focus

Connecting the Dots: Data Use in Afterschool Systems | 54

on system-building work given the continual demands of supporting implementation.
A Philadelphia stakeholder made a similar point, arguing that the afterschool system’s
ability to pay attention to system-level outcomes was restricted by challenges related to
getting the MIS established and convincing providers of the importance of consistently
entering their data.
System leaders and partners in some cities had considerable leverage in developing and
implementing data-driven approaches to enhancing their systems. In Nashville and
Denver, the leadership of the two main partnering agencies had significant experience
supporting these types of initiatives in both education and city government settings. In
other cities, these sources of expertise existed in pockets, such as system stakeholders
that had management rather than leadership roles within the afterschool system. For
example, the head of the Department of Parks and Recreation in Philadelphia was
described as “totally committed” to continuous improvement after a weeklong training
with NIOST. He found funding to pay for staff at all of his afterschool programs, not just
programs that were part of the afterschool system pilot, to participate in training to use
the SAYO and other data as part of a continuous improvement process. Another network
leader in that city was also reported to have spearheaded the implementation of a similar
process within her city agency.

Opportunities and Challenges for Building Capacity for Data Use
By the end of 2014, providers in several cities were beginning to see the potential
benefits of using data more effectively in their work. However, as one Baltimore system
stakeholder noted, “For the most part, our folks are checking the boxes and doing what
they need to do, but they could be going a little deeper in terms of talking about the
strategies or the program design or how things are being implemented.” This experience
was not unique to Baltimore. Informants in all nine cities saw room for growth in both
provider and system data use that could ultimately improve outcomes for children
and youth. Here, we discuss the common barriers we have observed in the efforts of
city afterschool systems to build capacity to use data as well as the facilitators of data
use. These efforts, summarized in Table 8, highlight the interrelationship of the people,
processes, and technology aspects of a data system.
DEVELOPING SKILLS OF FRONTLINE STAFF
Developing new data use capabilities among frontline staff, program managers, and
system stakeholders emerged as a challenge across the Next Generation cities. System
stakeholders talked about needing to regularly engage providers around data-use basics,
as opposed to being able to rely on knowledge from prior data trainings and familiarity
with the MIS. In Nashville, system stakeholders described the work of developing capacity
for systematic data analysis and interpretation as one of cultural change. As one Nashville
informant put it, the challenge was not that providers did not want to engage in data use.

“To the degree that providers have wanted to get that data and have been willing to
come to the table, it has been just great. It is just so outside the skill set of some of
them that it’s just a steep learning curve. And then with the staff turnover of frontline
staff on top of that, it’s hard for some of my partners to create and sustain a culture
inside their agencies.”
High turnover among frontline program staff was identified consistently as a challenge
to building provider capacity, as suggested by this informant. In response to these related
challenges, cities were finding that they have to offer their trainings on data analysis
and use frequently—much more often than at the startup of the school year. As one
Philadelphia stakeholder explained, “There’s high staff turnover, so we know there will

55 | Chapin Hall at the University of Chicago

be a need for ongoing basic training. Then there will be a need for more advanced training.”
Although the afterschool systems had designated staff to provide training and support—or
were able to make use of staff in partner agencies—the need to invest system resources in
onboarding new frontline staff limited resources for other tasks, including supporting more
sophisticated uses of data.

Table 8.

Opportunities and Challenges for Building Capacity for Data Use
OPPORTUNITIES

CHALLENGES

Developing data use skills of
frontline staff

•	 System leaders prioritize developing
human capital and expertise around
data use

•	 System may have limited prior experience
supporting systematic data use among
providers
•	 System leaders may not recognize the role
of training and sustained opportunities for
meaning-making in promoting data use
•	 High levels of frontline staff turnover
challenge efforts to develop human
capital around data use

Communicating and building trust
among stakeholders

•	 Strong, pre-existing relationships
between system and provider staff

•	 Limited communication about goals and
priorities between system and provider
staff

Accessing data visualizations
and reports

•	 MIS interface and infrastructure allows
system leaders to analyze system- and
neighborhood-level patterns
•	 Perceptions of the database as reliable
and functional, with useful data
visualization capabilities
•	 Dedicated staff to generate reports
and help providers and system
stakeholders to interpret them

•	 Time lag between data entry and
availability of reports
•	 Poor data quality

Collecting relevant data

•	 Strong partnerships among system
stakeholders or with contracted
research partners
•	 Effective data sharing agreements

•	 Validated, easy-to-use indicators and
tools are not always readily available
•	 Disagreement over which data are
relevant

Promoting a low-stakes,
improvement orientation to data use

•	 System leaders can advocate for a
low-stakes, continuous improvement
orientation to data use

•	 Systems have limited ability to influence
the broader stakes that providers may
experience around data use

Connecting the Dots: Data Use in Afterschool Systems | 56

COMMUNIC ATING AND BUILDING TRUST AMONG STAKEHOLDERS
Trust and strong relationships between system and provider staff, as well as among
system stakeholders, emerged as a key facilitator strengthening provider and system
capacity to use data. In particular, trusting relationships appeared to be especially
important in opening up communication across different levels of the system. A Baltimore
system stakeholder attributed growing engagement and interest around data use to the
fact that the system’s activities were informed by both “grassroots and grasstops.” In
Louisville, a system leader argued that these longstanding, trusting relationships boosted
the willingness of providers to engage in learning how to use data in the absence of
financial incentives to do so. Similarly, in Nashville, community-based providers were
initially disconcerted with the idea that school staff would be able to see data on the
children they serve, but strong relationships fostered by the NAZA zone directors helped
to reduce their concerns over time.
Regular communications and meetings also contributed to stronger relationships
between system partners and between system and provider staff. Almost all of the cities
instituted work groups to guide and support their system-building work, and these groups
helped create important opportunities for communicating about partner priorities and
values. In Philadelphia, one system stakeholder characterized the afterschool system data
group, which was made up of system partners, data experts, and providers, as “the most
consistent, productive work group” they had been involved with. Another stakeholder
from Philadelphia contrasted the robust communication within the system’s data, quality,
and coordinating entity work groups as failing to flow between the groups. She explained,
“We’re at the point right now where we’re connecting the dots between the conversations.”

ACCESSING TIMELY AND USEFUL DATA
VISUALIZ ATIONS AND REPORTS
The degree to which the MIS facilitated access to useful data visualizations and timely
reports was an important influence on data use. Having an established MIS, although
not essential to the work of creating data visualizations and reports, made it much
easier for systems to quickly relay information to their providers. Even so, generating the
data visualizations and reports from the MIS was not straightforward, with many cities
manually compiling data for programs or relying on providers to generate the information.
Baltimore provides one example of how systems can make data visualizations more useful
to providers. In that city, the system engaged providers in improving the usability of a
quarterly scorecard that summarized program performance. The revised template showed
providers where they were on enrollment, attendance, volunteers, partnerships, and
resources—all indicators that they saw as relevant to their work. A system stakeholder
described the improved tool as making it easier for providers to focus on interpreting and
using these data.
In some cities, data quality concerns prevented timely and accurate reports. In Denver,
for example, duplicate student records made it difficult to assess the impact of program
dosage (how often a youth participates) on various outcomes. Informants in Jacksonville,
Philadelphia, and Denver reported instances in which MIS-generated reports did not
match the internal records of providers. Thus reports that were supposed to facilitate data
use instead created questions about the data among providers.
Cities also recognized the need to create consistent methods of accessing and reporting
out progress as a system, with particular attention to generating and sharing knowledge
about the system’s successes and areas for growth. Implementing the communications
strategies they envisioned was not straightforward. Some of these challenges had
57 | Chapin Hall at the University of Chicago

to do with the design of the MIS. In Baltimore, choices made as the MIS was being
developed—often based on cost and efficiency concerns—meant that data entered by
providers could not easily be organized in ways that allowed them to analyze data about
particular communities or the system as a whole. In Saint Paul, because different providers
collected data on different indicators, custom reports needed to be built in order to answer
questions about provider outcomes. Thus, the MIS did not contain all the fields needed
to track provider outcomes in a centralized manner. Organizational capacity posed other
challenges. For example, in Jacksonville, system stakeholders aspired to generate monthly or
quarterly reports about their programs, but found that they did not yet have the necessary
technological and staffing capacity.
Generating data reports for providers was a common function undertaken by system staff. In
some cases, staff created customized reports based on provider requests for particular types
of information. In other cities, system staff were more proactive. For example, system staff in
Baltimore, Louisville, Nashville, and Saint Paul described pulling reports and e-mailing them
to providers on an ad hoc basis in order to encourage data use. In Nashville, zone managers
shared hard copies of these reports at monthly meetings and helped providers review and
interpret their data reports to improve performance.

COLLECTING “RELEVANT” DATA
Facilitating access to data that providers saw as relevant to their missions was, as noted
earlier in this chapter, important for generating excitement and sustaining interest in data
use. In cities like Grand Rapids and Nashville, the establishment of data sharing agreements
with school districts meant that providers had access to school day attendance and academic
outcomes data about their participants for the first time. System stakeholders felt these
data provided important contextual information that helped guide frontline staff in making
decisions about the types of programs to offer their participants.
Providers and stakeholders in several cities believed that social-emotional learning and other
nonacademic factors were at least as important indicators of youth development and thus
were more interested in this type of data. Informants in cities using the SAYO tool, including
Denver and Saint Paul, tended to feel that it provided data on important intermediate, youthfocused outcomes such as behavior and relationships and could be used in conjunction with
program quality data such as YPQA indicators to make programming decisions. Yet, over
time as they became more knowledgeable about the domain of nonacademic outcomes,
they found that the SAYO did not directly address some key social-emotional outcomes of
interest to providers, such as initiative, teamwork, empathy, self-control, and persistence in
learning. Thus, by the end of 2014, system stakeholders in at least a few cities were exploring
other ways to capture other youth development information that would help inform
program planning.
Some of the cities also were engaging additional agencies such as health and juvenile justice
in conversations about access to new and different data elements that would help maintain
the relevance of the MIS for stakeholders. Across the systems, however, leaders described
challenges to identifying new resources to support this work and to reaching agreement
about what elements to add to the MIS and how the system should be using data. Indeed, in
at least two cities, informants told us about disagreements over how data already collected
and housed in the MIS should be used. In Louisville, for example, system stakeholders had
different priorities about which youth should be targeted for recruitment, and how it should
be done. Similarly, stakeholders in Baltimore were still working to identify the right data
to understand how individual students’ connections to services and interventions might
relate to outcomes.

Connecting the Dots: Data Use in Afterschool Systems | 58

PROMOTING A LOW-STAKES, IMPROVEMENT
APPROACH TO DATA USE
Even as city stakeholders characterized a low-stakes, improvement orientation as a
linchpin in their efforts to build capacity for data use in 2014, they also reported ongoing
challenges in promoting this approach to funders. Generally, with scarce resources,
funders and other stakeholders continue to emphasize holding providers accountable
for program attendance and youth academic outcomes that they considered most
relevant to school and career success in the twenty-first century. Although the majority
of the cities collected data on program quality, that data was being used mainly to help
programs reflect on their progress and their self-identified areas of improvement. At the
same time, there was evidence of growing awareness of the importance of quality among
agency partners and funders, and increasing requests in proposals for quality indicators.
These requests came with different stakes. For example, major funders in Louisville asked
for evidence of minimum quality standards in funding proposals. A major funder in Grand
Rapids, on the other hand, was supporting programs that were participating in a quality
improvement process rather than asking for evidence that the programs had achieved a
particular quality rating.

MANAGING EXPECTATIONS AND BEING STR ATEGIC
Given the complexity of developing, managing, and effectively using a system-wide
MIS to effect change at multiple levels, system leaders adopted a range of strategies
to manage their work. Although it is premature at this stage in our study to talk about
general steps and strategies that all or most city afterschool systems have taken along
the way towards developing their data systems, we are beginning to discern some general
concepts and themes in their efforts: (1) starting small or simply with a pilot to test
processes and structures before scaling up and (2) establishing ongoing human capital
and expertise for data use through building relationships with partners and contracting for
outside expertise.

Starting Small
As indicated in the table at the end of Chapter 2, not all of the programs participating
in a city afterschool system were necessarily entering data into an MIS. Several cities,
including Denver, Grand Rapids, Philadelphia, and Saint Paul, were developing and refining
their data systems and training for providers by piloting new activities with only a subset
of providers. As informants in multiple cities pointed out, improvement rates depend
on baseline quality. Providers also need different levels of support and professional
development to align with and advance the system’s vision. In Denver, a cohort of diverse
providers was invited to pilot additional capacity-building supports, such as additional
data use training offered in a learning community format and coaching to support their
implementation activities. Philadelphia developed a pilot of 23 providers to collect SAYO
data and pilot the new ETO system before rolling it out to a larger number of providers.
Other cities, including Baltimore, Jacksonville, and Nashville, could also be characterized
as “starting small” because they focused their system-building activities on a subset of
afterschool programs. In Baltimore, the coordinating entity developed a Community
Schools engagement initiative in partnership with the school district, pulling back from its
prior, more diffuse strategy for funding afterschool programming. At the schools involved
in this work, school-based coordinators partner with afterschool providers to identify
strategies to address performance issues. Similarly, Nashville’s NAZA focused specifically
on developing a coordinated system of afterschool programming for middle school

59 | Chapin Hall at the University of Chicago

students. NAZA zone directors work with middle school principals to match afterschool
providers that offer particular types of programming to community needs. As mentioned
previously, Denver created the Demonstration project with a subset of the afterschool
providers to create a structured learning community and to pilot and refine training and
professional development materials prior to expanding to the broader afterschool system.
Philadelphia system stakeholders described customizing reports for one umbrella provider
organization, the After School Activities Partnership, that was advocating for greater data
use among its members. This effort was viewed as a potential model that the city could
use to strengthen its capabilities to support other providers’ data use. For these four cities,
demonstration or pilot projects with a subset of providers helped them both meet their
programs where they are and, simultaneously, move their afterschool system forward.

Working with External Researchers
As discussed in Chapter 3, system leaders in some of the cities turned to external
research partners to help build their capacity to collect, organize, and analyze data. These
research experts helped providers and system leaders interpret and use data for system
improvement. Grand Rapid’s research partner, CRI, is a vocal participant in discussions of
how data should be analyzed and used. A Grand Rapids informant reported:
“[CRI staff ] really help us to make the case for specific needs and issues that the kids
are experiencing, help us to think more thoughtfully about how to then reverse those
negative trends, and then help us to evaluate the effectiveness of what we’re doing
when we try to be responsive to what the data has shown is a problem…Having them as
a partner in every iteration of the process is huge.”
CRI’s research director also advised the afterschool system on what types of data would
be needed to establish a causal connection between participation in afterschool programs
and various outcomes, and specifically cautioned against making claims about the impact
of afterschool program participation on reductions in juvenile justice trends.
In other Next Generation cities, these research partners have played a somewhat less
extensive, albeit very important, role to foster data use. As noted above, Saint Paul’s
research partner, Wilder, assisted in designing and facilitating the M3 trainings in
data interpretation. Baltimore’s research partner, the Baltimore Educational Research
Consortium (BERC), both provides individual-level data for the MIS and analyzes the
performance of youth participating in afterschool programming. A specific focus of this
work is to better understand youth outcomes over time. As characterized by system
leaders, this research organization is a key thought partner who has supported the
coordinating entity through multiple phases in the development of their data system.
To bring some of this expertise into the system, a new analyst position has been split
between Family League of Baltimore and BERC. According to an informant, this analyst
was learning from colleagues at BERC “how to do the work in a deeper and more
sophisticated method” to serve the needs of the afterschool system.
In Nashville, as part of another initiative, NAZA leaders contracted with an external
research organization to assess whether higher-quality programs more effectively
reduce students’ academic challenges than lower-quality programs. Again, they were
particularly interested in the impact programs had on a set of early warning indicators
that were associated with future dropout. The resulting report examined correlations
between program participation and youth school day attendance, behavior, and academic
outcomes. Afterschool system leaders then used the findings on the impact of higher-

Connecting the Dots: Data Use in Afterschool Systems | 60

versus lower-quality programs to plan future evaluation strategies. In other cities, the
system turned to partners with strong research and evaluation capacity to carry out these
more sophisticated analyses of system data. Denver and Louisville, for example, created
shared positions with the school district to support this type of work.
System leaders in other cities focused on developing capacity across the system
recognizing the need to develop their own skills as well as the providers. Some of these
cities hired experts to train and support system leaders, similar to the trainings and
support offered to providers. Sprockets, in Saint Paul, contracted with two developmental
evaluators to conduct trainings, review academic literature on the developmental benefits
of afterschool programs, and organize periodic surveys of providers to get feedback on
their system-building activities. The experts also have worked with Sprockets staff to
develop a cycle of inquiry including: acquiring data, interpreting it in context, and creating
knowledge that leads to action. As one informant explained, the process included asking,
“What information would be helpful to reflect on? Now, what are we going to do based on
what the information is telling us? This is the most basic protocol; we call it the ‘what, so
what, now what’ kind of inquiry.”

Summary
By the end of 2014, a number of the Next Generation cities had begun to use data for
improving access to and quality of afterschool programs, leveraging their significant
investments in data collection and database development. Among the five cities that
were most actively analyzing and using data, a majority were engaged in identifying
gaps between current afterschool programming and community needs, supporting
providers to improve program quality and enhance youth engagement, and strengthening
relationships with partners. Differing philosophies about how to most appropriately use
data to improve access to programs and youth outcomes are reflected in the people
chosen to be involved in the data system, the processes that help the system function,
and the technology established to “fuel” the system with data. For example, some cities
decided to focus on individual student outcomes while other focused on the aggregate
impact of programs on groups of children. Decisions made about people, processes, and
technology during earlier phases of system development subsequently lead to both
opportunities and challenges to data use.
In 2014, systems were also developing diverse approaches to build the capacity
of providers and system leaders for data use. Five cities had already designed and
implemented professional development strategies that focused on how to make
meaning of and use data. Stakeholders in these systems were continuing to learn
what providers needed to know in order to leverage MIS reports and other data about
program performance. Beyond basic trainings focused on the interpretation of key data
elements, professional development strategies for provider staff included the creation
of advanced data use trainings such as using multiple data sources or an inquiry cycle
model, individualized coaching, and learning communities or cohorts of providers
meeting regularly around data. System stakeholders also recognized the importance of
providers’ buy-in and motivation to use data in moving their systems forward and used
strategies ranging from prioritizing the collection of particular data elements to producing
straightforward and usable data reports to create engagement. Although informants in
all cities emphasized the centrality of a low-stakes, improvement orientation to engaging
system stakeholders around data use, they also talked about the role financial incentives
played in their efforts to strengthen providers’ capacity for data use.

61 | Chapin Hall at the University of Chicago

At the system level, cities confronted tensions related to developing structures and
strategies that could help their systems move forward while simultaneously working to
create robust implementation supports for their programs. Improvement-focused data
use is fairly new work for the afterschool sector, and represents a shift in the culture
of the field. Leaders in many cities were actively learning how to support system-level
capacity building and continuous improvement efforts even as they were designing
and implementing strategies and structures to build provider capacity. Not all of the
afterschool systems had staff members with the expertise needed to support rigorous
data analysis to inform their agenda and work. In cities that contracted with external
researchers, these relationships were described as contributing positively to system
capacity to work with providers and partners, as well as furthering internal and external
conversations about system goals, priorities, and areas for growth.
Finally, there were a number of facilitators and barriers that influenced efforts to
build provider and system capacity to use data. Specifically, these factors affected
the development of knowledge and skills to use data for learning and improvement,
relationships between system and provider staff, access to relevant data, availability of
reports and data visualizations, and the stakes systems attached to data use. The specifics
of what each city experienced as facilitators and barriers along these dimensions varied.
However, the broad themes in the experiences of the nine cities suggest that attending to
these dimensions is a necessary part of their system-building work to help stakeholders
develop their capacity for data use.

Connecting the Dots: Data Use in Afterschool Systems | 62

5. SUMMARY AND EMERGING THEMES
“Does the data system meet our needs? Yes. Will it evolve? Yes. As the [afterschool network] evolves over
time and we begin to ask more in-depth questions of the data, and ask a series of those questions, that
leads us to look at other data components. So, as we move, the system has to be flexible enough to move
with us.”—Nashville stakeholder

KEY POINTS

•	 City afterschool systems have made
considerable progress in identifying the data
they need to demonstrate their value to
stakeholders, improve service quality, and
increase youth access to high-quality programs.
•	 Although technology is a critical component of
data use, it could not happen without human
expertise and social processes.
•	 The cities have faced a number of challenges
but also have made considerable progress
in developing the capacity to collect and
organize data.
•	 Cities are in various stages of using data to
advance the goals of their afterschool programs
and system. In about a third of those cities in
the study, the processes of data collection,
analysis, and use are becoming routine.
•	 Themes in early strategies include starting
small, building or purchasing research and data
expertise, and being prepared for change.

63 | Chapin Hall at the University of Chicago

Throughout this report, we sought to show that
developing the capacity to collect, organize, and use data
in city afterschool systems is complex. It can only happen
through long-term investments in people, processes, and
technology. In this interim report, which focuses on the
time period between 2012 and 2014, we described the
steps that the nine city afterschool systems in this study
have taken, and the decisions they have made, to develop
their capacity to use data.
The cities have invested significant resources and
wrestled with multiple components of system
building during this process, including developing
coordination and management structures and
processes, identifying stakeholders’ data needs,
documenting what data exist and determining what
new data are needed, developing agreements to access
and share data, evaluating the capacity of existing MIS,
learning about new technology, developing processes
to ensure data quality, hiring and training the right
people, and building relationships with partner
organizations. First, we summarize our key findings
with regard to each of these components and their
interrelationships as nine city afterschool systems
developed their capacity to use data to guide their
system-building activities. We then discuss
their implications for the cities going forward, and for
the next phase of this study.

Summary of Findings
In recent years, the overall volume of data being collected, stored, and analyzed in the
U.S. has exploded.34 However, the amount of information captured about afterschool
data systems still remains relatively small compared to other systems, such as education
and child welfare. As city afterschool systems expand the nature of data they want to
and can collect, they are faced with a new problem: how to use this additional data
meaningfully rather than adding data merely because it is possible. They also must
address their capacity to analyze and use the data effectively and efficiently as their
systems grow and evolve.

PEOPLE IN DATA SYSTEM DEVELOPMENT
People are a key factor in how the nine cities have embarked on the work of developing
afterschool data systems and building the capacity of system stakeholders to leverage
data in ways that inform their practice and plans for improvement. For example, the
individuals and groups involved in the data system determine what data can be shared,
with whom, and for what purposes. In an era in which data breaches are headline news
and federal and state governments closely regulate data use (e.g. FERPA), the specifics of
data sharing agreements and security issues are critical to building effective data systems.
However, key players in the city afterschool systems advocated for interpretations of
privacy laws that would allow the systems to use protected data to improve services for
children and youth. In Nashville, agencies were able to secure access to a range of data
elements to inform decisions about where to locate new programs and how to customize
programs to address students’ developmental and academic learning needs. In Saint Paul,
assigning a “neutral” research partner responsibility for much of the system’s data-related
activities was viewed as a successful way of managing these challenges.
In this report, we discussed providers’ engagement with and trust in data systems and
how the cities are building their capacity to use data. In each of the nine cities, system
leaders have been generally successful in making the case for maintaining a low-stakes
orientation to using the data collected through their MIS as of the end of 2014. This lowstakes approach was regarded as fundamental to providers’ willingness to participate.
All parties recognized the importance of learning from the data through conversations,
skills training, and collaboration prior to tying the data to funding decisions or other
“higher stakes” value for the data. For example, providers and system leaders worried that
data suggesting that the afterschool system was not making afterschool opportunities
available to the most vulnerable youth or that programs were not positively affecting
student’s academic progress would jeopardize their future funding.
The importance of bringing together the right “team” to support key aspects of
implementation emerged as a theme across the nine cities. Having players willing and
able to use their political capital on behalf of the afterschool system was important to
securing both the financial and human resources needed to build and maintain a data
system. Having a mayor and city council members that recognized the value of investing
in an afterschool data system for their own agendas created strategic opportunities to
position afterschool at the center of city conversations about improving outcomes for
their children and youth. Key afterschool system stakeholders also helped current and
potential partner institutions and organizations to recognize how the data system could
strengthen their own organizations’ work.

34.	Goren 2012. Roderick 2012.

Connecting the Dots: Data Use in Afterschool Systems | 64

System partners also have brought relevant knowledge and technical capacity to afterschool
data systems. In some cities—Denver, Nashville, and Louisville—expertise initially came from
technology and data specialists within the school district. In other cities, notably Baltimore,
Grand Rapids, and Saint Paul, research partners played a significant role in supporting
system-building work. As described in Chapter 3, Grand Rapids’ research partner proved
adept at helping its ELO Network develop indicators and data integration strategies that
informed the development of the MIS that minimized the burden on providers without
compromising data quality.
The experiences of the Next Generation cities suggest that there are some roles and functions
that are crucial to developing data systems. The people who carry out these roles and functions
are administrators and technical/database experts who are either formally part of the
afterschool system staff or are employees of system partners or contracted research entities.
Where they are housed is less important than whether their specified job responsibilities
include database development and maintenance.
The development of a database infrastructure and interface is a complement to, rather than
a substitute for, engaging system stakeholders in building data systems. This engagement
work takes time, as the experiences of Fort Worth, Saint Paul, and Grand Rapids illustrate. In
2014, without a strong coordinating entity in place, the Philadelphia system was struggling
to convince providers from all of its partner agencies to use its new database. Philadelphia’s
challenges suggest the importance of focusing on people and processes, not just technology, in
developing a system to collect and use data to inform system-building activities.
There was a sense among afterschool system stakeholders that when coordinating entities
use funding as leverage, providers adopt data systems more rapidly. Our evidence is not yet
clear on this point, although the conversation about engaging providers or obtaining their
“buy-in” to the data system seemed different in systems where coordinating entities lacked
this leverage. Leaders of an unfunded local network have to find other incentives to foster
participation. For example, Sprockets in Saint Paul has been able to engage more providers in
its data system each year without tying participation to funding; providers seem motivated
to use the data system to answer questions about their programs and the system as a whole.
Network leaders seek ways for providers to direct their own program improvement plans and
inform decisions about allocation of resources, which in turn fosters their participation in the
system. Nonetheless, some stakeholders have expressed concern about whether providers
would be willing to financially support the system if grant money were not available to support
the MIS and associated resources.

PROCESSES IN DATA SYSTEM DEVELOPMENT
Processes are another key factor in how system building is unfolding in the Next Generation
cities. One theme reflected in the cities’ approaches was to start small by focusing on one
domain or aspect of the afterschool data system and then building additional elements. These
pilots or incremental efforts can help to ensure that the cities “got it right” before going to
scale. As Foster-Fishman and Watson argue, “small wins increase momentum for change,”35
and the leaders of Next Generation cities were well aware of the potential of data to help
them advocate on behalf of their systems. The decision to focus on a particular subset of the
data system may arise from practical, financial, or philosophical considerations as part of a
larger rollout strategy.36 Examples of starting small included concentrating system-building
work with a subgroup of providers (Denver, Saint Paul), a subgroup of students (Baltimore,
Nashville), a city region (Nashville), or a small set of data elements (Louisville, Philadelphia).
35.	Foster-Fishman and Watson 2012.
36.	This issue arose not only with respect to the development of data systems, but around larger issues many cities face in deciding about which providers are
part of a city afterschool system.

65 | Chapin Hall at the University of Chicago

Starting small isn’t without challenges. A subset of data or a subset of programs cannot
paint a complete picture of the entire scope of an afterschool system. System leaders had to
balance anticipated benefits of leveraging a proven strategy for their data-related activities
with potential costs such as perceptions of the system not moving forward quickly enough,
losing provider interest, or losing a key political window to advance the work.
Another theme that emerged from exploring the contributions of processes to system
building was the importance of formalizing the routines and practices. Establishing rules for
data collection and entry, such as explicit definitions for data fields, became increasingly
important as data began to flow into systems. Similarly, data systems required effective
human and/or technological strategies to manage and enhance the quality (reliability and
validity) of the data collected and, in turn, the reports that were generated. Developing a
standard definition of how attendance is measured is one example of where formalization
and shared understandings contributed to the ability to collect higher quality data. Without
consistency across providers, what was entered into the system reflected the “garbage in,
garbage out” challenge that faces data systems more broadly. Further, stakeholders need
high-quality data to collectively establish comparable quality standards across the diverse
programming delivered by network providers.
A key challenge to establishing routines, norms, and practices was turnover among
afterschool frontline staff. Informants in all of the cities with established MIS reported that
turnover affected data collection and entry. System stakeholders were learning that they
needed to provide ongoing introductory trainings in using the MIS and using data. Coaching
and manuals also helped to mitigate the effects of turnover and to further the development
of more experienced staff. For example, Denver had developed multilevel training sessions
by the end of 2014, and other cities had identified the need to invest in a more flexible
approach that would ensure consistency among new users of the system while promoting
the development of more sophisticated data uses among more stable users.

TECHNOLOGY IN DATA SYSTEM DEVELOPMENT
Technology is a third key factor affecting how data systems unfolded in the cities. The
decision to build, buy, or adapt databases interacted in important ways with the contexts
in which databases were being developed. Much of the prior research on the development
of MIS for afterschool systems has focused on the costs in dollars and benefits of building,
buying, or adapting databases. However, the decisions of the nine cities suggest that
relationships with key partner agencies and local context also influence the practical costs,
benefits, and implications of these decisions. For example, system leaders with the option
of adapting an existing database viewed it as one that would cost less in dollar terms than
building or buying a new database from scratch. Most cities that pursued this option were
able to do so because partner agencies were willing to share their databases. As a result,
cities that implemented a composite MIS were able to invest fewer resources in database
development. However, this strategy limited their ability to customize their databases
to align with the primary goals and purposes of their afterschool system. In contrast, the
three cities that implemented a self-contained MIS invested significant amounts of time
and funds in developing their databases, which delayed their ability to use data in the ways
that they anticipated, even though the MIS was tailored more directly to address the goals
of the afterschool systems. Thus, the work of building databases and making choices about
technology reflected and responded in important ways to available resources, local context,
and the dynamic nature of key relationships and partnerships in the afterschool setting.
As system stakeholders developed their capacity to use data, they requested new data
elements and that data be made available in various formats, both aggregated and
disaggregated. Providers, for example, did not necessarily see their school-related outcomes
Connecting the Dots: Data Use in Afterschool Systems | 66

as the only or most appropriate indicators for understanding their impact on youth.
Among the cities that were already beginning to systematically use data, there were at
least two distinct ways in which system stakeholders hoped to strengthen the usefulness
of their databases: (1) they requested additional data elements that were not part of
the original build-out of the database, and (2) they started to compare youth who
participated in programs to similar youth in terms of grade levels or demographics who
did not participate to understand the potential impact of participation. In essence, they
wanted to identify patterns to guide their programming decisions.
Technological advances also outpaced the MIS that some cities initially selected to be
the cornerstone of their data systems. Some cities, including Baltimore, Jacksonville, and
Philadelphia, discovered over time that they needed to invest in a new system in order to
collect and analyze the data they needed to improve their programs and systems. Other
cities invested considerable time and resources to develop their own MIS, which slowed
their ability to produce useful data.

EARLY STR ATEGIES IN DEVELOPING DATA SYSTEMS
At this point in our study, it is premature to talk about the effects of the choices the
cities have made in developing their data systems. However, a few themes have emerged
with respect to the cities’ strategies. Other city afterschool systems planning to embark
on a similar journey or accelerate their current data strategies might consider these
approaches. Here we briefly mention three of these strategies from the decisions and work
of these nine cities, with the expectation that others will surface as the study continues.
With time, we will also be able to more clearly discern the impact of various strategies in
various contexts.
•	 Starting small with a pilot to test processes and structures before

scaling up. Examples of starting small included focusing data system development
activities on a subgroup of providers or a particular age group, a region of the city, or a
small set of data elements. In Denver, for example, this meant not trying to be “all things
to all people” and shifting the focus of its system building and data system development
to a smaller set of programs.

•	 Building human capital and data expertise by establishing relationships

with knowledgeable partners and contracting for outside expertise. Cities
relied on various sources of technical expertise in collecting and using data. In addition
to using Wallace funds to hire a data manager or knowledge manager, or drawing on
expertise from existing partners, some afterschool systems contracted with an outside
research partner, often a nonprofit or university organization with expertise in research
and evaluation. In some cities, these partners assisted with designing the data system,
producing reports, and training program staff. It appears that these more wide-ranging
relationships helped afterschool systems to use data for learning and improvement more
effectively than limited relationships with research entities.

•	 Being prepared for change as technology advances, as frontline staff and

leadership turn over, and as city priorities and funding shift. Afterschool
systems’ goals may evolve or change, intervention strategies may change, and individuals,
partner organizations, and/or institutions might leave the system. Several of the study cities
frequently revised plans and goals for the afterschool system as a whole. In some cities,
this meant making changes in the goals and structures of their network. For example,
Denver, Fort Worth, and Philadelphia modified governance and coordination structures, and
Baltimore found new ways for different groups of providers to work together. These changes
had implications for the data system, and the aim and the challenge was to establish a
system that is “resilient”—that is, flexible and adaptable—in the face of such changes.

67 | Chapin Hall at the University of Chicago

Connecting the Dots and Looking Ahead
The use of data in the afterschool field is fairly new
and is still evolving. Our findings thus far suggest that
afterschool systems will increasingly collect data to
help make decisions about how to allocate scarce public
and private resources to improve program quality and
access. Afterschool systems are using data to assess
program quality, professional skills of staff, and access
for youth most in need of services. Data also hold the
promise of informing and engaging more stakeholders in
a system, including providers, funders, partners, parents,
and youth themselves. Moreover, the growth of data use
has important consequences for other aspects of city
afterschool systems. Data use is helping to create new and
more exacting expectations about what an afterschool
coordinating entity should do, how program expansion
should happen, what quality looks like, and the different
ways that quality can be measured.

•	 What is the relationship between the afterschool data

The nine Next Generation cities have faced significant
challenges in developing their data systems including:

•	 What are the incentives for providers to participate in the

•	 changing systems due to outdated MIS or MIS without the

capacity to incorporate new types of data,
•	 renegotiating data sharing agreements because of

turnover in a partner organization, and
•	 hiring new database managers or providing another

round of training to providers because of staff turnover.
We expect that high staff turnover and low staff skills,
long-standing staffing problems that affect the quality
of afterschool programs, will continue to challenge the
nine city afterschool systems’ efforts to use data. But the
work of the past two years also suggests that most of the
study cities are strengthening their foundations to weather
these challenges. Some cities, for example, have designed
ongoing training programs on data collection, analysis, and
use, and some have established their coordinating entities
as independent nonprofits to protect them from changes
in city leadership.
As we reflect on the experiences of nine city afterschool
systems, there are several questions we believe are
important to pursue in the next phase of this study. Some
of these are new questions; others were asked during the
first phase the study that we want to further pursue in
the next phase.
•	 As city afterschool data systems mature, how does

data use change?

system and the other data systems in the cities?
•	 To what extent do cities extend the concept of continuous

improvement beyond its use in the improvement of
programs and apply it to the functioning of the system as a
whole? What path do cities take in regards to a continuous
improvement approach to their system-building work?
•	 What are the experiences of afterschool providers with

the data system and how data are used? How do their
experiences influence their capacity and motivation to
engage in data use for learning and improvement?
•	 What are the most effective workforce development

strategies to engage and prepare frontline staff to
collect and use data to inform their practices and build
program quality?
data system in their cities? How do they move beyond using
data for compliance purposes?
•	 What is the value for afterschool systems in developing

the capacity for data use in relation to the costs of
developing that capacity?
•	 What people, processes, and technology factors

allow a data system to be flexible, sustainable, and
forward looking?
In the second half of the study, we will continue to explore
the challenges and opportunities for data-use efforts in the
heterogeneous afterschool field and in a policy and funding
environment in which afterschool is framed as an answer
to multiple societal problems such as education, juvenile
delinquency, and youth engagement. We will examine
how the people involved in afterschool data systems
negotiate and manage change within the growing network
of partner agencies, including school districts, libraries,
parks, and funding entities. We will also explore adjustments
that afterschool systems have made in their pursuit of
effective data use. Finally, while continuing to deepen our
knowledge about data use from afterschool system leaders,
we will also look more closely at providers’ experiences to
better understand how to effectively support them. We
look forward to continuing to learn from the nine Next
Generation cities on their journeys to develop, expand, and
solidify both their afterschool systems and the data systems
that support them, and to deepening our understanding of
how these efforts can improve the well-being of children
and families in their communities.

Connecting the Dots: Data Use in Afterschool Systems | 68

BIBLIOGRAPHY

Ackoff, Russell L. (1989). “From Data to Wisdom.” Journal of Applied Systems
Analysis 16: 3-9.
Afterschool Alliance. (2014). Taking a Deeper Dive into Afterschool:
Positive Outcomes and Promising Practices. Washington, DC. http://
afterschoolalliance.org/documents/Deeper_Dive_into_Afterschool.pdf.
Balfanz, Robert, Jennifer L. DePaoli, Joanna Hornig Fox, Erin S. Ingram, Mary
Maushard, John M. Bridgeland. (2014). Building a Grad Nation: Progress and
Challenge in Ending the High School Dropout Epidemic. Washington, DC: Civic
Enterprises. http://www.civicenterprises.net/home.
Bodilly, Susan J., Jennifer Sloan McCombs, Nate Orr, Ethan Scherer, Louay
Constant, and Daniel Gershwin. (2010). Hours of Opportunity, Volume 1:
Lessons From Five Cities on Building Systems to Improve After-School, Summer
School, and Other Out-of-School-Time Programs. Santa Monica, CA: RAND
Corporation. http://www.rand.org/pubs/monographs/MG1037.html. Also
available in print form.
Coburn, Cynthia E., and Erica O. Turner. (2011). “Research on Data Use:
A Framework and Analysis.” Measurement: Interdisciplinary Research and
Perspectives 9, no. 4: 173–206.
———. (2012). “The Practice of Data Use: An Introduction.” American Journal
of Education 118, no. 2: 99–111.
Deich, Sharon, Meghan Neary, Heather Padgette. Governance of City
Afterschool Systems: A Scan and Analysis. Washington, DC: Cross & Joftus.
Manuscript in preparation.
Durlak, Joseph. A., and Roger P. Weissberg. (2011). “Promoting Social and
Emotional Development is an Essential Part of Students’ Education.” Human
Development 54, no. 1: 1–3.
Durlak, Joseph A., Joseph L. Mahoney, Amy M. Bohnert, and Maria E. Parente.
(2010). “Developing and Improving After-School Programs to Enhance
Youth’s Personal Growth and Adjustment: A Special Issue of AJCP.” American
Journal of Community Psychology 45, no. 3-4: 285–293.
Eccles, Jacquelynne, and Jennifer A. Gootman, eds. (2002). Community
Programs to Promote Youth Development. National Research Council and
Institute of Medicine. Washington, DC: The National Academies Press.
Every Hour Counts. Every Hour Counts Measurement Framework: How to
Measure Success in Expanded Learning Systems. (2014). New York, NY: Every
Hours Counts. http://www.afterschoolsystems.org/files/4060_file_Every_
Hour_Counts_Measurement_Framework.pdf.
Farnham, Lija, Gihani Fernando, Mike Perigo, and Colleen Brosman, with Paul
Tough. (2015). “Rethinking How Students Succeed: A Wave of Noncognitive
Skill Initiatives Holds Promise for Making Teachers More Effective and
Students More Successful.” Standard Innovation Review. http://ssir.org/
up_for_debate/article/rethinking_how_students_succeed.

69 | Chapin Hall at the University of Chicago

Fiester, Leila. (2004). Afterschool Counts! A Guide to Issues and Strategies for
Monitoring Attendance in Afterschool and Other Youth Programs. New York,
NY: The After School Project, Robert Wood Johnson Foundation. http://www.
theafterschoolproject.org/uploads/After_School_Counts-v1.pdf.
Foster-Fishman, Pennie G., and Erin R. Watson. (2012). “The ABLe Change
Framework: A Conceptual and Methodological Tool for Promoting Systems
Change.” American Journal of Community Psychology 49, no. 3-4: 503-516.
Gerstein, Amy. (2015) “Shifting the Norms: From Data for Compliance to
Data for Strategy.” Gardner Center Perspectives, John W. Gardner Center for
Youth and Their Communities. https://gardnercenter.stanford.edu/news/
shifting-norms-data-compliance-data-strategy.
Goren, Paul. (2012). “Data, Data, and More Data—What’s an Educator to
Do?” American Journal of Education 118, no. 2: 233–237.
Halpern, Robert. (2002). “A Different Kind of Child Development Institution:
The History of After-School Programs for the Low-Income Children.” Teachers
College Record 104, no. 2: 178–211.
———. (2003a). Making Play Work: The Promise of After-School Programs for
Low-Income Children. New York, NY: Teachers College Press.
———. (2003b). The Challenge of System-Building in the After-School Field:
Lessons from Experience. Discussion Paper. Wellesley, MA: National Institute
on Out-of-School Time. http://www.erikson.edu/wp-content/uploads/
halpern_system.pdf.
———. (2006). “Critical Issues in Afterschool Programming.” Monographs of
the Herr Research Center for Children and Social Policy 1, no. 1. Chicago, IL:
Erikson Institute.
Halpern, Robert, Julie Spielberger, and Sylvan Robb. (2001). Evaluation
of the MOST (Making the Most of Out-of-School Time) Initiative. Final
Report. Summary of Findings. Chicago, IL: Chapin Hall Center for Children,
University of Chicago. https://www.chapinhall.org/sites/default/files/
old_reports/70.pdf.
Hayes, Cheryl, Christianne Lind, Jean Baldwin Grossman, Nichole
Stewart, Sharon Deich, Andrew Gersick, Jennifer McMaken, and Margo
Campbell. (2009). Investments in Building Citywide Out-of-School-Time
Systems: A Six-City Study. Philadelphia, PA: Public/Private Ventures. http://
www.wallacefoundation.org/knowledge-center/after-school/coordinatingafter-school-resources/documents/investments-in-building-citywide-out-ofschool-time-six-city-study.pdf.
Holleman, Martha A., M. Jane Sundius, and Eric J. Bruns. (2010). “Building
Opportunity: Developing City Systems to Expand and Improve After School
Programs.” American Journal of Community Psychology 45, no. 3: 405–416.
Johnson, Elaine, Fran Rothstein, and Jennifer Gajdosik. (2004). “The
Intermediary Role in Youth Worker Professional Development: Successes and
Challenges.” New Directions for Youth Development 2004, no. 104: 51–64.

Kingsley, Chris. (2014). “Focus on Data sharing: Building Formal Connections
to Local Systems to Boost Afterschool Impact.” American Youth Policy
Forum, Better Together Conference. http://www.aypf.org/resources/focuson-data sharing-building-formal-connections-to-local-systems-to-boostafterschool-impact/.
Larson, Reed W. (2000). “Toward a Psychology of Positive Youth
Development.” American Psychologist 55, no. 1: 170–183.
Larson, Reed, Robin Jarrett, David Hansen, Nickki Pearce, Patrick Sullivan,
Kathrin Walker, Natasha Watkins, and Dustin Wood. (2004). “Organized
Youth Activities as Contexts for Positive Development.” Positive Psychology in
Practice: 540-560.
Little, Priscilla M.D., Christopher Wimer, and Heather B. Weiss. (2008). “After
School Programs in the 21st Century: Their Potential and What it Takes to
Achieve It.” Harvard Family Research Project, Issues and Opportunities in Outof-School Time Evaluation 10: 1–12.
Mahoney, Joseph L., Maria Parente, and Edward F. Zigler. (2009). “Afterschool
Programs in America: Origins, Growth, Popularity, and Politics.” Journal of
Youth Development 4, no. 3: 25–44.
Mandinach, Ellen B. (2012). “A Perfect Time for Data Use: Using Data-Driven
Decision Making to Inform Practice.” Educational Psychologist 47, no. 2: 71-85.
Mandinach, Ellen B., Margaret Honey, and Daniel Light. (2006). “A Theoretical
Framework for Data-Driven Decision Making.” Paper presented at the annual
meeting of American Educational Research Association, San Francisco, CA.
Mandinach, Ellen B., Luz Rivas, Daniel Light, Cricket Heinze, and Margaret
Honey. (2006). “The Impact of Data-Driven Decision Making Tools on
Educational Practice: A Systems Analysis of Six School Districts.” Paper
presented at the annual meeting of American Educational Research
Association, San Francisco, CA.
Mandinach, Ellen B., Edith Gummer, and Robert D. Muller. (2011). The
Complexities of Integrating Data-Driven Decision Making into Professional
Preparation in Schools of Education: It’s Harder Than You Think. Alexandria, VA,
Portland, OR, and Washington, DC: CNA Education, Education Northwest,
and WestEd. http://educationnorthwest.org/sites/default/files/resources/
Gummer-Mandinach%20full%20report.pdf.
McCombs, Jennifer Sloan, Susan J. Bodilly, Nate Orr, Ethan Scherer,
Louay Constant, and Daniel Gershwin. (2011). Hours of Opportunity,
Volume 3: Profiles of Five Cities Improving After-School Programs Through
a Systems Approach. Santa Monica, CA: RAND Corporation. http://www.
wallacefoundation.org/knowledge-center/after-school/key-research/
Documents/Hours-of-Opportunity-3-After-School-Programs-Systems.pdf.
Nagaoka, Jenny, Camille A. Farrington, Stacy B. Ehrlich, and Ryan D.
Heath. (2015). Foundations for Young Adult Success: A Developmental
Framework. Concept Paper for Research and Practice. Chicago, IL: University
of Chicago Consortium on Chicago School Research. http://files.eric.ed.gov/
fulltext/ED559970.pdf.
National League of Cities. (2012). Building Management Information Systems
to Coordinate Citywide Afterschool Programs. A Toolkit for Cities. Washington,
DC: Institute for Youth, Education & Families. http://www.nlc.org/Documents/
Find%20City%20Solutions/IYEF/Afterschool/YEF_MIS%20Toolkit_2012.pdf.

Roderick, Melissa. (2012). “Drowning in Data but Thirsty for Analysis.”
Teachers College Record 14, no. 11: 1–9.
Russell, Christina A., Monica B. Mielke, and Elizabeth R. Reisner. (2009).
Evidence of Program Quality and Youth Outcomes in the DYCD Out-of-School
Time Initiative: Report on the Initiative’s First Three Years. Washington, DC:
Policy Studies Associates, Inc. http://www.nyc.gov/html/dycd/downloads/
pdf/2012/OST_Evaluation_Report_September2009.pdf.
Simkin, Linda, Ivan Charner, Caitlin Dailey, Eric Watts, Hannah Taub,
and Abidemi Adelaja. (2013). Is Citywide Afterschool Coordination Going
Nationwide? An Exploratory Study in Large Cities. Durham, NC: FHI 360.
Smith, Charles. (2013). Moving the Needle on “Moving the Needle”: Next
Stage Technical Guidance for Performance Based Accountability Systems in the
Expanded Learning Field with a Focus on Performance Levels for the Quality
of Instructional Services. Ypsilanti, MI: David P. Weikart Center for Youth
Program Quality, a division of the Forum for Youth Investment, Updated.
http://www.cypq.org/sites/cypq.org/files/publications/Moving_the_
Needle_FINAL.pdf.
Spielberger, Julie, Tracey Lockaby, Leifa Mayers, and Kai Guterman. (2009).
Ready for Prime Time: Implementing a Formal Afterschool Quality Improvement
System by Prime Time Palm Beach County, Inc. Chicago, IL: Chapin Hall at
the University of Chicago. https://www.chapinhall.org/sites/default/files/
Prime_Time_Report_04_18_09.pdf.
United States. (2011). Office of Management. Department of Education.
Federal Register. December 2, 2011, vol. 76, no. 232, p. 75614. https://www.
gpo.gov/fdsys/pkg/FR-2011-12-02/pdf/2011-30683.pdf.
Vandell, Deborah Lowe, Elizabeth R. Reisner, and Kim M. Pierce. (2007).
Outcomes Linked to High-Quality Afterschool Programs: Longitudinal Findings
from the Study of Promising Afterschool Programs. Washington, DC: Policy
Studies Associates, Inc.
The Wallace Foundation. (2013). Better Together: Building Local Systems
to Improve Afterschool (A Conference Report). New York, NY: The Wallace
Foundation. http://www.wallacefoundation.org/knowledge-center/afterschool/coordinating-after-school-resources/Documents/Better-TogetherBuilding-Local-Systems-to-Improve-Afterschool.pdf.
Weiss, Carol H. (2013). “Knowledge Creep and Decision Accretion.” Science
Communication 1, no. 3: 381–404.
West, Darrell M. (2012). Big Data for Education: Data Mining, Data Analytics,
and Web Dashboards. Washington, DC: Governance Studies, The Brookings
Institution. http://www.brookings.edu/research/papers/2012/09/04education-technology-west.
West, Martin.R. (2014). The Limitations of Self-Report Measures of NonCognitive Skills. Washington, DC: The Brookings Institution. http://www.
brookings.edu/research/papers/2014/12/18-chalkboard-non-cognitive-west.
Yohalem, Nicole, Alicia Wilson-Ahlstrom, and Diana Yu. (2005). Youth
Program Quality Assessment and Improvement: Celebrating Progress and
Surfacing Challenges. A Meeting Report. Washington, DC: The Forum
for Youth Investment, Impact Strategies, Inc. http://forumfyi.org/files/
ProgramQualityMtgRpt.pdf.

Connecting the Dots: Data Use in Afterschool Systems | 70

APPENDIX: INTERVIEW GUIDE ( PHASE 1 )

Protocol for Year 1 Interviews: Data Use in Afterschool Systems
System-level Stakeholders and Research Partners
INTRODUCTION
This interview will help us understand where (city name) is in developing the capacity to
use data. I’ll be asking you about:
•	 the organizations involved in this work and how they collaborate,
•	 what you’ve accomplished together and key developments along the way, and
•	 what has been supportive and challenging.

I also have questions about:
•	 providers’ engagement and concerns and
•	 the current status of data use, including costs and benefits.

Finally, because (city) has a partnership with a research entity, I have some questions to
understand that partnership better.
We have a lot of ground cover so I’ll try to let you know along the way where we are in
the interview and what the topic is. I may ask a question that you may not know how to
answer. Just let me know and we’ll move on. If you have an idea of who might be best able
to respond, that would be helpful.
Before we get into our conversation, I have a list of purposes here describing uses for data.
I’d like you to indicate how important you think each one of them is to (city) on a scale of
one to five, with one meaning not important and five meaning very important. We’ll use
this information in the interview. It’s easiest to ask you for it upfront.
•	 Identifying community needs for afterschool programs.
•	 Increasing student participation in after school programs.
•	 Identifying and reaching out to struggling students for after school programs.
•	 Improving the quality of after school programming.
•	 Monitoring and holding program providers accountable.
•	 Enabling collaboration across stakeholders with different perspectives.
•	 Generating evidence to support public investments in afterschool programming.
•	 Satisfying the reporting requirements of funders.
•	 Informing the decisions of front-line staff.
•	 Informing decisions at the system level.

71 | Chapin Hall at the University of Chicago

BACKGROUND: ROLE AND RESPONSIBILITIES OF INTERVIEWEE
AND THEIR ORGANIZ ATION

Objectives:
•	 To understand the interviewee’s role in the afterschool system,
•	 To understand interviewee’s responsibilities with regard to data use and the
role of their organization in the data use work.

My first set of questions is about your role and responsibilities in the afterschool
system, particularly in relation to data use. As part of this, I have some questions
about what “data use” means to you.
What is your role in the afterschool system in [CITY]? How long have you been involved in
the afterschool system? Are you a volunteer or paid staff member in the afterschool system?
Probe for:
•	 Professional position and title; Role in organization
•	 Length of time in position
•	 How did you become involved in the field of afterschool time?
•	 OR What experience did you have in the field of afterschool prior to this?

What are your responsibilities with regard to use of data in the afterschool system and
how have these changed or developed since the start of the systems work?
•	 What training/experiences have best prepared you for the use of data in your current role?

BACKGROUND TO PARTNERSHIP REL ATIONSHIPS AND GOALS

Objectives:
•	 To understand the partner goals for data use, whether there is consensus on
goals and outcomes, and the nature of the collaboration.
•	 To probe on the school district as a partner.
•	 To also probe on the involvement of the broader stakeholder community in
the partnership.

My next questions are more about the collaboration of the organizations
involved in the data use efforts and their goals.
Would you briefly describe the role of your organization (or committee, etc.) with regard
to the efforts to use data in the afterschool system?
•	 What kinds of skills or capacities does your organization contribute?

Which organizations in the city are most involved in the efforts to develop the capacity
to use data in the afterschool system? Have there been changes in the organizations
involved since the start of the systems’ work and if so, why?
•	 Who do you see as leading the efforts? Have there been changes in leadership and if so, why?
Connecting the Dots: Data Use in Afterschool Systems | 72

What do the partners expect to accomplish by developing the capacity of the
afterschool system to use data?
•	 Do they share the same goals for data use? Why or why not?
•	 Do they agree on shared outcomes? Why or why not?

I have a list of purposes here describing uses for data. I’d like you to indicate how
important you think each one of them is—from not important to very important
and explain why.
•	 Identifying community needs for afterschool programs.
•	 Increasing student participation in after school programs.
•	 Identifying and reaching out to struggling students for after school programs.
•	 Improving the quality of after school programming.
•	 Monitoring and holding program providers accountable.
•	 Enabling collaboration across stakeholders with different perspectives.
•	 Generating evidence to support public investments in afterschool programming.
•	 Satisfying the reporting requirements of funders.
•	 Informing the decisions of front-line staff.
•	 Informing decisions at the system level.
•	 Do you think your partners would agree with you?
•	 Does the list leave anything out?

What has helped the collaboration of the partner organizations around data use since
the systems’ work started? What has hindered the collaboration?
(For select interviewees): What are the interests or aspirations of the school district in
supporting the use of data by the afterschool system? What are its concerns?
In what ways has the broader stakeholder community, for example funders, diverse
providers, community members, parents, or students, been involved in the data use
efforts? What strategies have you found to be effective at engaging them? What have
been the challenges?
•	 Probe in particular on the provider community to understand who is included and

who is left out.
•	 Probe on stakeholders of particular interest in that community, for example, community

groups in Saint Paul.
ACCOMPLISHMENTS AND DEVELOPMENTS TO DATE; FACILITATORS
AND CHALLENGES

Objectives:
•	 To understand what the city has accomplished to date and important decisions
points as related to specific tasks of developing an information system.
•	 To understand challenges and facilitators.

73 | Chapin Hall at the University of Chicago

The questions I ask next are intended to help us understand what you’ve
accomplished so far in moving toward data use in the afterschool system, what
has allowed you to make progress, and what has been challenging for you:
What do you see as the primary accomplishments for the afterschool system as it has
moved toward using data in the last two years?
I’d like to understand the critical or important decision points you’ve gone through in
developing your data system. There are four areas I’d like to focus on in my questions:
•	 First, what were the critical or important decisions points made with regard to designing

the data system? What were primary considerations in terms of collecting data?
Integrating data? What about considerations concerning who would have access? Who
was involved in the design?

•	 Second, what were the critical or important decision points in identifying what data

elements and indicators to include in the system? Who was involved?

•	 Categories of indicators include: Participation, program quality, school outcomes, social/

emotional indicators, juvenile justice, additional demographic, health, child welfare,
professional development		
•	 Third, what were the critical or important decision points in identifying or developing

shared outcomes. Who was involved?

•	 Fourth, what were the critical or important decision points in structuring the collection

and sharing of data, specifically with respect to data sharing agreements and data
ownership? Who was involved? What, if any, considerations were made for issues of
consent or confidentiality?
Do you have plans to connect data on quality with data on participation or outcomes?
Why or why not?

Now that we’ve discussed what you’ve accomplished and the process you’ve
gone through, I’m interested in understanding what has helped (city) or
presented challenges for (city) in making progress toward using data.
How do you think the afterschool system has helped or created challenges in your
efforts to use data?
•	 Probe on coordinating entity—as part of city government or separate, as funder or not.
•	 How do you think mayoral support has helped or created challenges?
•	 How do you think the school district support has helped or created challenges?
•	 Provider community?
•	 Technical expertise available to you?
•	 Research expertise?
•	 Wallace TA providers in particular? Any other technical assistance?
•	 Is there anything else about your city or the resources and support available to you that

you think has been an obstacle or a support in the work you are doing?

Connecting the Dots: Data Use in Afterschool Systems | 74

ISSUES VIS-À-VIS PROVIDERS

Objectives:
•	 To get a sense of the response of providers to an afterschool data system and
how their concerns, if any, have been addressed.
•	 To understand concerns about data collection by providers and if/how these
have been addressed.

Providers are important stakeholders. They are a source for a lot of the data.
They may have their own data systems. They may be held accountable to a lot
of different outcomes. They are (or will be) key users of the system. The next
questions are about providers and their involvement in and response to this work.
What has been the response of providers to the efforts to develop an afterschool data
system? What do they see as the benefits? What have their concerns been? How have
these been addressed?
Probe:
•	 concerns about multiple funders, reporting requirements, outcomes;
•	 concerns about maintaining own data system and/or burden of work in participating

in other systems.
What, if any, are your concerns regarding the collection of data by providers? How have or
are you addressing these?
RESEARCH OR DATA PARTNERS

Objectives:
•	 To gather background information on the relationship between the afterschool
system and the external research/data entity.
•	 To understand perceptions of the value of the external entity’s work and challenges
in the partnership.
(For Project Lead): Do you have a research or data partner? This would be an organization
or an individual who’s working with you to develop the capacity of the afterschool
system to use data.

We’d like to better understand how afterschool systems work with research or
data partner. The next questions are about this partnership.
When and how did the partnership start? How has your work together developed or
changed over time? How has the Wallace project advanced this work?
How do you do your work together around afterschool data use?
•	 how often do you meet
•	 what do you meet about
•	 how do you decide on an agenda for the work
•	 how do you coordinate and manage your work together?
75 | Chapin Hall at the University of Chicago

What contributions has your partner made to building an afterschool system?
OR
What contributions have you made as a partner to building an afterschool system?
What challenges have you experienced working with a research or data partner?
OR
What challenges have you experienced working with the afterschool system?
STATUS OF DATA USE, INCLUDING COSTS AND BENEFITS

Objectives:
•	 To get a sense of how data is being used.
•	 To begin to get at the issue of the value of data use and to specify the activities
and efforts that are “cost ingredients.”

We’ve been talking about the partnership efforts around the development of the
data system. My next questions are about what you’ve learned about the use of
data, including some of the costs and benefits.
Could you describe the ways data is currently being used in the afterschool system and any
plans for using data in the future?
What have you found to be the plusses and minuses for the afterschool system in trying to
make use of data?
•	 Probe: What do you see as the value for kids of the efforts to use data?

*Expanding participation?
*Improving quality?
*Sustaining the afterschool system?
We want to understand what it takes to use data at the systems level in terms of people,
time, money, and effort.
•	 Probe: If you to advise another city interested in using data in its afterschool system, what

would you include in a list for them to consider as important activities or efforts?
What do you expect it will take to sustain data use?

Connecting the Dots: Data Use in Afterschool Systems | 76

About Chapin Hall
Chapin Hall is an independent policy research center at
the University of Chicago focused on providing public and
private decision-makers with rigorous data analysis and
achievable solutions to support them in improving the lives
of society’s most vulnerable children. Chapin Hall partners
with policymakers, practitioners, and philanthropists at the
forefront of research and policy development by applying
a unique blend of scientific research, real world experience,
and policy expertise to construct actionable information,
practical tools, and, ultimately, positive change for
children, youth, and families.
Established in 1985, Chapin Hall’s areas of research include
child and adolescent development; child maltreatment
prevention; child welfare systems; community change;
economic supports for families; home visiting and early
childhood initiatives; runaway and unaccompanied
homeless youth; schools, school systems, and out-ofschool time; and youth crime and justice.

1313 East 60th Street, Chicago, Illinois 60637
www.chapinhall.org

Commissioned by:

5 Penn Plaza, 7th Floor, New York, NY 10001
www.wallacefoundation.org

79 | Chapin Hall at the University of Chicago