CEBMa’s Online Course Modules
In partnership with Carnegie Mellon University’s Open Learning Initiative (OLI), and with financial support from the Studer Foundation and New York University’s Kovner Fund, CEBMa has developed an online course on evidence-based management (EBMgt).
The course includes 15 modules that will develop students’ EBMgt skills and enhance their understanding of how an evidence-based approach can support organisations’ decision-making. The course combines an interactive textbook with learning activities, quizzes and assessments. An overview of all modules and learning outcomes can be found below.
CEBMa’s professional members have access to all 15 modules and receive an official CEBMa/CMU certificate of completion. Educational institutions and individual lectures can use the modules in their teaching (see below).
|Module||After completing this module, students will be able to ...|
|1. The Basic Principles||Summarize the basic principles of evidence-based management; Explain why we need evidence-based management; Explain what counts as evidence; Determine which sources of evidence were consulted; Assess (coarsely and in general terms) the quality of evidence; Determine whether the 'best available' evidence was used in a decision-making process; Correct common misconceptions about evidence-based management.|
|2. ASK||Identify (hidden) claims/assumptions regarding a practical issue; Determine whether an (assumed) problem is sufficiently clear; Determining whether there is sufficient evidence to support the (assumed) problem; Determine whether the preferred solution is sufficiently clear; Determine whether there is sufficient evidence (from multiple sources) to support the preferred solution.|
|3. ACQUIRE - evidence from practitioners||Determine what evidence to acquire from practitioners; Determine how to prevent selection bias when acquiring evidence from practitioners; Determine the best method(s) to acquire evidence from practitioners; Determine whether bias could have affected evidence from practitioners; Formulate clear, unambiguous, and unbiased questions.|
|4. APPRAISE - evidence from practitioners||Assess whether professional experience is valid and reliable; Grade the trustworthiness of professional experience; Recognize how system 1 thinking influences valid and reliable professional expertise; Determine whether a decision is based on system 1 or system 2 thinking; Recognize common cognitive biases; Identify ways cognitive biases can be overcome; Critically appraise evidence from practitioners.|
|5. ACQUIRE - scientific evidence||Determine the most relevant online research database(s) given the question; Determine whether a journal is peer reviewed; Determine the most important PICOC terms; Search the Internet for relevant alternative and/or related terms; Search Google Scholar for related or broader academic terms; Test search terms to identify terms that yield the most relevant results; Apply Boolean operators to specify a search query; Use the history function to combine search queries; Apply methodological filters to identify meta-analyses and/or longitudinal/controlled studies; Narrowing search results by adding additional PICOC terms; Limit a search result by limiting the date range.|
|6. A short Introduction To Science||Assess whether a study was conducted according to the scientific method; Recognize pseudo-science; Assess whether a statistically significant finding is of practical relevance; Assess whether methodological bias may have affected the results; Determine whether confounders may have affected the results; Assess whether a placebo effect may have affected the results; Identify moderators or mediators that may have affected the results; Distinguish quantitative research methods from qualitative research methods; Determine a study's research design; Efficiently read a research paper.|
|7. APPRAISE - scientific evidence||Assess the impact of an effect size; Assess whether a statistically significant finding is of practical relevance; Assess whether a confidence interval is sufficiently narrow; Assess whether an outcome was measured in a reliable way; Distinguish cause-and-effect questions from non-effect questions; Determine a study's research design; Assessing whether a study's research design is appropriate given the research question (methodological appropriateness); Assessing a study's methodological quality; Grading a study's trustworthiness on the basis of it's methodological appropriateness and quality; Summarizing a study's main findings, weaknesses, and overall trustworthiness.|
|8. ACQUIRE - organizational evidence||Explain the added value of organizational evidence; Distinguish data, from information and evidence; Determine what organizational evidence to acquire; Determine what types of organizational evidence are available and where they are kept; Distinguish 'normal' data from Big Data; Determine where to find relevant organizational evidence; Determine whether the evidence concerns operational data, metrics, KPIs or benchmarks; Explain the difference between descriptive and inferential measures; Identify potential barriers to acquire organizational evidence.|
|9. APPRAISE - organizational evidence||Determine whether a logic model was used to collect and anaylze evidence from the organization; Assess whether organizational data are relevant; Identify steps in the collection and processing of data that could introduce risk of inaccurate data; Determine whether contextual information is missing; Determine whether there could be measurement error; Assess whether there could be a small number problem; Determine whether a metric is a good representation of the data; Interpret a metric's standard deviation; Assess whether a graph represents the data in a valid and reliable way; Interpret a correlation or regression coefficient; Determine whether a correlation- or regression coefficient is practically relevant; Assess whether there are outliers that may distort the evidence; Assessing whether range restriction may have affected the evidence; Assess whether a confidence interval is sufficiently narrow.|
|10. ACQUIRE - stakeholder evidence||Identify and assess evidence from stakeholders; Identify and distinguish different types of stakeholders; Determine which stakeholders could be affected by a decision; Determine which stakeholders could affect a decision, its implementation, or its outcome; Identify the most relevant stakeholders; Determine how to acquire evidence from stakeholders in a valid and reliable way.|
|11. APPRAISE - stakeholder evidence||Explain why stakeholders' subjective feelings and perceptions should always be taken into account; Determine the practical and/or ethical impact a decision may have on stakeholders; Determine whether relevant stakeholders can freely express their views and feelings regarding a (proposed) decision; Determine whether there could have been selection bias in the way evidence from stakeholders was obtained; Determine whether the evidence from stakeholders is sufficiently representative.|
|12. AGGREGATE||Explain what proof, evidence, chance, and 'conditional' probability means; Assess the impact of a prior probability; Estimate the likelihood of the evidence: P(E|Htrue) and P(E|Hfalse); Update the probability of a claim/assumption/hypothesis when new evidence comes available; Aggregate evidence from multiple sources by applying Bayes Rule.|
|13. APPLY||Use the PICOC method to determine whether the evidence applies to the organizational context; Determine whether a decision/intervention gives you the biggest bang for your buck; Assess the level of risk inherent in a decision/intervention; Identify ethical issues that need to be considered; Determine whether (and if so, how) the evidence is actionable; Determine whether there are moderators that need to be taken into account; Determine, given the type of decision at hand, how and in what form the evidence can be applied.|
|14. ASSESS||Identify the type of decision (to be) made (routine, non-routine, or novel/hyper complex); Determine whether a decision was executed as planned; Assess an outcome using the gold standard method; Assess an outcome using quasi- or non-experimental methods; Suggest ways to improve the validity and reliability of an outcome assessment; Assess whether an outcome was measured in a reliable way; Assess whether indirect and intangible costs were taken into account; Assess the (unintended) consequences of a decision on stakeholders.|
|15. Building EBMgt Capacity||Explain how to develop a questioning mindset; Explain how to make more mindful and
explicit decisions more explicit; Create opportunities for people to practice and apply
learning; Explain why evidence alone does not change people’s minds; Illustrate how to
give people’s minds a way out; Help people to develop a new operating logic; Explain why
increasing people’s accountability leads to more information-seeking behaviour; Explain
why it is important to take small steps; Determine the best way to build evidence-based
capacity in your organization.
Information For Lecturers
Educational institutions and individual teachers can use the modules in their teaching for a small fee. You can use (some, or all) modules as a stand-alone course, or integrate them in an existing course or program. The course modules are accessible through the CMU’s OLI website or through your institution’s Learning Management System. The modules can be completed in students’ own time and at their own pace. Each module takes, on average, 3 to 5 hours to complete. Student data is displayed in the instructor’s Learning Dashboard. It includes information about student participation and performance at both the class/cohort and the individual level. This way instructors can quickly see how many students have attained each sub-objective (skills) and which ones are struggling.
If you would like to use CEBMa’s course modules,