Machine Learning, Machine Bias A Systematic Survey on ML-based Flood Model Prediction Clio Tsao 1 Machine Learning, Machine Bias A Systematic Survey on ML-based Flood Model Prediction Clio Tsao UO student studying math and computer science Introduce title and topic “bias in ML-based flood model prediction” Thank PTA Daniel Lowd, HC advisor Trond Jacobsen, Miriam Alexis Jordan for coordinating thesis logistics 1 Interest in ML bias, research gap for climate modeling Floods cost U.S. >$180B annually (JEC 2024) Focus on ML-based flood model prediction and bias 2 Motivations primarily race and gender bias in hiring, criminal justice, and healthcare applications Define what ML is: algorithms that learn from datasets without explicit programming Limitations of traditional flood modeling: extensive hydro-geomorphological monitoring datasets, difficult to make real time predictions due to complexity ML-based models allow for improved data processing capacities, potential for real-time response and warning applications Nonetheless, ML-based modeling face barriers in application due to bias: Is a limitation in trustworthiness/interpretability of applying ML predictions Thus important to research and identify the presence of bias in models; discuss accountability with these uses - Images via https://www.vox.com/climate/2023/7/11/23791452/nyc-flooding-brooklyn-weather-climate-change and https://www.reuters.com/world/asia-pacific/taiwan-clears-up-slowly-re-opens-after-hit-typhoon-krathon-2024-10-04/ Top: NYC flooding in 2023 – just two weeks ago, flash floods killed two in NYC and caused damage across the city Bottom: Kaohsiung flooding in 2024 after Typhoon Krathon – Typhoon Fung-wong causing floods in Taiwan; over 8000 evacuated and 51 injured Joint Economic Committee’s 2024 estimates flood cost (https://www.jec.senate.gov/public/index.cfm/democrats/2024/6/flooding-costs-the-u-s-between-179-8-and-496-0-billion-each-year) Reason for interest Prior research Problem and purpose 2 Bias Machine bias Systematic bias Representational bias Individual fairness vs group fairness 3 What is machine bias? Data bias: systematic errors during in the dataset from collection or processing; leading to skewed or inaccurate results Systematic bias - inherent tendency of a process, system, or institution to support particular outcomes, leading to results that consistently deviate from the true value in a specific direction Representational bias: form of data bias where a dataset fails to accurately reflect the diversity of the population or problem it is intended to model, leading to unfair or inaccurate outcomes for underrepresented groups. Can amplify existing biases Also important to define fairness, a lens through which we can consider bias Individual fairness: assumes that individuals who are similar should be treated similarly; individuals should receive consistent and equal treatment based on their features, regardless of what demographic group they belong to Group fairness: assumes demographic groups should be treated similarly as a whole; protected groups should receive equal outcomes to address historical and structural biases (The (Im)Possibility of Fairness by Friedler et al.) 3 Research Questions 4 RQ1: Bias concerns of ML-based flood prediction models RQ2: Methods for debiasing flood prediction models RQ3: Efficacy of methods in addressing ethical concerns of ML-based flood prediction models RQ4: Directions of future research RQ1: What bias concerns are raised regarding the use of machine learning in flood prediction models? RQ2: What are the most frequently implemented methods for debiasing flood prediction models? RQ3: Based on the findings from RQ1 and RQ2, how closely do proposed solutions address key ethical concerns in the research and development of ML-based flood prediction models? RQ4: What are the recommended directions of future research for addressing bias in ML-based flood prediction modeling? 4 Methods 5 IEEE Xplore, Science Direct, and SpringerLink Zotero for tracking sources Excel for documenting data Quality assessment table guideline by Kitchenham and Charters (2004): identification of the need for a review, 2) specifying the research question(s), and 3) developing a review protocol Talk about inclusion and exclusion criteria Inclusion criteria: The study is written in English. The study is published in a peer-reviewed publication channel: IEEE Xplore, Springer Nature Link, or Science Direct. The study was published between 2020 – 2025. The study is related to flood prediction and bias in machine learning predictive models. Exclusion criteria: The study is not relevant to any research questions. The full text of the study is not available in the search database and accessible using institutional accounts. The study is related to disaster prediction but not machine learning. The study is related to machine learning but not to disaster prediction. The study in a language other than English. The study is not identified as peer reviewed. The study is duplicated. 5 Findings RQ1: What bias concerns are raised regarding the use of machine learning in flood prediction models? 6 Bias Concerns Count Insufficient data, limited monitoring 8 Model bias 5 Data imbalance 4 Data privacy 2 Computational constraints 2 Algorithmic transparency, accountability 1 Unequal access to climate prediction tools 1 Interpretability barriers 1 Overburdening of certain socio-demographic populations 1 Selected a total of 26 papers for review based on the search criteria Count: papers are organized in terms of most frequently mentioned to least (thus more prevalent/crucial – not necessarily by severity) 6 Findings RQ2: What are the most frequently implemented methods for debiasing flood prediction models? 7 Debiasing methods Count Statistical bias correction 8 Model development 5 Data augmentation 4 Crowdsourced/open-source data 3 Federated learning 1 Post-hoc interpretation methods 1 Data augmentation: oversampling methods like SMOTE (to minimize bias toward the minority class/extreme events) Statistical bias correction: post-processing technique used to adjust model outputs to match observational data, by identifying and correcting systematic errors in a model's output for a historical period and then applying that correction to future projections Example: Multivariate Hawkes process - statistical model that describes relationships between multiple events across time (temporal relation); the occurrence of an event in one process can increase the probability of future events in the same or other processes post-hoc interpretation method: evaluate feature effects for hydrometeorological prediction - MSE-based feature importance evaluation 7 Findings RQ3: Based on the findings from RQ1 and RQ2, how closely do proposed solutions address key ethical concerns in the research and development of ML-based flood prediction models? Effective for technical definitions of bias Limited for interpretability and representational bias 8 oversampling tends to yield more accurate models than undersampling Ensemble methods tend to be effective in mitigating individual model biases May not be effective in reducing burden on vulnerable communities 8 Findings RQ4: What are the recommended directions of future research for addressing bias in ML-based flood prediction modeling? 9 Future research Count Generalizability 9 Expanding datasets 9 Compare against alternatives 8 Integration (of other techniques, models) 5 Validation/Follow-up monitoring 4 Scaling model application/capacity 3 Improve interpretability 2 Enhancement/Further development 2 Qualitative research of communities 1 Integration of explainable AI (XAI) to improve intepretability (use ML, statistics, human-computer interactions to explain AI outputs – helps build accountability/trust) Auditing Count: many of the papers focus on generalizability (important for applications in real world) but also fall short on other key considerations for real world applications like interpretability (most direct next steps vs long term improvements to applicability) Need for interdisciplinary collaboration between ML researchers, policy experts, and community representatives to define protocols that acknowledge the existence of biases and make the most of the valuable insight that AI can provide while maintaining human judgment 9 Limitations 10 Limited number of papers Limited time period (recent 5 years) Scope of search string Scope of search string: does not bring up studies that don’t address bias at all 10 Future Research 11 Compare to ethical regulations in other fields Compare cost of computational demands vs provided benefits Identify data-poor regions to guide further data collection Scope of search string (results may not capture scope of papers that don’t discuss bias at all) Machine learning is a promising next step to enhance the abilities of flood model prediction, and working on this project has given me a greater appreciation for the nuance of bias and fairness in the AI-age. 11 Thank you! 12 Thank you! 12 image1.jpeg image2.jpeg image3.png image4.png