Generative AI Research Program

Deadline: Sunday, April 28, 2024 at 11:59pm

Application instructions: https://uraf.harvard.edu/sites/projects.iq.harvard.edu/files/uraf/files/app_genai.pdf

Brief Description

The Generative AI Research Program is launching for the summer of 2024, with the support of the Office of the Vice Provost for Research and the Office of Undergraduate Research and Fellowships. This program provides students with diverse research opportunities in Generative AI across an exciting range of research settings. Students will contribute to an important phase of academic discovery in Generative AI at Harvard, through work on focused projects with Harvard faculty in a variety of disciplines.

Program Overview

This program is intended to enable Harvard undergraduates with an interest in GenAI to work closely with a member of the Harvard faculty on a research project in this area. The intent of the program is to provide a formative and substantive research experience over ten weeks of the summer, working on a project designed by specified Harvard faculty. Prospective participants will indicate preferences from an array of proposed research projects as part of the application; students may apply to up to three different projects. Selected participants will then be matched with a research project, with one or two students per project.

Benefits

A living stipend of $7,000 to support 10 weeks of full-time research

Eligibility

Any continuing Harvard College undergraduate student in good standing.
Must be able to commit 10 weeks to summer research.
May accept funding for only one Harvard-funded summer experience, per the Harvard College Summer Funding Policy.

Students in any concentration may apply. A range of priority experience with GenAI or related technologies is expected in the applicant pool, and matching will be based on project-specific needs.

Summer 2024 Project Descriptions are now all posted; please see options in the drop-down menu below!

Summer 2024 - Project Descriptions

Gen AI 2024 - Amin (Project #1)

Project Supervisor:

Nada Amin, Assistant Professor of Computer Science (FAS)

Project Title:

Improving Generative Programming via Search, Execution and Constraint Solving

Project Overview:

This project builds on our work on verified multi-step reasoning using Large Language Models (LLMs) and Monte Carlo Tree Search (MCTS). https://github.com/namin/llm-verified-with-monte-carlo-tree-search https://arxiv.org/abs/2402.08147 In this work, we showed that it is possible to synthesize a verified program comprising a datatype for arithmetic expressions, an evaluator, an optimizer and a proof that the optimizer preserves the semantics as defined by the evaluator. We call our prior approach VMCTS, because a verifier is used to predict whether an LLM completion is fruitful. We propose to explore multiple avenues for using LLMs and logic engines (such as verifiers, SMT solvers, interpreters, &amphellip;) in tandem. The LLMs provide fruitful suggestions and the logic engines ensure truthful outcomes. Specifically, we propose two different avenues, one for each student: (1) We propose to develop a synthesis sketching method, where we have a program containing holes and we use constraint solving and LLMs to fill those holes, based on specifications such input/output examples. (2) We propose to extend our VMCTS approach to work with Python programs instead of verified programs (like in Dafny, Coq, Lean). The idea is to use test cases to guide the multi-step reasoning. More broadly, based on proposals (1) and (2), we will need to develop methods to execute partial programs, and combine search and execution. The students can work together on these methods.

Opportunity for the Fellow:

Students will learn to develop software using open LLMs. They will write search and execution algorithms that combine with LLMs. They will learn about software verification, SMT (Satisfiability Modulo Theories) solvers, programming language theory, and AI.

Outcomes:

Software synthesis tools that perform well on benchmarks such as https://github.com/facebookresearch/cruxeval https://github.com/microsoft/PythonProgrammingPuzzles Some new methods for combining LLMs and logic engines

Keywords:

generative programming, verification, synthesis, SMT solvers

Gen AI 2024 - Amin (Project #2)

Project Supervisor:

Nada Amin, Assistant Professor of Computer Science (SEAS)

Project Title:

LLMs applied to Precision Medicine

Project Overview:

Biomedical knowledge graphs contain subject-predicate-object triples such as "imatinib treats cancer" and "androgens upregulate TMPRSS2". By performing an inference over a large biomedical graph, one can predict potential new treatments. For example, SARS-CoV-2 cell entry depends on the gene TMPRSS2, so an anti-androgen might help mitigate COVID. This methodology is particularly fruitful to treat rare monogenic diseases, because one can target a mutant gene for regulation through repurposing FDA-approved drugs. This project will create a natural language interface for analysts to perform inferences over a knowledge graph to predict potential treatments, without the burden of writing low-level queries and with the flexibility of carrying out investigations beyond canned queries. The LLM responses should include provenance information, so that the claims are backed up by the scientific literature or other evidence (such as p-values from an experiment). This project will also focus on ranking results. There are multi-dimensional criteria: confidence that the result is correct, safety profile of the predicted drug, novelty of the result, relevance to the query. This project will explore using an LLM to automatically sift through the literature with the precise requirements at hand.

Opportunity for the Fellow:

Students will learn about how to combine structured information such as biomedical knowledge graphs with LLMs. They will research and implement different options to support this integration (for example, RAG and fine-tuning). They will also learn to use LLMs to extract and cross-check claims in the literature. These are open ended problems and tasks.

Outcomes:

Solutions to integrate biomedical knowledge graphs with LLMs to create a biomedical-fluent hybrid LLM system. Solutions to rank predictions based on precise requirements.

Keywords:

precision medicine, knowledge graphs, LLMs

Gen AI 2024 - Bol

Project Supervisor:

Peter Bol, Charles H. Carswell Professor of East Asian Languages and Civilizations (FAS)

Project Title:

Discovering Social Networks in Chinese History

Project Overview:

The China Biographical Database is a freely accessible relational database with biographical information about approximately 535,181 individuals as of Feburary 2024, currently mainly from the 7th through 19th centuries. With both online and offline versions, the data is meant to be useful for statistical, social network, and spatial analysis as well as serving as a kind of biographical reference. The long term goal of CBDB is systematically to include all significant biographical material from China's historical record and to make the contents available free of charge, without restriction, for academic use. That data is regularly being enriched and new biographical entries are being created for Tang, Five Dynasties, Liao, Song, Jin, Yuan, Ming and Qing figures. CBDB is the most extensive and detailed prosopographical database ever developed for China's history. It is used around the world and has provided the data for a number of path-breaking books and articles. Or what were the spatial and temporal distributions of the subjects about which books were written. However, our traditional methods have been an abject failure in correctly identifying the social relationships between the persons who are mentioned in a biographical text. This is because in contrast to data relating to kinship and career, social relationships are not expressed in regular patterns. We have experimented with some of the AI models in the Harvard sandbox and believe this could be a solution. Our plan, should we have the help, is to apply AI to the biographies included in the Chinese dynastic histories, covering the period 589 CE to 1911. There are about 9000 such biographies available to us in searchable text files. We have a considerable amount of training data, allowing us to do some fine tuning, and we have scholars here and in China who can evaluate outputs.

Opportunity for the Fellow:

Students will learn how to build a relational database, the traditional computational methods from the first step of OCRing digitized texts, through extracting and coding data, to the final process of disambiguation and integration into the database.

Outcomes:

We hope to have fine-tuned a model that can then be applied to the tens of thousands of biographical entries in the 8000 local histories from later imperial China.

Keywords:

Chinese history, prosopography, biography, data-mining

Gen AI 2024 - Brennan

Project Supervisor:

Karen Brennan, Timothy E. Wirth Professor of Practice in Learning Technologies (HGSE)

Project Title:

Enhancing K-12 Computer Science Education: ChatGPT-Powered Simulations for Debugging Student Projects in Scratch

Project Overview:

Project-based learning can be an effective approach to advancing equity and inclusion in computer science education (Hasni et al., 2016). In these types of learning opportunities, students are able to work on personally meaningful programming projects, bringing their identities and interests into the classroom. However, a challenge of including more opportunities for project-based learning in K-12 computer science education is the need to develop teachers' pedagogical content knowledge; to develop teachers' understanding of content and practices specific to computer science, as well as teachers' understandings of how to teach computer science (Hubbard, 2018). One way to support teachers' pedagogical content knowledge is through professional learning experiences where teachers can practice engaging in the types of interactions which might occur in the classroom (Levin & Muchnik-Rozanov, 2023). When students work on programming projects, they will inevitably run into errors, also known as bugs. Opportunities to practice debugging programming projects can support teachers in developing greater confidence and capacity in supporting project-based learning.

Members of my research group at the Harvard Graduate School of Education, the Creative Computing Lab, are designing an online professional learning experience to help K-12 teachers practice debugging Scratch programming projects. Scratch is a block-based programming language widely used by students ages 8-16 around the world. ChatGPT is already being used in computer science education to support novice programmers in debugging (Kosar et al., 2024). We are leveraging ChatGPT's abilities to debug programming projects and emulate human conversation to help teachers practice talking to students about their work. The core of the professional learning experience is a set of simulations where teachers can interact with a fictional student (a chatbot powered by OpenAI's GPT 4) to debug various Scratch projects. These simulations are organized around 10 common misconceptions which novices face when learning to create Scratch projects (Frädrich et al., 2020). For example, here is an excerpt of a conversation between a teacher and a chatbot, where the chatbot roleplays as a 5th grader with an error in their Scratch programming project: the background of their animation will not change. Through engaging with the chatbot, teachers can expand their content knowledge, by developing deeper understandings of various common errors which students encounter. Teachers can also expand their set of pedagogical strategies and work on their ability to converse with students about their code, rather than directly intervening and fixing things for a student. In each simulation, there are multiple solution states; teachers will be provided with additional information about these various solutions, which will support teachers in developing understandings that there are multiple ways to address most computational errors. To develop these simulations, we draw on the research literature about common novice programmer errors, as well as our prior work in designing Scratch curriculum for K-12 classrooms and working with classroom computing teachers to understand what challenges they face in their work.

The research questions we are investigating in this project are: What are teachers' experiences of interacting with ChatGPT as a practice space for debugging Scratch projects? What are the affordances and limitations of ChatGPT for designing professional learning simulations for teachers? Through this project, we aim to understand how chatbots can support teachers in engaging in low-stakes, high-frequency practice with debugging student projects, and how this practice can support teachers' pedagogical content knowledge and their capacity for facilitating opportunities for students to engage with computer science learning opportunities.

References: Frädrich, C., Obermüller, F., Körber, N., Heuer, U., & Fraser, G. (2020). Common bugs in Scratch programs. Proceedings of the 2020 ACM conference on innovation and technology in computer science education, 89-95. Hasni, A., Bousadra, F., Belletête, V., Benabdallah, A., Nicole, M.-C., & Dumais, N. (2016). Trends in research on project-based science and technology teaching and learning at K-12 levels: A systematic review. Studies in Science Education, 52(2), 199-231. Hubbard, A. (2018). Pedagogical content knowledge in computing education: A review of the research literature. Computer Science Education, 28(2), 117-135. Kosar, T., Ostoji , D., Liu, Y. D., & Mernik, M. (2024). Computer science education in ChatGPT era: Experiences from an experiment in a programming course for novice programmers. Mathematics, 12(5), Article 629. Levin, O., & Muchnik-Rozanov, Y. (2023). Professional development during simulation-based learning: Experiences and insights of preservice teachers. Journal of Education for Teaching, 49(1), 120-136.

Opportunity for the Fellow:

We are looking to collaborate with two undergraduate students to support the revision of our professional learning simulations, prepare data collection instruments to assess the efficacy of the simulations, engage in data analysis of the transcripts of teacher-chatbot interactions, and support the facilitation of the professional learning experience. This work will be a combination of independent and collaborative work with Paulina Haduong (postdoctoral fellow), Jacob Wolf (doctoral student), Brian Yu (research associate), and Karen Brennan (Timothy E. Wirth Professor of Practice in Learning Technologies, and Faculty Affiliate, Computer Science). We are excited to work with two students because the nature of this project requires opportunities to test prototypes with one another. In prior work with undergraduate students, we have found that undergraduates have often spoken about how helpful it has been to collaborate not only with graduate students and researchers, but also to have opportunities to learn alongside other undergraduates. These positions are designed to support undergraduates in developing fluency with quantitative and qualitative research methods and familiarity with the disciplines of computing education research and teacher professional learning.

Activities for this project include: In-person meetings and co-working sessions: Students will engage in weekly meetings and co-working sessions with the Creative Computing Lab. Design journal: Students will keep a design journal, documenting weekly progress on their research work and reflecting on what they have learned. Prompt engineering and web development: Students will test and revise prompts with the OpenAI GPT API, to check for how well the chatbot reacts like a 5th grader might, as well as how accurate the code solutions are. Data collection instruments: Students will develop and test pre- and post-surveys and interview protocols for the professional learning experience. Data analysis: Students will engage in data analysis of transcripts of teachers' interactions with the chatbots, generating thematic codes, developing a codebook, and using natural language processing methods to analyze a large corpus of data. Presentations: Students will give a mid-point and final research update presentation to members of the lab, in addition to engaging in bi-weekly lab meetings to learn more about other projects going on in the lab.

The plan for the 10 weeks is as follows: Week Phase Activities 1 Onboarding Both students: Set up design journal and weekly meeting times; discuss students' learning goals for the summer and develop individual workplans tailored to students' interests. Begin explorations of simulations. 2-3 Simulation Revision and Testing Student 1: Test and revise simulations and the web interface for interacting with the simulations, preparing them to be publicly available for the online learning experience. Student 2: Develop prototypes of the archived professional learning experience. 4-5 Data Collection Preparation Student 1: Develop and test survey instruments to evaluate effectiveness of the simulations. Student 2: Develop interview protocols. Both students: Give a mid-point research update presentation to members of the lab. 6-7 Professional Learning Experience Student 1: Support facilitation and provide technical support for online professional learning experience, responding to any technical challenges which users face. Student 2: Begin examining transcripts of user interactions with chatbots and making real-time revisions as needed. 8-9 Data Analysis and Archive Deployment Student 1: Begin preliminary data cleaning and analysis of transcripts and survey instruments; support selection of participants for interviews. Student 2: Deploy archive of professional learning experience online. 10 Wrap-up Document current work and wrap-up data analysis, writing a reflection on the learning experience and giving a final presentation to members of the lab.

Outcomes:

This summer, we will be revising our prototypes of professional learning simulations, preparing our data collection instruments (e.g., surveys, interview protocols), and launching and facilitating the two-week long online professional learning experience. Data collection methods include collecting transcripts of teachers' interactions with the chatbots, pre- and post-experience surveys, and conducting interviews with a subset of teachers about their experiences. Our previous online professional learning experiences have involved 300-400 teachers from around the world; we expect to engage with a similar number of participants, and to conduct interviews with 10-15 teachers. We anticipate the following set of outcomes: 1. A professional learning experience: We will facilitate a two-week online professional learning experience for K&ampndash;12 computing teachers, to support their fluency with programming with Scratch. This online learning experience will involve a set of (1) daily emails to teachers, to frame the content of the simulation; (2) the simulations, which will each focus on a different computational concept 2. An archive of the experience: After the experience, we will turn the synchronous experience into an asynchronous and interactive archive, hosted on our lab website, for teachers to engage in the simulations at their own pace, and view recordings and transcripts of how others have worked through the simulations of students' challenges with programming projects. 3. Data collection and research article development: We expect to collect a large corpus of text data of teachers' interactions with the chatbot. While we do not expect data analysis to be complete over the summer, we expect to begin preliminary data analysis, which will inform the development of research articles, to be submitted to computer science and learning sciences journals and conferences. If students are interested, we look forward to further collaboration during the academic school year as well.

Keywords:

chatbots, simulations, project-based learning, debugging, Scratch, K–12 computer science education

Gen AI 2024 - Calmon

Project Supervisor:

Flavio Calmon, Assistant Professor of Electrical Engineering (SEAS)

Project Title:

Generative Fairness

Project Overview:

Over the past few years, research in generative AI has unlocked new image, language, and video generation capabilities. Generative AI models are shifting the landscape in applications ranging from drug discovery to art and education. However, generative models also expose new challenges, such as the widespread propagation of fake and biased content. In response to these challenges, governments are enacting policies to protect users against discrimination by automated systems, such as the AI Bill of Rights and the California Consumer Protection Act. The increased scrutiny of machine learning systems by policymakers motivates a need for methods to 1) interpret the reasoning behind generative AI behavior and 2) develop solutions to ensure they are used in a fair and safe manner.

This project focuses on mechanistic interpretability (MI) in generative AI and large language models, particularly those used for image and language generation. We aim to quantify the information exchange between input queries and generated outputs, enhancing transparency and interpretability. Analogous to neuroscience techniques probing neural activity for understanding human behavior, we seek to unravel how neural network components contribute to interpretable features in language and images. We will delve into MI techniques applied to complex architectures like transformers and U-nets; the latter being essential for diffusion models. Mechanistic interpretability is vital for transparency and trust in models like GPT, Claude, Sora, Gemini, and Stable Diffusion, which are deployed across natural language processing, content generation, and decision support systems. Understanding these models' inner workings, decision-making processes, attention mechanisms, and hierarchical structures is critical to achieving interpretability. Recent works focus on the so-called "sparse feature circuits." These are causally implicated subnetworks of human-interpretable features for explaining language model behavior. The key idea is to use sparse auto-encoders (SAE) to identify the interpretable "atoms" in a neural network with many parameters since studying individual neurons may not necessarily yield interpretable results. SAE finds the elements that represent human-interpretable features in a latent space.

The student working on this part of the project will study and leverage dictionary learning concepts used for this mechanistic interpretability technique. Moreover, we will investigate the information-theoretic limits of such techniques. In its second phase, we will study discrete diffusion models that recently emerged for text generation tasks. Such models rely on continuous diffusion models renowned for generating image samples of remarkable quality and diversity while preserving essential structural information compared to other generative models. Their discrete counterparts have been presented as an alternative for transformer-based autoregressive models which struggle to perform several tasks such as infilling text given a prefix and a suffix as the model completes text from left to right. These discrete diffusion models have recently surpassed the capabilities of GPT-2, demonstrating their efficacy. The idea is to use the techniques developed in the first phase to understand the fundamental semantic blocks of generative models. The project's third phase explores practical solutions to bias in generative image models. Google's recent rollout of image generation via Gemini AI drew attention to generating biased images, underscoring the need for fair generation. Fine-tuning on representative datasets is one approach, but its computational demands and inability to account for all biases pose challenges.

Our objective is to propose a simplified approach to fair image generation without fine-tuning, utilizing oversampling and subselection techniques. By querying the model for a larger number of images and selecting a representative subset, we aim to mitigate biases. Leveraging techniques from statistics and information theory, such as rejection sampling, will allow us to precompute the expected number of generations needed for fair outputs. We'll also investigate the complexities of this framework when considering intersections of sensitive attributes like race, gender, and age. Ultimately, we intend to unify both aspects of the project and how techniques driven by information theory and dictionary learning can help leverage interpretability and fairness in generative AI.

Opportunity for the Fellow:

This 10-week project centered on mechanistic interpretability and fairness within the realm of generative AI presents a unique opportunity for undergraduate students at Harvard to develop skills in AI and the mathematical fundamentals necessary to work in this field. Throughout the internship, the students will delve into cutting-edge research at the intersection of interpretability, fairness, statistics, information theory, and advanced AI techniques. Another additional feature is access to a rich network of researchers and to the AI safety team at Harvard through Dr. Claudio Mayrink Verdun (a post-doctoral fellow at PI Clamon's group). It will also provide mentorship and guidance, fostering a collaborative environment conducive to intellectual growth and exploration. Engaging with leading-edge concepts such as sparse autoencoders, compressive sensing, fairness, and diffusion models for image and natural language generation will enable the student to understand the theoretical underpinnings of these methodologies and gain hands-on experience in their practical application.

By grappling with real-world challenges and contributing to ongoing research at PI Calmon's group, the student will develop critical analytical and problem-solving skills in the AI field. The project requires two students for the interpretability and fairness components, respectively. By the end of the program, students will acquire expertise in applying theoretical concepts to research, understanding models, employing visualization methods, engaging in critical analysis, and collaborating effectively within a team. The students are expected to work directly with Dr. Claudio Mayrink Verdun and Dr. Sajani Vithana, both post-doctoral fellows at SEAS Harvard, and with Alex Oesterling, a PhD student working under the supervision of Dr. Flavio du Pin Calmon and Dr. Hima Lakkaraju. Undergraduate students will have bi-weekly meetings with following senior members of the team (in addition to PI Calmon): Dr. Claudio Mayrink Verdun is a research scholar at Harvard with a PhD in mathematics (summa cum laude) from the Technical University of Munich, Germany, with a focus on optimization, signal processing, and information theory for machine learning and AI. He contributed to provable optimization algorithms in machine learning, uncertainty quantification techniques and high-dimensional statistical techniques used in AI. He is a member of the AI safety team and working on mechanistic interpretability and fairness. Dr. Sajani Vithana is a research scholar at Harvard with a PhD in Electrical and Computer Engineering from the University of Maryland, College Park. Her research interests include information theory, distributed coded computing, and machine learning. Alex Oesterling is a G2 PhD Student in Computer Science at Harvard and is interested in fair, interpretable, and trustworthy machine learning, and his current projects apply information-theoretic tools to problems in fairness and interpretability. In summary, the two students will have the opportunity to: Conduct research at the intersection of computer science, applied mathematics, ethics, and policy. Engage with state-of-the art generative models and develop programming skills in a research environment. Learn theory and applications related to probability, statistics, optimization and information theory. Collaborate with researchers to develop mathematical models, generate algorithms and design experiments Connect to the members of AI Safety Student Team (https://haist.ai/).

Outcomes:

Week 1-2: Familiarize with literature on mechanistic interpretability and fairness in generative AI. Define key terms and concepts related to fairness metrics and mechanistic interpretability. Week 3-4: Student 1. Identify existing fairness metrics for image generation and assess their limitations. Student 2. Identify existing techniques in mechanistic interpretability with sparse autoencoders and assess their limitations. Begin developing new fairness metrics considering intersectional fairness aspects. Week 5-6: Student 1: Implement initial version of fairness metrics and evaluate their effectiveness using synthetic data. Student 2: Implement dictionary learning and sparse autoencoders on toy language models. Collect feedback and iterate on the metrics to enhance their robustness. Week 7-8: Student 1: Start designing the practical algorithm for fair generative AI, incorporating insights from literature review and initial metric development. Student 2: Leverage techniques from compressed sensing to improve on sparse autoencoder methods in toy settings and expand results to real-world data. Develop prototype versions of the algorithm and test them on small-scale datasets. Week 9: Student 1: Refine the fair generative AI algorithm based on test results and feedback. Student 2: Benchmark improved autoencoder techniques against state-of-the-art methods in mechanistic interpretability. Conduct preliminary experiments to evaluate the algorithm's performance and fairness. Week 10: Finalize the respective algorithms, documenting their implementation and performance metrics. Prepare a presentation or report summarizing the project outcomes, which will be turned into a paper. This will include the developed algorithm, benchmarks, and novel theoretical contributions in the form of metrics or rigorous results.

Keywords:

Diffusion models, Mechanistic Interpretability, Fair Machine Learning, Rejection Sampling, Compressive Sensing, Dictionary Learning

Gen AI 2024 - Chetty

Project Supervisor:

Raj Chetty, William A. Ackman Professor of Public Economics (FAS)

Project Title:

Deploying GenAI Tools to Improve Social Science Instruction

Project Overview:

This project is to develop new customized generative artificial intelligence tools for our class Economics 50 Using Big Data to Solve Economic and Social Problems. For example, in our Spring 2024 class we launched a customized "tutor bot" trained on the corpus of course materials. The "beta version" of the tool was integrated with Slack, allowing students to interact with the AI in the same way that they interact with the course staff.

Opportunity for the Fellow:

Raj Chetty and Gregory Bruich are developing customized generative artificial intelligence tools for our class Economics 50 Using Big Data to Solve Economic and Social Problems, a course that has approximately 500 students enrolled. The tools will be used to provide real-time feedback to students on their coursework (e.g., answering questions about the content or help with coding). They may also be used in pilot form to help with grading. We are seeking two students with expertise in developing such tools to support our efforts.

Outcomes:

Better feedback for students, more efficient grading, evidence on a path to improving undergraduate education using AI tools.

Keywords:

Economics, Scaling Courses, Teaching, Social Impact

Gen AI 2024 - Dell

Project Supervisor:

Melissa Dell, Andrew E. Furer Professor of Economics (FAS)

Project Title:

Comparing the Performance of Generative AI to Fine-Tuned Models, Across Diverse Social Science Applications and Languages

Project Overview:

I am an economist, and my research focuses primarily on the use of artificial intelligence (both computer vision and natural language processing) to curate social science data at scale, with an emphasis on low resource settings. In a review article that I am currently preparing for the Journal of Economic Literature - a high impact economics journal - a key focus is comparing the performance of generative AI to bespoke models for common tasks in social science research. We do this across a variety of different languages, applications, and historical and modern settings. The article that I am preparing is meant to serve as a high profile, detailed guide for economists and other social scientists considering whether Generative AI is the right tool for their work. It will be accompanied by a website with numerous resources and tutorials that is kept up-to-date, as well as open-source packages. Thus far, with extensive involvement from Harvard undergraduate Ras, we have compared performance on topic classification (19 different topics on English texts, comparing GPT 3.5 and GPT 4 to fine-tuned RoBERTa distil and large), embedding models for topic classification (in English), record linkage (in seven languages and multilingually), and product and industry aggregation (across three languages). I am requesting funding for two undergraduate Ras to significantly extend these analyses. They will add additional common tasks such as named entity recognition and sentiment classification, and also expand the languages covered. This will give them the opportunity both to learn how to use the GPT and Anthropic APIs and to tune their own custom models. They will also learn about rigorous evaluation methods. I have worked extensively with undergraduate research assistants over the past several years. I have a variety of resource materials and hands on tutorials to guide them in learning how to perform these analyses, even if they lack prior experience with deep learning frameworks. I will also add comparisons across different generative AI tools (currently, we have used GPT 3.5 and GPT 4). If the student has the appropriate skills and interest, they can also be involved with maintaining our packages, which allow social science users to easily deploy our bespoke models, Hugging Face models, or OpenAI models using just a few lines of code. Finally, if the student(s) have appropriate statistical background, we would also be happy to have them be involved in the statistical work we are developing to create optimal sampling strategies for evaluating generative AI performance. The end objective is both to expose the students to rigorous research in deep learning and to provide resources that will guide economists and social scientists more generally on using AI in their work.

Opportunity for the Fellow:

The students will learn how to use the OpenAI and Anthropic APIs, how to tune their own custom language models, how to use our GPUs, and how to rigorously evaluate the performance of deep learning models using carefully constructed evaluation data. I have been working with undergraduates on related tasks for the past several years, through programs such as the economics department's SUPER program and the BLISS program. Hence, I have developed a variety of resources, notebooks, and packages to guide students in successfully acquiring these skills, even when they lack prior experience with deep learning. During the summer, I meet daily with my research assistants, so that they receive daily hands-on guidance. Often, students have elected to continue working with my group (contingent on available resources), following the duration of the program. For students with an interest in longer-term involvement, co-authorship opportunities are available, and I have a variety of top machine learning publications jointly authored with undergraduates. Upon completing the summer program, students will be able to confidently use generative AI in their own work. In several cases, I have gone on to supervise senior theses or independent research projects for students who have previously worked as research assistants. It is particularly rewarding for me to see students using these methods in their own work.

Outcomes:

The end result is a high impact publication in the Journal of Economic Literature comparing the performance of Generative AI to fine-tuned models for a diversity of social science tasks, across a variety of different languages. Some results may also be expanded on in publications for Machine Learning proceedings. All analyses will be easily accessible to users through packages and accompanying tutorial notebooks on the website, EconDL, that will accompany the article. Rigorous methods will also be used to create samples for evaluating the performance of Generative AI. The student will learn both how to use generative AI through APIs and how to tune and evaluate their own custom models on our GPUs. By the end of the summer, they will be able to confidently deploy generative AI in their own work.

Keywords:

GenAI, fine-tuning, social science analyses

Gen AI 2024 - Elani

Project Supervisor:

Hawazin Elani, Assistant Professor in Oral Health Policy and Epidemiology (HSDM)

Project Title:

Promoting Algorithmic Fairness in Oral Health Research

Project Overview:

Health disparities, especially in oral health, are well-documented, with socioeconomic, demographic, and geographic factors contributing to existing inequalities. Access to oral healthcare remains challenging for low-income, racial, and ethnic groups. Algorithmic bias in machine learning (ML) models poses a significant concern, potentially exacerbating these disparities. Despite this, there is a notable lack of awareness and expertise in addressing algorithmic fairness within the oral health research community. This project aims to address these challenges by assisting researchers and users in identifying and mitigating algorithmic biases in oral health research.

We seek to achieve this through three primary objectives: Training a Customized Large Language Model (LLM): We plan to develop a large language model customized for algorithmic fairness in oral health research. This model will be based on a Generative Pre-Trained Transformer (GPT-4) version and incorporate fairness knowledge to ensure it does not perpetuate biases. By utilizing Python libraries such as FairLearn, AI Fairness 360, and FairML3, we will assess bias during the training process. Developing an Interactive Engagement Engine: Our project will include developing an interactive engagement engine to raise awareness and enhance users' knowledge of algorithmic fairness. This platform will offer real-time feedback to users, guiding them in understanding and addressing bias concerns throughout the research process. Providing Users with Personalized Decision Support: We intend to provide researchers with a personalized decision support platform equipped with curated data resources to facilitate the detection and mitigation of biases in ML models. By offering access to diverse datasets, including oral health outcomes data from public data sources, researchers will be able to assess the impact of their models on various populations.

Our approach involves a comprehensive research strategy: The platform will be user-friendly and openly accessible, with an interactive interface powered by the customized LLM. Supervised by experts in AI development, fairness evaluation, and oral health inequities, the training process will ensure that the model remains unbiased. Users will have access to the platform throughout different stages of their projects, from research design to data analysis and reporting. The platform will guide users in selecting datasets, prioritizing fairness metrics, and identifying biases in their models. Tailored recommendations will be provided to improve fairness in ML models. The platform will generate detailed reports showcasing the approaches taken and their impact on ML models. These reports will promote transparency and accountability in research practices. Our project will incorporate fairness quantification and interpretation into an interactive platform, enabling robust bias detection in AI. It provides guidelines for fair algorithm development and evaluation, facilitating systematic auditing of models for fairness. It offers curated datasets with diverse populations, allowing researchers and users to gain hands-on experience with fairness metrics and mitigation strategies. By empowering researchers with customizable codes, guidelines, and data resources, we aim to contribute to equity in oral health research and enhance the robustness of findings in this field.

Opportunity for the Fellow:

Our project presents an exciting avenue for students to engage with cutting-edge technologies, gain hands-on experience in machine learning (ML) and artificial intelligence (AI), and directly address health disparities. Health disparities, particularly in oral health, represent a pressing issue affecting diverse populations worldwide. Socioeconomic, demographic, and geographic factors contribute to these inequalities, with marginalized communities often bearing a disproportionate burden of poor oral health outcomes. At the same time, the emergence of algorithmic bias in ML models poses a significant challenge, potentially exacerbating existing disparities if left unaddressed.

As a student involved in this project, you will have the opportunity to: Gain Specialized Skills: You will receive training in ML techniques, including customizing large language models (LLMs) for algorithmic fairness in oral health research. This hands-on experience will equip you with valuable technical skills and knowledge applicable across various domains. Contribute to Cutting-Edge Research: By actively participating in developing an interactive engagement engine and decision support platform, you will contribute to innovative research at the intersection of healthcare disparities and AI. Your contributions will directly impact the advancement of inclusive practices within dental research. Make a Real-World Impact: Through your involvement in detecting and mitigating algorithmic biases in ML models, you will play a crucial role in promoting equity and fairness in oral health research. Your efforts will contribute to improving healthcare outcomes for underserved populations and addressing systemic inequalities. Collaborate with Experts: You will have the opportunity to collaborate closely with experts in data science, oral health, AI development, and fairness evaluation. This collaborative environment will foster mentorship, professional growth, and interdisciplinary learning. Access Cutting-Edge Resources: As a team member, you will have access to state-of-the-art resources, including high-performance computing resources and advanced neural networks such as ChatGPT-4. These resources will support your research endeavors and enable you to explore innovative approaches to addressing algorithmic bias.

Outcomes:

Skill Development: students will enhance their data science, machine learning, and artificial intelligence skills. They will gain hands-on experience developing ML models, assessing algorithmic bias, and implementing fairness strategies in healthcare research. Interdisciplinary Learning: Students will have the opportunity to collaborate with experts from diverse backgrounds, including data science, oral health, AI development, and fairness evaluation. Contribution to Research: Student participants will contribute directly to cutting-edge research at the intersection of healthcare disparities and AI. They will play a vital role in addressing algorithmic bias in oral health research, making meaningful contributions to advancing equitable practices in healthcare. Impact on Healthcare: By actively participating in the detection and mitigation of algorithmic biases in ML models, students will contribute to promoting fairness, transparency, and ethical practices in healthcare research. Career Opportunities: Engaging in research at the intersection of AI and healthcare disparities will open doors to exciting career opportunities in academia, industry, and the public sector. Students will develop valuable skills and knowledge that are highly sought after in the rapidly growing field of health informatics. Conference and poster Presentations: Students can present their research findings at academic conferences, symposiums, or workshops.

Keywords:

Dental, oral health, algorithmic fairness, chatbot, GenAI, LLM

Gen AI 2024 - Gaudet

Project Supervisor:

Rachelle Gaudet, Professor of Molecular and Cellular Biology (FAS)

Project Title:

Exploring and benchmarking AI tools to predict protein conformational ensembles and the effect of mutations

Project Overview:

The Gaudet lab is interested in uncovering sequence-function relationships of membrane proteins such as metal ion transporters and ion channels. We combine experimental and computational approaches to determine structures of membrane proteins and study how point mutations in their sequences affect their functions. Transformer-based deep learning models such as Deepmind's Alphafold2 have revolutionized protein structure prediction but are trained to predict a single, lowest-energy structure for each protein rather than an entire conformational ensemble and are poor predictors of the effects of point mutations on stability (Pak et al., 2023). Nonetheless, several research groups have demonstrated how changing the diversity of multiple sequence alignments used as inputs to the AlphaFold2 prediction algorithm can help generate an ensemble of structural models of the same protein, and in some cases even help predict how point mutations will change the protein's dominant conformation (del Alamo et al., 2022; Wayment-Steele et al., 2024; da Silva et al., 2024). Novel algorithms have also been recently put forward with the explicit goal of predicting ensembles of structures (Jing et al., 2023; Jing et al., 2024), although these models are not yet well benchmarked. In this project we aim to apply this new class of models to transporter proteins by exploring whether these modifications to AlphaFold2 allow us to predict which specific amino acid point mutations will lock transporters. This will both help develop a tool for better understanding and engineering transporters, as well as shine light onto whether modern deep learning-based structure prediction algorithms learn any fundamental principles of protein energetics. Citations: Pak, M. A., Markhieva, K. A., Novikova, M. S., Petrov, D. S., Vorobyev, I. S., Maksimova, E. S., ... & Ivankov, D. N. (2023). Using AlphaFold to predict the impact of single mutations on protein stability and function. Plos one , 18 (3), e0282689. Del Alamo, D., Sala, D., Mchaourab, H. S., & Meiler, J. (2022). Sampling alternative conformational states of transporters and receptors with AlphaFold2. Elife , 11 , e75751. Wayment-Steele, H. K., Ojoawo, A., Otten, R., Apitz, J. M., Pitsawong, W., H&ampouml;mberger, M., ... & Kern, D. (2024). Predicting multiple conformations via sequence clustering and AlphaFold2. Nature , 625 (7996), 832-839. Monteiro da Silva, G., Cui, J. Y., Dalgarno, D. C., Lisi, G. P., & Rubenstein, B. M. (2024). High-throughput prediction of protein conformational distributions with subsampled AlphaFold2. Nature communications , 15 (1), 2464. Jing, B., Erives, E., Pao-Huang, P., Corso, G., Berger, B., & Jaakkola, T. (2023). Eigenfold: Generative protein structure prediction with diffusion models. arXiv preprint arXiv:2304.02198 . Jing, B., Berger, B., & Jaakkola, T. (2024). AlphaFold Meets Flow Matching for Generating Protein Ensembles. arXiv preprint arXiv:2402.04845 .

Opportunity for the Fellow:

Students will learn basic principles of structural biology, structure-function relationships of transporters and learn how to use AlphaFold2 to generate conformational ensembles of proteins.

Outcomes:

We hope to determine potential point mutations that modify functional properties of a metal ion transporter in our lab and verify them using experimental techniques. The student will also have an opportunity to submit an abstract and present their work at a machine learning conference workshop such as Machine Learning in Structural Biology.

Keywords:

AlphaFold, proteins, structural biology, mutations, bioinformatics, computational biology

Gen AI 2024 - Henrich

Project Supervisor:

Joseph Henrich, Ruth Moore Professor of Human Evolutionary Biology (FAS)

Project Title:

LLMs in the Collective Brain

Project Overview:

Innovation is often assumed to be driven by the talented few – the lone geniuses – but research has suggested that innovations in reality emerge from our species' cultural learning abilities applied within networks of individual humans. In this sense, our societal & social networks act as collective brains. In our prior work on "Innovation in the collective brain", we lay out a theoretical framework for thinking about the dynamics of the collective brain – specifically how the key factors of sociality, transmission fidelity, and cultural variance drive the mechanisms of innovation through facilitating serendipity, recombination, and incremental improvement.

Recent advancements in generative AI have enabled simulations of human personae, as well as multi-agent simulations of interactions between personae. While such simulations do not accurately represent all aspects of human behavior, these simulations can act as human proxies in certain contexts, often with human revision or supervision. In comparison to conventional computational methods used in cultural evolution research, LLMs have unique affordances to simulate the effect of transformations of semantic information induced by cognitive mechanisms and psychological profiles.

Combining our theoretical foundation in evolutionary biology with new capabilities in generative AI, this project investigates the dynamics of LLM and LLM + human collective brains. Specifically, we will develop software and experiments toward two frontiers of research:

1. Phase I: LLM-only collective brains – collective brains composed of LLM personae can help us investigate how varying properties of the design – such as network structures, interconnectivity or "sociality", rules for transmission and mutation of information, and the range and diversity, aka "cultural variance", of personae psychological profiles – affects innovation rates, creativity and diversity of ideas, and idea evolution dynamics.

2. Phase II: LLM + human collective brains – LLMs brought "into the culture loop" creates new human-AI collective brain dynamics that can be investigated to anticipate how varying designs lead to different paths of cultural change. Akin to recommender algorithms altering social dynamics in unanticipated ways, understanding how the structures of the human-AI collective brain might alter social dynamics will be crucial to informing wise design decisions going forward.

The Summer 2024 phase of the project focuses on Phase I, including 1) developing simulation software for LLM collective brains, and 2) investigating the effects of varying design parameters on innovation outcomes. The simulation software will allow for varying network structures, interconnectivity, transmission rules, and personae design. Through systematically altering these design parameters corresponding to concepts in our framework of sociality, transmission fidelity, and cultural variance, we can observe the patterns of idea evolution, emergence of novel concepts, and overall creativity of the collective LLM brain in experiments on innovation tasks.

These insights can help us 1) generate hypotheses about the dynamics of human collective brains, which can later be tested in human transmission chain experiments, 2) identify new ways to design human networks for increasing innovation, and 3) understand the dynamics of LLM collective brains, which helps inform researchers developing multi-agent AI systems across domains.

The second phase of the project will build on the software and experimental pipelines developed during the Summer 2024 phase, expanding to investigate LLM + AI collective brains.

Opportunity for the Fellow:

This project offers an opportunity for student researchers interested in the intersection of artificial intelligence and cultural evolution. Students have the opportunity to apply their skills in software development, experimental design & execution, and data analysis to contribute to investigating the future of our collective brain in the age of AI.

Some key benefits for students include:

• Exposure to interdisciplinary research, drawing from diverse methods across evolutionary biology, human-computer interaction, collective intelligence, and more.

• Honing experimental research skills, including design, execution, & data analysis

• Publication and presentation opportunities

Outcomes:

Anticipated outcome 1. Development of simulation software for LLM collective brains – We will develop a simulation platform where LLM personae can interact within complex network structures. Akin to Perez et. al, will include features for adjusting network density, connectivity patterns, rules / prompts for information transmission and mutation, and ability to design personae through prompting or fine-tuning. Such software can be reused by other research groups across fields.

Anticipated outcome 2. Design of transmission chain experiment with different tasks – To investigate the effects of varying design parameters on innovation outcomes, we will design experiments based on the classic transmission chain model, combined with innovation and creativity tasks, such as the classic alternative uses test.

Anticipated outcome 3. Analysis of experimental LLM collective brain dynamics – We will apply both our theoretical framework and other frameworks within evolutionary biology and collective intelligence to formulate theories based on our findings of the dynamics of LLM collective brains.

More broadly, the proposed project integrates theoretical insights from evolutionary biology with emerging capabilities in generative AI – we anticipate this approach will yield contributions across domains, including:

Deepened understanding of human cultural dynamics – insights can be extrapolated to better understand human social and cultural processes, particularly how ideas evolve and proliferate within networks, and how the types of personalities / psychological profiles, their interconnections, and transmission processes affect innovation dynamics.

Advancements in multi-agent / swarm AI & collective intelligence research – along with producing software tooling for designing and operating collective brains, insights may help inform research on properties, dynamics, & capabilities of multi-agent / swarm AI systems, as well as collective intelligence more broadly.

Insights for improving real-world network designs – insights could be used to inform design of more effective social networks or digital platforms, optimizing for healthier information exchange and enhanced innovation.

Keywords:

LLMs, creativity, cultural evolution, human-ai co-evolution, collective brain, collective intelligence

Gen AI 2024 - Holland

Project Supervisor:

Alisha Holland, Professor of Government (FAS)

Project Title:

The Detection and Prevention of Corruption in Public Procurement

Project Overview:

How can generative AI models help to identify and prevent corruption? Governments spend a staggered $10 trillion in contracts for public goods and services. Public procurement is widely recognized as a major source of corruption, particularly in developing countries, where the annual cost of corruption is estimated to cost 5% of GDP. Advances in AI point to several ways that governments can improve their ability to detect corruption in public procurement. First, graph technologies can make sense of relationship-based risk in public procurement data. Most governments sanction businesses for poor performance in past contracts. Yet a key challenge is that sanctions fall on a business entity. The same individuals that run a company can create new businesses or subsidiaries that then escape government sanctions. A relationship-based approach uses richer data, such as the addresses, bank accounts, and names associated with a firm, to understand the relationships between companies. A second potential use of generative AI in the public procurement space is to identify suspect contracts and improve "red flag" systems. These can be developed in several directions. One approach is to look at the timing of contract issue, with particular concern for contracts that are issued near election and that go to contractors that have made campaign donations in the past. Perhaps open AI models can improve the efficiency of identifying these electorally sensitive contracts, particularly to international donors that might be unaware of the campaign finance dynamics behind projects. A second issue concerns the inclusion of terms within contracts that suggest auctions that are being directed to particular firms. Many governments have tried to move towards "model contracts" that standardize the terms for auctions. However, these contracts are hard to apply in sectors like infrastructure that require modifications for different geographic conditions and project idiosyncrasies. It's possible that generative AI tools can be trained on existing contracts to identify and distinguish terms that reflect genuine project requirements from those that may be included to narrow competition and direct contracts to favored bidders. Finally, there are important questions about the politics behind the use of generative AI in preventing corruption. Which countries are investing in AI tools and willing to adopt them? How can the risks flagged by AI model be communicated to government authorities and made consistent with legal requirements to establish probable cause to suspend a contract process?

Opportunity for the Fellow:

Microsoft Research has developed early tools purely based on graph theory to identify relationship-based risks in procurement. They are about to release a new open-source software platform based on generative AI to improve these measures. I have contact with the Microsoft Research team (led by Darren Edge) responsible for the development of the open-source tool. The hope is that a research assistant this summer would work with the open-source code to apply it to procurement data in a range of countries that maintain transparent platforms of their public contracts (ideally starting in Latin America where most of my work is based, such as Brazil, Chile, and Colombia). A research assistant may also work to develop partnerships with international organizations like Open Contracting to collaborate on tools to identify contract risks and learn more about how these organizations are starting to incorporate AI tools. Finally, a research assistsant would engage in more qualitative research studying which governments are using big data and AI models to flag corruption in their public procurement and what the political barriers may be to apply these tools.

Outcomes:

The aim of this project is to adapt existing open-source tools based on open AI to the public procurement context. This tool will help to identify the underlying structure of corruption and lead to the publication of academic articles, as well as partnerships with anti-corruption authorities and international organizations working to improve transparency in public procurement.

Keywords:

corruption; public procurement; AI for the public good

Gen AI 2024 - Janapa Reddi

Project Supervisor:

Vijay Japana Reddi, John L. Loeb Associate Professor of Engineering and Applied Sciences (FAS)

Project Title:

GenAI for Chip Design

Project Overview:

We are seeking two undergraduate researchers to join our team on the GenAISys project. This exciting initiative that is part of my group, titled "GenAISys: Generative AI-Assisted Microprocessor System Design," investigates the potential of generative AI technologies to revolutionize digital chip design, particularly for microprocessors. The project explores how large language models like ChatGPT and advanced machine learning methods can simplify the design process, streamline workflows, and deepen our understanding of intricate hardware systems. GenAISys aims to significantly shorten design times and unlock innovative chip designs by automating and streamlining various tasks like layout optimization and HDL code generation. This could lead to the creation of microprocessors with lower power consumption, enhanced performance, and increased functionality, paving the way for more robust and efficient computing systems across various fields.

The ten-week program focuses on two key areas: (1) Dataset Design and Curation for Chip Design: A crucial aspect involves creating and organizing a comprehensive dataset for microprocessor chip design. This dataset is essential for building advanced AI models that can support various stages of the design process, such as understanding, creating, and enhancing microprocessor chip designs. It will encompass diverse data types like circuit diagrams, snippets of HDL code, design records, performance indicators, and simulation results. (2) Foundation Model Training and Evaluation for Hardware Design: We will also focus on creating and evaluating machine learning models specifically designed for hardware design, focusing on microprocessors.

This emerging field merges chip design principles with the power of AI and machine learning. We will investigate, test, and assess the utility of publicly available generative models that could support the hardware design process. These models will be customized using the carefully curated dataset from the project's initial phase and other relevant data sources. The trained models will then be evaluated across various microprocessor chip design tasks encompassing functions like design space exploration and optimization, HDL code generation and synthesis, performance prediction and analysis, and verification and validation of chip designs. The assessment will compare the AI models' performance against conventional design methods and tools. Key metrics like design quality, efficiency, accuracy, and scalability will be evaluated to gauge the effectiveness of employing AI in the design process. Furthermore, the study will delve into the comprehensibility and transparency of the trained models, ensuring the decision-making mechanisms of the AI system are clear and understandable to chip designers. This transparency is crucial for building trust and fostering the acceptance of AI-supported design tools within the industry. Similar to the initial phase, the team will collaborate with academic researchers, industry partners, and chip design specialists to ensure the relevance and applicability of the proposed solutions. Research Mentoring and Support: The GenAISys project offers a unique opportunity for undergraduate researchers to benefit from a comprehensive mentoring program.

Throughout the ten-week program, you will be guided by a dedicated team of myself, my graduate students, and potential industry experts. This collaborative approach ensures you receive ongoing support and mentorship. We will be readily available to answer your questions, provide constructive feedback on your work, and help you navigate the various stages of the project. This ensures a valuable learning experience where you can develop research skills while contributing meaningfully to the GenAISys initiative. Research Significance: The GenAISys initiative, by harnessing the power of generative AI and machine learning, strives to push the boundaries of digital chip design, leading to the emergence of more effective, intelligent, and groundbreaking design methodologies. The project's results have the potential to profoundly impact the industry by facilitating faster time-to-market, reducing design costs, and enhancing chip efficiency. Consequently, it presents a unique prospect. Advancements in AI-supported chip design have the potential to pave the way for the creation of enhanced and more efficient computing systems in diverse sectors, accelerating scientific breakthroughs, promoting the development of eco-friendly technologies, and fostering economic growth through technological advancements.

Opportunity for the Fellow:

Work with State-of-the-Art Generative AI: The GenAISys project offers our undergraduate students an unparalleled opportunity to work directly with state-of-the-art generative AI models, such as GPT-3 or similar large language models, gaining practical experience in their application to hardware design. Students will learn how to fine-tune and adapt these models to the specific domain of microprocessor chip design, developing valuable skills in AI model customization and optimization. This hands-on experience will deepen their understanding of AI's potential to revolutionize the chip design process.

Master the Intricacies of Chip Design: Students (from CS141) will learn to explore the complex world of microprocessor chip design, gaining insights into the various stages involved, such as architecture definition, RTL design, verification, and physical layout. Through interactions with experienced chip designers from my lab and industry experts, the undergraduate students will receive valuable knowledge and mentorship, allowing them to develop a comprehensive understanding of the chip design lifecycle. This exposure will equip them with the skills and knowledge necessary to tackle the challenges of modern chip design. This will set them up strongly for industry internships and interviews.

Contribute to the Future of AI in Chip Design: As active participants in the GenAISys project, students will have the unique opportunity to contribute to curating the microprocessor chip design dataset and developing generative AI models. They will learn about data collection, preprocessing, and annotation techniques, gaining valuable experience in dataset curation. By assisting in the development and evaluation of AI models, these students will play a crucial role in creating innovative tools that have the potential to revolutionize the chip design process.

Foster Interdisciplinary Collaboration: The GenAISys project brings together a diverse team of researchers and professionals from various fields, including computer science, electrical engineering, and machine learning. Thus, our students will have the chance to work alongside this interdisciplinary team, fostering collaboration and exposing them to different perspectives and approaches to problem-solving. This collaborative environment will provide our students with valuable networking opportunities, enabling them to build connections with experts in the field and lay the foundation for future collaborations.

Conduct Groundbreaking Research & Gain Recognition: Students will be at the forefront of cutting-edge research in AI-assisted hardware design, gaining hands-on experience in conducting groundbreaking research. Their contributions to the GenAISys project may lead to co-authorship on research papers or presentations at top conferences, providing them with valuable academic and professional recognition. This research experience will enhance their skills and showcase their ability to contribute significantly to the field.

Develop Highly Sought-After Skills: Undergraduate students will develop essential data analysis, interpretation, and visualization skills working with large datasets and complex models. They will learn to use various tools and techniques to extract meaningful insights from the data and effectively communicate their findings. They will learn these from the graduate students. These skills are highly sought-after in academic and industry settings, making students valuable assets in their future endeavors.

Gain Industry-Relevant Skills: The GenAISys project allows students to work with industry-standard tools and methodologies used in chip design, such as EDA (Electronic Design Automation) tools, HDL (Hardware Description Language) simulators, and verification frameworks. This exposure will equip students with practical skills and knowledge that are highly relevant to the semiconductor industry, making them well-prepared for future industry roles or advanced research positions.

Opening Doors to Future Opportunities: Participating in the GenAISys project will open up a wide range of future opportunities for students, whether they pursue graduate studies in AI-assisted hardware design or seek industry positions in the semiconductor or AI domains. The skills, experience, and network gained during the project will make them highly competitive candidates for further research or professional roles, setting them toward a successful career in this cutting-edge field.

Outcomes:

Research: Advancing AI-Assisted Chip Design The GenAISys project aims for significant research advancements in AI-supported hardware design. We anticipate expanding the capabilities of generative AI models for microprocessor chip design, demonstrating AI's ability to streamline and enhance various design stages. GenAISys will establish new benchmarks for efficiency, functionality, and effectiveness in chip design processes by introducing cutting-edge AI methods and exploring innovative chip architectures. The project's findings will be disseminated through publications in respected journals and conferences, solidifying its influence and intellectual leadership within the field.

Mentoring: Empowering Students The GenAISys project prioritizes student growth and development. Selected students will have the unique opportunity to work on a cutting-edge research project, gaining hands-on experience in AI and hardware design. Through close mentorship and collaboration with experienced researchers, students will acquire valuable skills in data curation, model development, and evaluation, preparing them for successful careers in research or industry. The project fosters a supportive learning environment that encourages creativity, critical thinking, and problem-solving, enabling students to thrive and make meaningful contributions to the field. Students will enhance their technical expertise and develop important soft skills like teamwork, communication, and project management through participation in GenAISys. Additionally, hands-on involvement in cutting-edge research will provide students with a deep understanding of the latest advancements in AI and hardware design. They will learn to apply their knowledge to real-world challenges and develop innovative solutions.

Future Researchers: Igniting Passion for Academia A key outcome of GenAISys is to inspire and motivate participating students to pursue academic research in AI-assisted chip design. With a proven track record of guiding undergraduates toward successful Ph.D. programs at top universities, I am passionate about nurturing the next generation of researchers. By engaging students in a transformative research experience and exposing them to the exciting possibilities of AI-assisted chip design, we aim to ignite a lasting passion for academic research. The GenAISys project will offer students a glimpse into the rewarding nature of scientific discovery and the satisfaction of pushing the boundaries of knowledge in their field.

Outreach: Contributing to Open Science The GenAISys initiative champions open science and actively participates in the open-source community. The project is committed to open-source principles, making the AI models, datasets, and research code publicly accessible. By sharing these resources, GenAISys encourages collaboration and reproducibility and accelerates advancements in AI-assisted chip design. Open-source contributions empower researchers and professionals worldwide to leverage the project's findings, customize the models for their specific needs, and foster further developments in the field. Additionally, GenAISys will provide comprehensive documentation and tutorials to facilitate the use and expansion of the open-sourced materials, ensuring they are readily accessible and adaptable to a broader audience.

Keywords:

Generative AI, Microprocessor Design, Hardware Acceleration, Dataset Curation, Interdisciplinary Research, Open-Source Innovation

Gen AI 2024 - Koumoutsakos

Project Supervisor:

Petros Koumoutsakos, Herbert S. Winokur, Jr. Professor of Computing in Science and Engineering (SEAS)

Project Title:

Generative AI for modeling and Optimization of Complex Systems

Project Overview:

Recent advances in generative AI have led to promising results for both image and video generation. In this project we aim to extend, adapt and develop genrative AI techniques to revolutionize modeling of complex multiscale systems. The proposed Generative Multiscale Models (GeMMs) will facilitate the reliable prediction of critical phenomena, such as weather and epidemics, the development of new materials and aerodynamic vehicles that depend on the efficiency and veracity of numerical simulations. In particular, we are planning to identify a coarse-grained representation and coarse-grained dynamics for the systems of interest and then reconstruct the high-dimensional system of interest using generative diffusion model. We will extend our state of the art work on Learning Effective Dynamics (LED) and its recent extension to Generative LED. The latter has already produced remarkable results for turbulent flows but it is domain specific and cannot be applied to molecular dynamics or other parameterized dynamical systems. A key challenge is the incorporation of physical information such as rotation and translation invariance as well as of the parameter values into the diffusion model. We propose to solve this problem by interpreting constraints and parameters as virtual observables and include them in the diffusion process using Bayes Law. This would allow us to combine the flexibility of diffusion models with physical constraints and enable novel modeling and simulation techniques for challenging applications such as molecular reactions during catalysis or control of turbulent flows.

Opportunity for the Fellow:

Engage in Generative AI research for Computational Science.

Outcomes:

Generative State of the Art Simulations of Turbulent Flows A Publication

Keywords:

Generative AI, Computational Science, Fluid Mechanics

Gen AI 2024 - Kozinsky

Project Supervisor:

Boris Kozinsky, Gordon McKay Professor of Materials Science and Mechanical Engineering (SEAS)

Project Title:

Generative AI For Learning Transferable Coarse Grained Representations of Atomic Systems

Project Overview:

The Materials Intelligence Research Group, led by Professor Boris Kozinsky, is looking for undergraduate researchers keen on pushing the boundaries of molecular simulation. Over the course of 10 weeks, we are looking for one student to contribute to ongoing efforts in the group towards enabling more robust, transferable, large scale simulations. Motivation Atomistic simulation has enabled new fundamental insights into material behaviour. In particular, machine learning force fields (MLFFs) have enabled molecular dynamics (MD) simulations to be performed with near quantum accuracy, thereby significantly improving the accuracy affordable by modern techniques. However, the representation of all degrees of freedom (DOFs) of a system can become computationally prohibitive when system sizes reach the order of millions of atoms, or when timescales of interest exceed 100s of nanoseconds. Coarse graining (CG) methods that aim to reduce the number of DOFs in the system into larger "beads" are a popular method for overcoming these challenges. By removing fast DOFs, larger MD timesteps may be used, and fewer force calculations are needed. This can lead to orders of magnitude gains in efficiency compared to fully atomistic methods. Despite the appeal, the significant complexity of the most accurate CG models has inspired recent work in integrating ML methods into the design of these potentials. Moreover, our group has recently demonstrated a new, state-of-the-art CG technique that utilises uncertainty-driven active learning that has begun to enable autonomous learning of CG models.

While promising, a significant hurdle remains in the design of flexible frameworks for moving between the atomistic and CG scales, which is necessary during the active learning loop. The autonomous collection of new data requires an infrastructure for going between these two resolutions that is to date primarily hard coded. Such techniques, often referred to as reconstruction or backmapping, would greatly expand the use cases of current CG methods. Early work in the community has aimed at addressing some of these challenges. Of particular note are those backmapping techniques that leverage new developments in machine learning, including equivariance. Similarly, generative AI has emerged as a leading candidate for infrastructures in performing reconstruction. As such, this research project is aimed at exploring these avenues, advancing our understanding of the possibilities and limitations of existing and new generative AI models, as well as developing a path towards integrating these technologies into state of the art active learning routines. Finally, improvements in these directions will allow us to demonstrate much larger reach in the capabilities of our CG models.

Methods and technical details: We present two possible pathways for the reconstruction of all-atom configurations from coarse-grained configurations through a generative AI model that would constitute the core of the research project: Integrating an offline GenAI model in an online learning CG MLFF procedure. Very recent efforts have displayed good effectiveness in backmapping CG DOFs to all-atom configurations in biomolecular systems, showing also promising transferability properties. Adopting a pre-trained generative AI model in the reconstruction step of our active learning pipeline will enhance transferability of the approach, getting around the ad-hoc-defined backmapping step. Online learning or fine-tuning of a generative AI model: The active CG MLFF framework currently developed in our group produces a lot of configurations as a side product of the constrained dynamics. These could be employed to train an online generative AI model, by updating or fine-tuning it with newly generated configurations. The literature about online training of generative models is very recent, but encouraging results have been obtained in this direction, e.g. by coupling reinforcement learning with the training of diffusion models. The possibility of taking the second, fully-online learning, path will also depend on the reliability and feasibility of implementing this framework in our case. Indeed, the assessment of the best strategy to be adopted will constitute an interesting and instructive part of the project.

Possible timeline for the project weeks 1-2: learning the relevant background in coarse graining methods. If the student has no prior experience, this time will also be used for them to get up to speed on principles of generative AI. weeks 3-4: literature review on a selection of key results in generative AI that could point to relevant architectures for our problem weeks 5-7: software implementation of the selected generative AI protocol, with initial tests performed on molecular systems weeks 8-9: exploration of key research objectives outlined in the Anticipated outcomes section week 10: wrap up, presentation of results to the Kozinsky group, and further tests of key objectives.

Opportunity for the Fellow:

In this position, the student will have the opportunity to explore novel generative AI approaches for learning how to represent atomistic systems in low dimensional spaces, as well as how to recover the removed DOFs in a generative fashion. The student will have opportunities in terms of: Academic growth: The internship experience aligns closely with students' academic goals by providing them with practical experiences that complement their coursework. Through the hands-on approach typical of research, the student will have the opportunity to deepen their understanding of molecular simulations and artificial intelligence and gain insights that cannot be obtained solely through academic study. Mentoring: this summer internship offers the unique opportunity to learn directly from two postdoctoral researchers, who will be mentoring closely and will be available for frequent discussion. This experience will help the student grow as a researcher in a guided manner. Research experience: during the internship, the student will delve into a real-world scientific research processes. They will engage with well-defined research questions set within an open-ended problem space: there is no pre-established solution to the problem, but rather the necessity to draw upon existing literature to select and integrate techniques in a coherent manner. The student will be encouraged to contribute their unique expertise in a peer-to-peer manner, enriching the expertise of the research group with theirs. The convergence of molecular simulations and generative AI not only presents an intellectually stimulating challenge but also aligns with current trends, positioning students at the forefront of cutting-edge research endeavors. Professional development: the student will have the chance to develop competitive and timely skills in a job market in which generative AI is seeing an enormous interest with a broad spectrum of application; moreover, in specific conjunction with molecular simulations, generative AI is growingly becoming a hot topic e.g. in the fields of materials research, molecular biology and drug discovery. Given significant progress is made, the student may also have the opportunity to develop their presentation skills by presenting their work at leading physics conferences. Networking: in order to address the challenges emerging in the research project, the student will be included in discussions with other members of the Kozinsky group and, if appropriate, with other collaborators from Harvard, MIT, or industry. These contacts could enhance future opportunities of collaboration and employment for the student. Further, participation in conference settings later on would also serve to expand the students network.

Outcomes:

The objectives of this research opportunity are aimed at outcomes in two categories. The student's development as a researcher and computationalist is anticipated to grow, while their contributions to the research community may lead to multiple insightful outcomes. Student Development Outcomes: &ampbull; New or improved knowledge on molecular simulation techniques &ampbull; New or improved knowledge on the use of machine learning for the design of interatomic potentials Research Development Outcomes &ampbull; Identification of appropriate generative AI architectures for the task of forward and reverse atomistic to CG mappings &ampbull; Understanding of the limiting behavior of these generative AI architectures in different data regimes. Specifically, what learning strategies can be implemented when models have seen "sufficient" data that could enable more efficient integration into active learning methods &ampbull; Understanding of the limits of model performance in problems of transferability, i.e. when are models capable of extrapolating to new systems or chemistries.

Keywords:

Generative AI, Interatomic Potentials, Active Learning, Free Energy Methods, Coarse Grained Molecular Dynamics

Gen AI 2024 - Lewis

Project Supervisor:

Jennifer Lewis, Hansjörg Wyss Professor of Biologically Inspired Engineering (SEAS)

Project Title:

AI-based 3D Printing Materials Formulator

Project Overview:

Objective: In the realm of materials science, complex fluids and elastomers are pivotal in a wide array of applications, from everyday items like mayonnaise and rubber bands to industrial uses in concrete structures and engine gaskets. Despite their ubiquity, the development of these materials with targeted properties remains a formidable challenge due to the intricate interplay of multi-modal molecular interactions. Traditional methods of formulation often rely on exhaustive experimentation and trial-and-error approaches, frequently resulting in suboptimal solutions. The advent of generative AI presents a groundbreaking opportunity to leapfrog these limitations by harnessing deep learning to predict the behavior and properties of these materials based on their composition and structural constituents.

Opportunity for the Fellow:

Participants will: -- Lead or co-lead initiatives within the project’s key focus areas. -- Acquire hands-on experience in generative AI applications and the physics of complex materials. -- Gain the opportunity for authorship in a high-impact publication, contingent on significant contributions to the project.

Outcomes:

-- A validated deep learning model with dual predictive capabilities: property prediction from molecular combinations and vice versa. -- A robust, validated dataset from public and proprietary sources for training and benchmarking the AI model. -- A scholarly article delineating the developed model, its predictive accuracy, and directions for future enhancements.

Gen AI 2024 - Lu, Junwei

Project Supervisor:

Junwei Lu, Assistant Professor of Biostatistics (HSPH)

Project Title:

Generative Causal AI for Alzheimer’s Disease Therapeutic Management

Project Overview:

Our proposed research "Generative Causal AI for Alzheimer's Disease Therapeutic Management" targets the intricate challenges of Alzheimer's Disease (AD) through an innovative generative causal AI framework. We endeavor to bridge the existing gaps in AD research by integrating advanced generative AI with causal inference methods, enhancing the diversity and richness of multimodal clinical data. This novel methodology facilitates robust causal analysis, enabling the discovery of precise therapeutic pathways previously obscured by traditional research limitations. Structured across three phases: (1) we develop a multimodal generative model that synthesizes brain imaging and medical records, leveraging state-of-the-art models to overcome generative AI's hallucination issues; (2) we introduce a generative causal AI framework, employing refined weighting techniques to merge generative AD data with real-world observations, thereby uncovering novel AD treatment effects; (3) we utilize generative reinforcement learning to simulate AD therapy trajectories, dynamically optimizing treatment strategies for personalized patient care.

Opportunity for the Fellow:

This project invites undergraduate students at Harvard to engage in frontier research at the intersection of Generative AI and Alzheimer's Disease therapeutic management during the summer of 2024. Students selected for this project will contribute to the development of innovative Generative AI methodologies aimed at enhancing our understanding and management of Alzheimer's Disease. The research focuses on three major phases: developing a generative model for multimodal clinical data, creating a generative causal AI framework for clinical discovery, and optimizing AD therapy with generative reinforcement learning. Students will work alongside an interdisciplinary team of experts in machine learning, statistics, and clinical neurology, gaining hands-on experience in cutting-edge AI techniques, such as Schrödinger bridge diffusion models and Vision Transformer encoders, and their application in medical AI systems. This project not only offers a deep dive into advanced AI research but also provides a platform to contribute to meaningful advancements in medical science, particularly in improving the lives of those affected by Alzheimer's Disease.

Outcomes:

The anticipated outcomes for students involved in this pioneering research project on Alzheimer's Disease (AD) treatment optimization through generative AI are substantial and multi-dimensional. The project is structured into three meticulously designed phases, each with specific deliverables that the students will contribute towards. In Phase 1, students will help develop a multimodal generative model that integrates brain imaging with medical records using advanced AI techniques, the results of which will be shared with the research community through an interactive website. During Phase 2, the focus will shift to clinical discovery through a novel generative causal AI framework, culminating in the release of a comprehensive software package that encapsulates the framework and findings on AD therapeutic strategies. Finally, in Phase 3, students will be instrumental in optimizing AD therapy via generative reinforcement learning, contributing to another software package and co-authoring publications that detail the innovative AD treatment pipeline. This hands-on experience will not only enhance their skills in cutting-edge AI and medical research but also place them at the forefront of significant contributions to the field, potentially setting new standards in the treatment and management of Alzheimer's Disease.

Keywords:

Medical AI; Alzheimer's Disease; Multimodal Data; Causal Inference; Reinforcement Learning; Clinical Discovery

Gen AI 2024 - Lu, Huan-Tang

Project Supervisor:

Huan-Tang Lu, Lecturer in Human Development and Education (HGSE)

Project Title:

Social-Emotional Learning AI for Adolescents and Young Adults

Project Overview:

Project Overview: The rapid advancement of artificial intelligence offers unprecedented opportunities to address complex societal challenges. Our project, leveraging the capabilities of generative AI, aims to develop a pioneering tool designed to support the social and emotional development of adolescents and young adults. This tool will be particularly beneficial for individuals in need of support or experiencing crises, enabling interactions that foster a deeper understanding of self and promote mental well-being.

Context and Innovation: Our approach is centered around integrating generative AI with foundational counseling skills. The AI will be trained to engage in basic therapeutic interactions, guiding individuals through processes that enhance self-awareness and emotional understanding. This innovative use of AI as a therapeutic tool represents a significant step forward in mental health technology, where the focus is not only on crisis intervention but also on developing long-term coping strategies and resilience.

Development and Testing Phases: The project will commence with the initial development of the AI tool during the first two weeks, followed by two rigorous testing phases: First Round of Testing (Weeks 3-6): This phase involves collaboration with experts in social emotional development and mental health&ampmdash;scholars, practitioners, and students in training. These experts will role-play using designated case studies to test the AI tool's responsiveness and effectiveness. Subsequent interviews with these participants will gather insightful feedback, which will be instrumental in refining and fine-tuning the AI tool. Second Round of Testing (Weeks 7-10): We will engage 30 college students to use the tool, applying it to their personal life experiences. Their interaction with the AI will be followed by a structured survey designed by our team to capture their feedback and suggestions. This stage is crucial for assessing the tool's user experience and identifying any additional enhancements needed.

Training and Capabilities: To ensure the efficacy and sensitivity of the AI, we will employ a rigorous training regimen that encompasses both technical and ethical dimensions. The AI will be trained on a dataset curated to include diverse scenarios and responses that align with the Collaborative for Academic, Social, and Emotional Learning (CASEL) framework, which identifies five core competencies of social and emotional learning (SEL): self-awareness, self-management, social awareness, relationship skills, and responsible decision-making (CASEL, 2024). Moreover, to address the critical aspect of cultural competence, the training will also incorporate elements designed to recognize and respect various cultural backgrounds, ensuring that the AI's interactions are inclusive and anti-racist.

Cultural Sensitivity and Anti-Racism: In alignment with contemporary needs for culturally competent technologies, our project prioritizes an anti-racist orientation. The AI will be programmed to understand and reflect on cultural nuances, which is essential for engaging effectively with diverse youth populations. This aspect is crucial, as the effectiveness of SEL interventions is greatly enhanced when they are sensitive to the cultural dynamics that shape individuals' experiences.

Anticipated Outcomes and Expansion: The anticipated outcomes of this project are multifaceted: Development of a Scalable Tool: Creation of a generative AI tool that can be scaled and adapted for various educational and therapeutic contexts. Enhanced Access to Support: Providing adolescents and young adults with immediate, reliable access to emotional and social support in times of need. Empowerment Through Self-awareness: Facilitating growth in CASEL's five SEL competencies, empowering young individuals to navigate their social environments effectively. Cultural Competence in AI: Establishing a precedent for integrating cultural competence into AI development, particularly in applications dealing with mental health and education. Publications and Knowledge Dissemination: The project will produce scholarly articles and present findings at significant conferences, spreading knowledge and encouraging further research in AI-enhanced emotional learning. Long-term Impact and Tool Adoption: The tool's design will aim for adoption in educational and therapeutic settings, with ongoing assessments to refine its effectiveness and user engagement. Upon successful testing and refinement, we aim to distribute the tool to a larger participant pool, including middle schoolers, high schoolers, and young adults, to gather comprehensive data on its effectiveness and usefulness in promoting social and emotional development.

Conclusion: By integrating cutting-edge AI technology with the nuanced demands of social and emotional learning, this project aims to not only advance the field of AI but also contribute significantly to the well-being and development of young individuals across diverse communities. This aligns perfectly with the grant's goal to incorporate and grow Generative AI components across various research disciplines, making significant strides towards innovative, inclusive, and empathetic technological solutions.

Opportunity for the Fellow:

The summer research project presents a unique and comprehensive opportunity for Harvard College students to engage deeply with cutting-edge technology in Generative AI, while also acquiring critical skills in social and emotional learning (SEL), therapeutic communication, and research methodologies. Participation in this project will not only enhance their academic and professional trajectories but also contribute to their personal growth and development in meaningful ways.

Educational and Professional Development Learning and Application of SEL and Therapeutic Communication: Students will receive training in SEL and therapeutic communication, foundational elements in mental health and education disciplines. This knowledge is crucial for understanding how to support individuals in crisis and will be directly applied in developing and refining the AI tool.

Training the AI Tool: Participants will actively engage in the training phase of the AI tool. This involves programming the AI to process and respond to emotional cues, a skill that intersects technology with psychological insights. Research Skills: Students will be involved in mixed-methods research, including data collection and analysis. This will provide them with hands-on experience in both qualitative and quantitative research methodologies, enhancing their analytical and critical thinking skills.

Publication and Presentation Opportunities Scholarly Writing and Publication: Students will have the opportunity to contribute to scholarly articles about the project. This experience is invaluable for those considering academic careers, as it provides insight into the publication process and establishes their credentials in the field.

Professional Conferences and Workshops: The project will afford students the chance to present their work at professional conferences and workshops. This exposure is vital for networking with professionals and academics in the field, and it provides a platform to discuss their research with a broader audience.

Career and Personal Growth Continuation of Research: There will be opportunities for students to continue their research work with the team beyond the summer project. This ongoing engagement can lead to further development in their areas of interest and more profound mentorship relationships with faculty.

Creativity and Entrepreneurial Mindset: By working on an innovative project at the intersection of AI and mental health, students will be encouraged to think creatively and explore entrepreneurial aspects of technology and therapy. This mindset is crucial for leadership roles in technology and healthcare sectors.

Conclusion: The opportunity for students involved in this project is multifaceted, offering a blend of technical training, academic writing, professional development, and personal growth. This program not only prepares students for future careers in a variety of fields, including technology, psychology, and academia, but also empowers them to make a significant impact on society through innovative approaches to social and emotional challenges.

Outcomes:

The project aims to leverage generative AI to significantly enhance the social and emotional development of adolescents and young adults. By integrating advanced AI tools with evidence-based therapeutic and educational practices, the project expects to achieve several specific and measurable outcomes:

1. Development of a Scalable AI Tool: Objective: Successfully develop and refine a generative AI tool capable of conducting basic therapeutic interactions. Measure: Completion of a fully functional AI prototype that can demonstrate understanding and processing of emotional input across at least five different emotional scenarios by the end of the 10-week project period.

2. Enhanced Access to Support: Objective: Provide immediate, accessible emotional support to adolescents and young adults through the AI tool. Measure: Deploy the tool in a controlled environment with at least 100 users by the end of the first year following the project, assessing usability and accessibility through user feedback surveys with a satisfaction rate target of 85%.

3. Empowerment Through Self-awareness: Objective: Improve participants' competencies in self-awareness, self-management, relationship skills, social awareness, and responsible decision-making through interactions with the AI tool. Measure: Achieve an improvement in these competencies by at least 20% as measured by pre- and post-interaction assessments using standardized SEL scales.

4. Cultural Competence in AI: Objective: Ensure the AI tool incorporates and respects cultural diversity and promotes anti-racist practices in its interactions. Measure: Conduct cultural sensitivity training and reviews, with at least two rounds of feedback from diverse focus groups to refine the AI's responses. Post-deployment feedback should indicate at least 90% approval regarding the tool's cultural sensitivity.

5. Publications and Knowledge Dissemination: Objective: Disseminate findings and knowledge through scholarly articles and presentations at major conferences. Measure: Publish at least two peer-reviewed articles and present findings in at least three national or international conferences within 18 months following the conclusion of the project.

6. Long-term Impact and Tool Adoption: Objective: Facilitate the adoption of the AI tool in educational and therapeutic settings to support broader populations. Measure: Within two years post-project, have the AI tool adopted by at least 10 different educational or mental health institutions, with ongoing evaluations to monitor effectiveness and user satisfaction.

Conclusion: These anticipated outcomes not only fulfill the immediate goals of the project but also lay a foundation for sustained impact on the field of mental health and education technology. By setting specific, measurable objectives, the project ensures accountability and provides clear benchmarks for success. The integration of generative AI into social and emotional learning represents a forward-thinking approach to addressing the complex emotional needs of today's youth, ultimately aiming to equip them with the skills necessary for personal and social success.

Keywords:

Social Emotional Learning (SEL); Mental Health; Cultural Competence; Adolescents; School; Education

Gen AI 2024 - Munoz-Najar Galvez

Project Supervisor:

Sebastian Munoz-Najar Galvez, Bluhm Family Assistant Professor of Data Science and Education (HGSE)

Project Title:

Writing for college adminissions with AI Assistance

Project Overview:

Our goal: We want to inform a practitioner audience of high school counselors of the various model behaviors they might encounter when writing letters of recommendation for college applications with AI assistance. Motivation: Our prior research shows that letter writing is a time-consuming process, particularly for new counselors. Counselors device various writing instruments to reduce the time costs and ensure a fair representation of each student. These instruments are collections of templates, model letters, and outlines that counselors cultivate and share over time. We expect that free, general use AI models will transform how counselors amass writing tools of various types and degrees of quality. Prompts are likely to disseminate through the same informal networks that benefit some junior counselors with access to writing instruments capable of framing the experiences of college applicants in their school. This expectation, grounded in ongoing mixed methods research, motivates us to contribute in advancing public access to resources about what counselors can expect from AI writing assistance.

The project: The Letters Project is an investigation employing both quantitative text analysis and qualitative interviews to examine the writing practices of recommendation letter writers in college admissions. One of the goals of the project is to describe the behavior of LLMs in realistic scenarios of writing assistance. Through analyzing real letters and consulting with high school counselors, we have crafted a synthetic letter generation system. This system utilizes a keyword bank based on actual recommendation letters to simulate writing assistance requests from high school counselors. For example, our system simulates how a counselor might request help to write a paragraph about student extracurriculars. These simulations are grounded in realistic scenarios of rhetorical choice that we learned about from interviews. Our team: We are a team of social researchers with various disciplinary backgrounds (Sociology, Psychology, English, Business) and a common focus on higher education.

Opportunity for the Fellow:

Collaborators will work with a python pipeline to produce synthetic recommendation letters. This process will primarily involve modifying prompt sequences with randomized student traits to generate letters that, for example, maximize differentiation between documents in a sample defined by a common topic. Below we illustrate the three kinds of assisgnments we will co-design with our research assistants Hack-a-prompt: Based on our ongoing analysis of interviews with counselors, we want to analyze a sample of synthetic lettters that share the inclusion of experiences of bereavement. Operationally, we first generate a sample of synthtetic bragsheets that randomize various student traits and experiences but hold constant the module that seeds keywords about bereavement into bragsheet answers. These bragsheets are the data input that RAs will process with different prompt chains to generate highly differentiated letters. (1) Write research memos: Critically, the RAs will document their own process, what they learn from inspecting the synthetic letters and associated bragsheets, as well as their rationale to try a different prompting strategy. They will also have a codebase to evaluate the distributiion of similarity scores with different prompting strategies. (2) Present results in lab meetings: RAs will receive feedback from our team of graduate students working on the other arms of the Letters project. RAs will participate in setting analysis priorities and designing the next hack-a-prompt investigation. (3) Code: We have a working code base for letter generation but we are interested in improvements and neater packaging to advance replicability.

Outcomes:

A short report for counselors about AI-writing assistance. What kinds of behavior should they expect from AI when using different prompt chains to write about given topics--e.g. extra-curriculars, adversity, awards? An ACL short paper submission. Reference: https://aclrollingreview.org/cfp A Github repository to reproduce analyses.

Keywords:

linguistics, sociology, education

Gen AI 2024 - Nair

Project Supervisor:

Gautam Nair, Assistant Professor of Public Policy (HKS)

Project Title:

Business and the Representation of Weak Interests in Politics and Policy

Project Overview:

How do weak interests get represented in politics? Producers seeking protection from import competition have the incentive and capability to lobby for favorable economic policy. Consumers, the beneficiaries of free trade, are the archetypal diffuse interest, being too dispersed and numerous to organize collectively. But trade barriers on consumer goods have nevertheless decreased in recent decades, ushering in an era of unprecedented variety and affordability, at the expense of import-competing manufacturers. This project develops a novel explanation for this puzzle: the retail sector's transformation from one comprised of many small establishments to a sector dominated by behemoths created a powerful, concentrated interest that profited from cheaper imports, leading to the "incidental representation" of dispersed consumer interests in trade policy. Qualitative and archival data, Congressional voting on US trade policy, and cross-national data on trade flows are consistent with the theory. We will gather additional data to rigorously evaluate the theory and illustrate the applicability of the concept to domains beyond trade politics, such as the representation of low-income consumers in social policy. The data sources used for our applications of GenAI will include speeches by legislators, archives of Congressional hearings and policy-making, a unique open-ended survey of thousands of businesses, and public positions and party manifestos in the United States and India.

Opportunity for the Fellow:

The student will have the opportunity to learn about a number of important domains in policy and politics, including trade policy, business influence in politics, and social policy, while also learning about applications of new GenAI tools to social science research. It should also be fun.

Outcomes:

The principal outcomes will include the analysis of a new dataset of Congressional hearings, legislative speech, business positions, and political party positions. These will be used in ongoing projects on the representation of consumer interests and the evolution of party positions in the United States and India.

Keywords:

Politics, Political Economy, Trade Policy, Social Policy, United States, India

Gen AI 2024 - Neel

Project Supervisor:

Seth Neel, Assistant Professor in the Department of Technology and Operations Management (HBS)

Project Title:

A Novel Approach to Safety Training for Large Language Models

Project Overview:

State of the art training of Large Language Models or (LLMs) follows a standard pipeline: (i) "pre-training" via the task of next token prediction on a large corpus of web data (ii) instruction-tuning where the model learns to mimic questions and responses on a high quality carefully crafted dataset and (iii) safety training where the model learns to avoid undesirable behaviors like producing racist or toxic output, teaching users how to make weapons etc. This last part of the pipeline, henceforth referred to as "alignment," is the focus of this summer project. Current approaches for "aligning" base language models attempt to "steer" models away from undesirable behaviors (e.g., via RLHF or instruction tuning). One obvious shortcoming of these methods is that they rely on human-generated labels, and many grant proposals will likely focus on overcoming this important limitation.

Our proposal focuses on a second, arguably equally important failure mode of current alignment methods - namely, their (lack of) robustness. In particular, it has become increasingly clear (both through anecdotal [3] and scientific reports [1] ) that while current alignment methods have taken huge strides towards making models safer in practical settings, they are still quite brittle. Attackers can circumvent safety guardrails using simple prompting schemes or more sophisticated attacks (e.g., prompts optimized against open-source LLMs [2] ). Given a specific attack, specific defenses suggest themselves [8], and a familiar security "arms race" seems set to ensue, with attacks always staying one step ahead of defenses. A key reason behind the brittleness of alignment methods is that steering models towards safe answers does not remove their ability to output unsafe answers.

In this project we take an alternative data-centric perspective on alignment. Consider an LLM M that was trained on a corpus of text S. Given an unsafe response q to a prefix p, rather than steering M(p) to maximize a reward function learned using safe answers, we propose to: (i) Identify what underlying training data c(q) \subset S is responsible for the model M producing q; and then (ii) Efficiently remove the effect of training on c(q) from the LLM. Note that this data-centric approach is largely orthogonal to existing methods, and can thus be applied in conjunction with, rather than in place of, preference-based techniques. Until recently, both steps (i) and (ii) above would be infeasible, particularly in the context of large models and datasets. This approach is newly possible due to independent breakthroughs on each of these problems, several of which my lab is actively working on. The problem (i) of identifying what training data is responsible for a given model output is known as data attribution [4]; recent advances leverage the surprising fact that very simple models with a small amount of supervision are capable of predicting the effect on a given output of training on a specific piece of data, in both image recognition models and in LLMs [5] . As for step (ii), given the attributed data c(q), the task of efficiently removing that information from the model is known as machine unlearning, and has been extensively studied in recent years in the context of privacy [6 , 7] but never in the context of data attribution or alignment. While the task of tying these pieces together into a scalable and effective tool for alignment is a very ambitious one, we believe it is very possible to devise principled methods for solving this problem, which can later be expanded upon by companies like OpenAI.

Opportunity for the Fellow:

This project is a great opportunity for a team of two students. Lab Culture & Interaction. My research lab, which is located in the engineering school, (SAFR AI Lab @ Harvard) is hosting 2 other interns over the summer, as well as 1-2 PhD students, and 2 post docs. We will have weekly lab meetings, as well as lab events, where they will get to interact with the other researchers. They will join our lab slack, where they can ask questions, and swap notes about the related topics everyone in the lab is working on. They will benefit from technical mentorship, and get a realistic idea of what pursuing a career in A.I. research through a PhD would look like. Research Mentorship and Skills. In addition to our weekly lab meeting, I will meet with the students once per week to review progress. They will learn how to digest state of the art research papers, work with the accompanying research codebases, run experiments at scale on our computing cluster, present results weekly at lab meetings, and formulate novel hypotheses and experiments. Career Development. Given how competitive the field of A.I. currently is, relevant undergraduate research in the subject is becoming increasingly essential to obtain admission to a top PhD program. In addition to helping them progress towards a potential research career, the topics we are investigating are of high relevance to potential employers like OpenAI or Microsoft or Google. Working in the lab will prepare our students well for these important opportunities down the line.

Outcomes:

This project is ideal for a team of students, because although it is quite ambitious to complete the entire project, there are many intermediate outcomes that could lead to successful projects in their own right. When I say successful project, I mean a research paper submitted for publication with an accompanying code base. It is not my goal for the students to finish such a project in a single summer, but to lay the groundwork for a later publication if the results are promising. As a result in August I will have the students write up their findings as a final report, and collate their codebase into a github repository with documentation. This will serve two purposes (i) it will teach them how to write up a project and (ii) it will allow subsequent researchers (or them if they continue on during the semester) to build on ther results. In terms of topics we investigate, I imagine the following tier of possible outcomes, with Tier 1 being what I absolutely expect us to achieve over the summer, and Tier 1-3 reflecting what could be achieved by subsequent students or as part of ongoing engagement with the lab. Tier 1: Methods & Evals: evaluation of attribution and unlearning in large models. There has been very little work studying attribution in LLMs and so we would first (i) verify results in the TRAK paper on unlearning in LLMs (ii) scale their results up to larger models. Tier 2: Proof of Concept: Alignment pipeline of attribution followed by unlearning on smaller open source models (Llama-7B etc) Tier 3: Application to Strong Models (LLama-70B or larger) and tackling of associated issues.

Keywords:

LLMs, machine unlearning, explainability, alignment

Gen AI 2024 - Pfister (Project #1)

Project Supervisor:

Hanspeter Pfister, An Wang Professor of Computer Science (SEAS)

Project Title:

Fairness in AI: A Fair(er) Evaluation Benchmark Using Vision-Language Models

Project Overview:

Vision Language Models (VLMs) underpin much of the powerful AI technology today. They are used for image retrieval, visual question answering, image captioning, image generation, and dozens of other applications. Recent works have demonstrated that these models do not perform equally on different groups. For example, when using CLIP - perhaps the most popular VLM backbone - for retrieving images of say, "doctors", results are unfairly skewed along gender and race attributes. The issue, however, is that our understanding of these model's fairness is entirely limited to the datasets used to evaluate them. Current fairness datasets have many issues, including not being private, inclusive, diverse, and relying on annotators. To address these issues, our research project aims to develop a new, comprehensive, large-scale fairness benchmark for evaluating vision-language models (VLMs). Our goal is to include images and texts that are diverse along sensitive attributes, such as skin tone and perceived gender presentation. A crucial component is the inclusion of historically underrepresented groups in AI benchmarks, such individuals utilizing assistive devices, to enhance our understanding of VLM performance in these contexts. Unique to our approach is the generation of the entire benchmark dataset using generative AI techniques, ensuring both text and image modalities are synthetic, private, yet highly realistic. For text, we will leverage large language models (LLMs), and for images, we will employ cutting-edge methods like Stable Diffusion 2. This project stands at the intersection of AI fairness, generative AI, and synthetic data, driven by the vision to promote more inclusive and equitable AI technologies. Through this endeavor, we aspire to address the critical need for benchmarks that accurately reflect a wide spectrum of human diversity, ultimately guiding the development of VLMs that serve all segments of society equitably.

Opportunity for the Fellow:

Students participating in this project will engage in cutting-edge research at the forefront of AI and ethics. They will gain hands-on experience with generative AI technologies, including LLMs and stable diffusion, and contribute to the critical task of benchmark development. Moreover, students will explore the ethical dimensions of AI through practical research, learning to assess and mitigate bias within VLMs. This opportunity not only equips students with technical skills but also fosters a deep understanding of the social implications of AI technologies.

Outcomes:

By the end of the project, we anticipate: -- The development of a novel fairness benchmark for VLM evaluation, complete with unsupervised annotations. -- Comprehensive analysis and documentation of the fairness and biases present in current VLMs using the new dataset. -- A set of recommendations for the AI research community on improving fairness in VLMs. -- The publication of research findings in a peer-reviewed computer vision venue, contributing valuable insights to the broader AI community.

Keywords:

Fairness Benchmark; Vision-Language Models; Generative AI; AI Ethics; Bias Mitigation

Gen AI 2024 - Pfister (Project #2)

Project Supervisor:

Hanspeter Pfister, An Wang Professor of Computer Science (SEAS)

Project Title:

Dynamic Data Visualizations for Sports Game Highlights with Large Language Models

Project Overview:

Creating personalized visualizations to enhance data accessibility is critical in visualization research across multiple domains, including sports analytics. Enabling individuals to craft visualizations that convey data insights effectively can significantly improve performance analysis and communication among coaches and players. Currently, a gap exists due to the complexities of generating informative video highlights, as informed by our collaboration between SEAS and the Harvard Men's Basketball Team [1]. This research project aims to bridge this gap by utilizing language as a natural form of interaction, assisting sports experts in visualizing dynamic spatial data within sports videos. By using advanced language generative models (LLMs), our research aims to provide a new paradigm for authoring dynamic data visualizations based on natural language. This will open up opportunities for non-visualization experts to present their insights with intuitive user interfaces, enhancing data accessibility and literacy for both creators and audiences. [1] From game footage to great footage -- Visual Computing Group debuts SportsBuddy highlight program . https://seas.harvard.edu/news/2024/02/game-footage-great-footage

Opportunity for the Fellow:

The student involved in this project will gain hands-on experience with Large Language Models (LLMs), learning to develop LLM-based interaction to generate dynamic data visualizations. They will learn advanced techniques in computer vision and machine learning pipeline development, transforming textual data into dynamic visual narratives tailored for sports analytics. The student will be mentored by experienced researchers and trained skills in problem-solving, collaboration, data visualization and software development. This experience promises to be a valuable asset to their academic and professional development, offering insights and skills that are highly sought after in both academic and tech industry.

Outcomes:

The student is expected to experiment with the LLM model and its capabilities to transform verbal insights into dynamic spatial visualization in video. The outcome will be a software prototype involving an intuitive user interface, backed with a machine learning pipeline, that allows a user to type in textual input, and output informative highlights of a sports game video. This research prototype will contribute to broadening the impact of data visualization in real-world sports applications.

Keywords:

data visualization, sports analytics, text-based video creation, informative game highlight

Gen AI 2024 - Rajpurkar

Project Supervisor:

Pranav Rajpurkar, Assistant Professor of Biomedical Informatics (HMS)

Project Title:

Advancing Uncertainty Estimation in Multimodal Medical AI Models

Project Overview:

Vision language models (VLMs) have demonstrated remarkable capabilities in medical AI, enabling tasks such as medical image interpretation and visual question answering. However, a key challenge in deploying these models in clinical settings is the lack of reliable uncertainty estimation. Overconfident incorrect predictions can lead to serious consequences in patient care. This project aims to advance uncertainty estimation techniques for multimodal medical AI models, focusing on VLMs that process both medical images and text. We will explore methods such as Bayesian deep learning, ensemble modeling, and test-time augmentation to quantify predictive uncertainty. By developing robust uncertainty measures, we can enable selective prediction, expert referral, and informed decision-making. The models will be evaluated on large-scale datasets spanning multiple imaging modalities. Through collaborations with clinicians, we will assess the impact of uncertainty-aware predictions on clinical workflows and patient outcomes. This research will contribute to building trustworthy and reliable medical AI systems.

Opportunity for the Fellow:

The selected undergraduate student will have a unique opportunity to work at the forefront of uncertainty estimation research in medical AI. They will gain hands-on experience in developing and implementing advanced techniques for quantifying uncertainty in multimodal models. The student will collaborate closely with a multidisciplinary team of AI researchers and clinicians, learning to navigate the complex intersection of machine learning, computer vision, and healthcare. They will have access to state-of-the-art computational resources and large-scale medical datasets. The student will be mentored by experts in the field, receiving guidance on research methodologies, technical skills, and scientific communication. They will have opportunities to contribute to research publications, present at conferences, and engage with the broader medical AI community. This experience will provide a strong foundation for future graduate studies or industry roles in the rapidly evolving field of medical AI.

Outcomes:

Anticipated Outcomes: Development of novel uncertainty estimation techniques for vision language models in medical AI, leveraging approaches such as Bayesian deep learning, ensemble modeling, and test-time augmentation. Comprehensive evaluation of uncertainty measures on large-scale, multimodal medical datasets, assessing their effectiveness in capturing predictive uncertainty. Integration of uncertainty estimation into medical AI models for tasks such as medical image interpretation and visual question answering, enabling selective prediction and expert referral. Collaborations with clinicians to study the impact of uncertainty-aware predictions on clinical decision-making, workflow efficiency, and patient outcomes. Research publications and conference presentations showcasing the advancements in uncertainty estimation for medical AI. Hands-on experience and skill development for the student in cutting-edge AI techniques, uncertainty quantification, and interdisciplinary research at the intersection of AI and medicine.

Keywords:

Uncertainty Estimation, Vision Language Models, Medical AI, Multimodal Learning, Clinical Decision Support

Gen AI 2024 - Sayegh

Project Supervisor:

Allen Sayegh, Design Critic and Senior Interaction Technologies Fellow (GSD)

Project Title:

Responsive Design: Real-Time Architectural Adaptation through Generative AI and Eye-Tracking

Project Overview:

GenAI Research Program/ Summer Funding for Harvard College Students 2024 PI: Allen Sayegh This ten-week long summer initiative will add to the research done at the Harvard GSD Responsive Environments and Artifacts Lab, on quantifiable methods for understanding architectural spatial qualities, and the influence of built environments on human experience. The student will be part of our research project, that proposes an innovative application of Generative AI and eye-tracking technology in mixed-reality environments as a novel research method in the fields of architecture, environmental psychology, and spatial ethnography. Our goal is to leverage GenAI to dynamically create architectural spaces that respond in real time to how people perceive their surroundings, measured through advanced eye-tracking indicators like gaze fixation, pupil dilation, and blink rate. Typically, the influence of architectural design on human experience is only fully understood after a structure is built, with little room for adaptive changes based on immediate feedback. Our approach disrupts this by using Generative AI in the research, allowing for faster capture and analysis of how spatial characteristics may affect perception by adapting designs immediately to participant feedback. The participants' arousal is documented within a sequence of spaces, defined by objective spatial characteristics, and is compared with augmented iterations of the same sequence prompted by subjective characteristics from participant feedback. This research method may benefit risk-averse industries such as healthcare, education, and public infrastructure, where optimizing human experience is crucial yet experimental changes to spaces are constrained. The findings are expected to contribute valuable insights into the field of architecture, particularly in understanding how different spaces can elicit positive and negative emotional arousal. The relation between spatial characteristics and emotional arousal has broad implications spanning a variety of public and private institutions and industries that require humans to occupy built environments, from patient anxiety in a hospital waiting room, passenger sensory overload in an airport, or fear in classrooms. This research stands to benefit risk-averse industries such as healthcare, education, and public infrastructure, where optimizing human spatial experience is crucial yet experimental changes to physical spaces are constrained. By providing a method to rapidly adapt and iterate on design variations virtually, architectural designs can be designed to navigate emotional arousal with confidence, minimizing risk while maximizing potential benefits to human well-being.

Opportunity for the Fellow:

This research project offers undergraduate students a rare and invaluable opportunity to engage directly with the cutting-edge intersection of Generative AI (GenAI), eye-tracking technology, and mixed reality (MR) environments, within the context of architectural design and environmental psychology. Students will participate in a hands-on research experience that extends beyond conventional academic boundaries, gaining insights into the innovative application of technology in understanding and influencing human emotional responses to spatial characteristics. The primary focus for students will be to explore and establish how various prompting techniques can influence the generation of spatial characteristics in GenAI-created images, videos, and spaces. This exploration is crucial for developing a robust framework that will guide future research, including the detailed investigation of spatial characteristics on emotional arousal in Mixed Reality environments. Students will have the opportunity to immerse themselves in the theoretical and practical aspects of Generative AI, learning how to craft effective prompts that lead to the desired spatial outputs, and ones that evoke specific subjective architectural characteristics. This will involve a blend of creative thinking, technical skills, and an understanding of architectural principles. While the direct application of eye-tracking technology and mixed reality (MR) environments will be part of a subsequent phase of research, students will gain preliminary exposure to these technologies, understanding their potential role and application in analyzing spatial experiences. This involvement will provide students with practical experience in cutting-edge technological tools and methodologies, positioning them at the forefront of architectural and environmental psychological research. Furthermore, students will develop a nuanced understanding of the ethical considerations and potential societal impacts of integrating AI in architectural design, fostering a responsible approach to technological innovation. By participating in this ten week long project, students will lay the groundwork for cutting-edge research, contributing to the development of guidelines that will inform the iterative design process using GenAI. This experience will not only enhance their technical and analytical skills but also offer them a unique perspective on the intersection of technology and spatial design, preparing them for future endeavors in a variety of fields.

Outcomes:

The anticipated outcomes of this foundational research phase are centered on establishing a comprehensive understanding of how GenAI can be leveraged to generate and refine spatial characteristics based on specific prompting strategies. Through the students' contributions, we aim to:

Development of Advanced Prompting Techniques for Spatial Design: We aim to refine and advance the methods used to prompt GenAI, creating a nuanced understanding of how specific prompts can influence the generation of spatial characteristics in images and videos. This outcome will include a comprehensive catalog of effective prompts that can evoke distinct spatial experiences, contributing to the body of knowledge in GenAI and its application in architectural design.

Framework for Iterative Architectural Design Feedback: Using GenAI: A critical outcome will be the establishment of a preliminary framework that outlines guidelines for the iterative use of GenAI in the architectural design process. This framework will detail the methodologies for integrating real-time human emotional feedback into the design of spaces, thereby enabling future researchers and practitioners to incorporate GenAI into their workflows efficiently and effectively. The framework will serve as a foundational guideline for future projects that seek to explore the dynamic interplay between human emotion and spatial characteristics.

Insights into Human-Spatial Interaction: Through the use of eye-tracking technology and the analysis of participant responses to GenAI-generated environments, the project will yield deep insights into how individuals perceive and react to different spatial configurations. These insights will contribute to a richer understanding of the psychological and emotional impacts of architectural spaces, offering valuable data that can inform design principles aimed at enhancing human well-being.

Enhancement of Multidisciplinary Research Methodologies: The project stands to significantly advance research methodologies at the intersection of technology, architecture, and psychology. By demonstrating the effective integration of GenAI and eye-tracking within mixed-reality environments, this research will provide a blueprint for future studies looking to explore complex human-environment interactions.

While the ten-week project's primary focus is on the exploratory work with GenAI and prompting strategies for videos and images, the exposure to eye-tracking and MR technologies will provide valuable contextual knowledge. The research initiative will not only contribute to academic knowledge but also offer practical guidelines and insights that can be applied in the design of more responsive, human-centered environments. The project thus holds the promise of significantly impacting both theoretical understanding and practical approaches to architectural design and environmental psychology. These outcomes are expected to contribute significantly to the field of architectural research by providing a novel approach to integrating GenAI into the design and analysis process. The groundwork laid by this project will be instrumental in guiding future research efforts, including the exploration of spatial characteristics' effects on emotional arousal in more advanced phases.

Keywords:

Generative AI, Eye-Tracking Technology, Mixed Reality Environments, Architectural Design, Emotional Arousal, Environmental Psychology

Gen AI 2024 - Slade

Project Supervisor:

Patrick Slade, Assistant Professor of Bioengineering (SEAS)

Project Title:

Generative AI for modeling and predicting human movement

Project Overview:

The field of biomechanics treats mobility disorders by analyzing human motion to diagnose disorders, track rehabilitation, or assist people with a robotic device. This motion analysis is limited to the laboratory and desperately needs the ability to estimate the rich dynamics of motion from accessible, wearable sensor data. However, biomechanics only has small, specialized datasets because capturing and processing ground-truth motion data requires hours of hands-on work to generate a few seconds of data. A recent automation method has solved this problem and merged disparate datasets to create a rich repository of 2.4 million frames of ground-truth movement, a 30 times increase in size. I propose to use generative AI to seize this field-altering opportunity to create the first large human motion model. Specifically using a diffusion model to estimate the full dynamics of human movement from a subset of wearable sensors worn on the body, making laboratory-grade motion analysis possible in the real-world. This will enable: 1) inferring full-body dynamics from any number of sensors, providing application flexibility and robustness to sensor dropout; 2) a general motion model that can be personalized by conditioning on some subject-specific data; and 3) predicting future motion trajectories based on a time history of motion. The potential for this technology is massive, impacting many domains like rehabilitation, geriatrics, sports, animation, and human-computer interaction. This technology could improve the health and mobility for tens of millions of people by initiating a wave of AI-enabled diagnosis and rehabilitation tools.

Opportunity for the Fellow:

The student will develop a generative AI model that captures human motion using this first-of-its-kind dataset. Specifically, the student will likely develop a diffusion model to estimate the full dynamics of human movement from a subset of wearable sensors worn on the body, making laboratory-grade motion analysis possible in the real-world. We propose a diffusion model due to its success in text-to-motion generative systems, which outperformed other approaches like variational autoencoders. Conditioning diffusion models on a subset of the full-body motion has proved viable in initial tests with small datasets. The new, large dataset contains full-body dynamics for ground truth labels. We will simulate wearable sensor data from this ground-truth data, a common method we perform extensively in our lab. This is an exciting opportunity for the student to create and open-source the first large motion model to provide a landmark shift in the field biomechanics and movement sciences to perform laboratory-grade motion analysis using wearable sensors. The student will learn how to apply generative modeling to a human-motion project to understand and estimate key motion features. The student will have a hands on experience where they can utilize an existing large dataset to train a model and utilize our existing wearable sensors and laboratory sensors to evaluate how well their model performs in practice. The student will have direct mentorship from both a graduate student and Professor Slade to recieve guidance in research and development as a scientist.

Outcomes:

This technology could improve the health and mobility for tens of millions of people by initiating a wave of AI-enabled diagnosis and rehabilitation tools. The potential for this technology is massive and spread over domains in rehabilitation, geriatrics, sports, animation, human-computer interaction, and many others. This could be used for diagnosing musculoskeletal diseases like Parkinson's based on their gait, tracking recovery after a surgery, predicting fall risk for older adults, informing exoskeleton control to augment movement, informing other human-interaction research like shared-autonomy systems, and simulating physically realistic motion for movies. As a part of the committee planning the AI Intellectual thrust for SEAS, I believe this proposal fits the generative AI mission for FAS and both SEAS thrusts in AI and Human Health: augmenting human capabilities and health with AI.

Keywords:

Diffusion models, biomechanics, movement, rehabilitation

Gen AI 2024 - Snyder

Project Supervisor:

James Snyder, Leroy B. Williams Professor of History and Political Science (FAS)

Project Title:

OCR for Digits 0-9

Project Overview:

The internet now contains millions of pages of useful but unused tables of numbers. These are scans of books and reports, official government documents, and so on. Examples for the U.S. include: historical election returns at the county, town or precinct level; public records on state, county, and municipal government spending and taxes; and tables of demographic and socio-economic information at varioius levels of aggegation. One key reason the information in these pages goes unused is that the error rates of the existing open-source software for doing optical character recognition is too high. The project will use a mix of unsupervised and supervised machine learning techniques to build an open-source OCR specializing in tables of numbers. AI-driven OCR software frequently uses supervised learning, or training the model on images of characters with matching labels. While this method is certainly useful, it can only accurately recognize characters written similarly to the characters it was trained on. Many historical data sources, even typed ones, feature varied fonts and damage levels that impede the ability of current open-source OCR software to achieve near-perfect accuracy. An unsupervised learning approach removes the reliance on similarity between the document in question and the documents the model has seen before. The model will be asked to treat each number as a data point and assign it to one of exactly ten clusters (or a few more, if including punctuation such as commas, periods, and dashes). Hierarchical agglomerative and K-means clustering will be the first algorithms used, but the model will be attempted with many clustering methods over the course of development. The algorithms will be optimized to exploit the fact that the exact number of clusters is small and known in advance. Combining the unsupervised and supervised approaches -- in many cases weighted heavily towards the unsupervised algorithms, we conjecture -- should produce algorithms that accurately classify the digits for a much wider set of documents than any existing OCR software.

Opportunity for the Fellow:

The student will have the opportunity to build an OCR program specialized for tables of numbers. This experience will develop the student's skills both in using existing AI techniques and in developing novel AI approaches if deemed necessary. The need for accurate analysis of diverse documents will require the student to explore a variety of machine learning techniques, ranging across the unsupervised learning realm and perhaps into other methods like reinforcement learning. With the latitude to build and test OCR models using a variety of algorithms, the student will be exposed to the real-world problems of optimization and model selection. The student may also be asked to craft entirely new approaches to the problem, depending on the performance of more conventionally constructed models. The student will be expected to innovate as part of their investigation of OCR, and will come away from the project having made their own contribution to a novel AI solution.

Outcomes:

The project will produce OCR software that is ready to turn any table of numbers into a usable dataframe. The software will have an immediate impact on countless research projects, allowing researchers to more easily utilize historical documents and avoid hours of data cleaning otherwise needed to correct OCR mistakes. Election data, economic data, and demographic data from decades and centuries past are stored in documents that require OCR, and analyzing these data at the county, city, or other local level means OCRing thousands of rows of numbers. Building a dedicated OCR for this task will advance countless research projects, in the U.S. and around the world.

Keywords:

Optical Character Recognition, Historical Tables, Machine Learning

Gen AI 2024 - Usmani

Project Supervisor:

Adaner Usmani, Assistant Professor of Sociology and Social Studies (FAS)

Project Title:

Punishment in Comparative and Historical Perspective: Using GenAI to Read Archival Documents at Scale

Project Overview:

In the early to mid-20th century, the United States incarcerated around 150-200 people per 100,000 people. This is comparable to the rate at which, today, most developed countries incarcerate their populations. But in the 1970s, something changed. By 2007, the US incarceration rate had more than quintupled, to 767 per 100,000 people. This was among the highest incarceration rates ever observed in world history, not too far behind Stalin's Soviet Union at the height of the gulags. The risk of incarceration is not equal by social position. Men, African-Americans, and people with low levels of education face higher incarceration rates than women, whites, and college graduates. These inequalities have not always changed together. Since the punitive turn that began in the 1970s, men's share of the prison population decreased, racial inequalities initially rose but have recently fallen, and educational inequalities have risen significantly. In seeking to explain the puzzle of American mass incarceration, scholars have focused almost exclusively on the US. In my view this represents a key weakness of this literature (Clegg, Spitz and Usmani 2023). Although American mass incarceration is a comparative and historical puzzle, scholars have mostly sought to understand America by studying only America and often only its very recent past. This project seeks to remedy this America-centrism and lack of attention to class by collecting and analyzing comparative and historical data on class inequality in punishment. For almost four years, I (together with one of my PhD students and John Clegg from the University of Sydney) have been leading a team of research assistants who have been collecting comparative and historical data on crime, punishment, policing, and related statistics. To date, we have built a dataset based on more than 2,900 primary and secondary sources. Our data collection efforts to date have already yielded data on punishment for over 200 countries. For a few dozen countries, these data span much of the modern period. We have built this dataset thanks to funding from my start-up fund, the Milton Fund, Lakshmi Mittal Institute, Brazil Cities, Weatherhead Center, and Canada Program.

This projects' results will be disseminated to a scholarly and public audience in three main ways. First, I am writing a book based on these data (co-authored with John Clegg), titled From Plantation to Prison. This book is already under contract with Harvard University Press, which has designated it an academic-trade book in the expectation that it will be widely read. It is currently expected to be published in 2025. Second, we are publishing multiple papers based on these data, one of which was recently published in the Annual Review of Criminology (Clegg et al. 2024) and another is forthcoming in Oxford Handbook of Comparative Historical Sociology (Clegg, Spitz, and Usmani 2024). In our future work, we plan at least one paper which will be focused on class inequality in punishment over time, and we are preparing a second methodological and theoretical paper addressing the complexities of measuring inequality across time and place. Third, upon the publication of the book and these papers, we plan to release the dataset for public use. We hope this will allow others to build on our contribution. We expect it will encourage the literature to pursue comparative and historical approaches in general, and class inequality and incarceration in particular.

Opportunity for the Fellow:

Thus far, much of our data collection has involved having RA's manually enter data located in scanned primary source reports. We might have a RA entering data from the 1910 West Virginia penitentiary report, or the 2000 Sri Lankan prisons report. This has been fruitful, but highly time consuming. Generative AI will help to speed up this process. This past academic year, we have begun to use Chat GPT as part of our data collection. Chat GPT is capable of reading a scanned table, and converting it into a usable format, such as a CSV file. With appropriate prompting, Chat GPT can replace much of the manual data entry we have been doing, speeding up data collection and allowing us to look through a broader range of sources. This past academic year, we have tested this approach on decades' worth of report from the International Criminal Police Organization (Interpol). The Interpol reports are ideal for Chat GPT; they were issued in the same format, year after year, with easily read tables. Our efforts have been successful at digitizing the Interpol data. Over the summer, we are planning on scaling up our Chat GPT efforts. We would like to use Chat GPT on more sources. The research assistant will be tasked with using Chat GPT to generate data for our dataset, from a set of assigned sources. This will probably include back-end commands through the API. Since we have had an RA working on this part-time, for the past year, this work will not start from scratch and the RAs can build on earlier code. Students working on the project over the summer will be considered full members of the Histpun team. They will be expected to attend weekly meetings, and invited to participate in all aspects of the project, including attending our substantive "ideas" sessions, working on papers, managing the dataset, and non-GPT coding opportunities. They will have significant oversight from an experienced PhD student and myself.

Outcomes:

By the end of the internship, students will have developed significant experience working on a collaborative social science research project. They will have digitized a significant number of records for use in our dataset, expanded their coding skills, and explored how Chat GPT can be used to improve dataset quality. Students who wish to produce a writing sample will be encouraged to do so. On our end, the anticipated outcome is, of course, the significant expansion of our data collection.

Keywords:

crime, punishment, policing, mass incarceration, racial inequality

Gen AI 2024 - Villar

Project Supervisor:

Ashley Villar, Assistant Professor of Astronomy (FAS)

Project Title:

SpeakYSE: Enhancing Astronomical Data Exploration through Natural Language

Project Overview:

Astronomical datasets are rapidly growing to petabyte scales. Our team sifts through nightly datastreams to identify unusual explosions of stars (supernovae) in real time, and triggers limited follow-up observations on just a handful of events. We are interested in using generative AI to interact with our data as we do with our colleagues: through natural language. Our goal is to translate descriptions of our science priorities in target selection queries, while bypassing the obstacle of building bespoke SQL queries. We have access to thousands of supernovae through the Young Supernova Experiment ("YSE"), a survey of the northern sky using the Pan-STARRS telescopes in Hawaii. Each supernova in our dataset is associated with several key modalities: an image of the galaxy where the supernova exploded; a sparse and multivariate "light curve" (time series of flux over time); and just a few spectra (the flux over wavelength at a single snapshot in time). This is in addition to other inferred features: the distance to the supernova; the amount of dust between us and the event; neural network-generated latent representations of the supernova; etc. The students will build upon the "SpeakYSE'' initiative, a fine-tuned, generative LLM for interacting with the YSE database. Given natural language queries submitted through a web interface, this model produces and completes SQL queries to the YSE database in real-time, and uses the retrieved data to guide its response. Our goal is to enhance, more broadly, the interactions between astronomers and our databases through natural language.

Opportunity for the Fellow:

Two projects are available for students: Enhancing SpeakYSE: One student will work to enhance the SpeakYSE model for more complex tasks. The student will integrate a reliable function-calling module that allows users to call python commands within the chat environment. Of particular use is the integration of plotting functions within the client for rapid visualization of data (in both observed- and latent-space), as well as functions for light curve-fitting and spectroscopic classification. Particularly motivated students could also develop functionality to schedule follow-up observations directly when prompted by a user. Building a Zooniverse Platform for SpeakYSE training: Development of SpeakYSE has, thus far, focused on retrieval of traditional data modalities. However, we are interested in exploring how expert knowledge can be integrated into our fine-tuned LLM. A student will develop a Zooniverse-like GUI to show users a combination of spectroscopic, image-based and light curve data. They will create an application that records audio of experts (and the expert's interaction with the screen) who are prompted to discuss the supernova data that they see. These descriptions will be automatically transcribed and associated with the supernova data shown through a multi-modal embedding network. This data will, eventually, be used to train SpeakYSE to return meaningful natural-language descriptions of supernovae based on their data alone. Students will work with PI V. A. Villar (Assistant professor of Astronomy) and Dr. A. Gagliano (IAIFI Fellow and SpeakYSE lead). They will engage in summer group meetings, talk series and local workshops for summer students at the Center for Astrophysics. Students are welcome to work individually on either project, or work in pairs on both.

Outcomes:

The SpeakYSE enhancements will be integrated into the public-facing SpeakYSE model (Gagliano et al., in prep), which is fully open-source. The Zooniverse-like enhancement will be used in a future study of fine-tuning LLMs given expert input, and the consolidated expert descriptions will be released to the public upon publication. The students will present their work at a time-domain CfA seminar, and (if applicable) at the 2025 AAS meeting.

Keywords:

Astronomy, supernovae, large language models, function-calling, front-end programming, SQL

Gen AI 2024 - Voulgaris

Project Supervisor:

Carole Voulgaris, Assistant Professor of Urban Planning (GSD)

Project Title:

Structured Data Generation from Narrative Text on Transit Capital Projects

Project Overview:

Most rail transit projects that have been constructed in the United States over the past fifty years have been funded through the Capital Investment Grant Program administered by the Federal Transit Administration. Each year, the Federal Transit Administration publishes an annual report with information on each project for which it recommends funding to support project planning, design and construction. Taken together, the set of annual reports that have been published since about 1998 represent a rich dataset that can allow us to answer important questions about the points in time and circumstances under which project budgets and timelines are likely to change over the course of project development and construction. However, this information is primarily embedded within unstructured narrative text, spread across dozens of annual reports. The purpose of this project is use large language models to mine these texts and assemble a structured dataset from these unstructured texts. We will use this database to answer questions about how project budgets and timelines generally evolve over the course of project development and construction.

Opportunity for the Fellow:

The student would develop a method and workflow to utilize the OpenAI API to extract data from each of approximately dozens of separate documents, including both narrative text and a data tables within PDF documents. They would develop prompts to direct the large language model to generate a structured dataset that includes every fixed-guideway transit project that has received federal funding over a period of 25 years and tracks how anticipated project costs and timelines have changed over each project's development and construction period.

Outcomes:

At the conclusion of the project, the student will produce: A database documenting changes in project budget and timeline for all fixed-guideway transit projects that have been funded through the United States Capital Investment Grants program over the past 25 years. A Python script that calls the OpenAI API and uses a set of predefined prompts to query existing Funding Recommendation documents to produce the database described above, and that can be used with future funding recommendation documents to add to the database in subsequent years.

Keywords:

Text mining, Urban planning, public transit, public finance, structured data generation

GA4 tracking code

Brief Description

Program Overview

Benefits

Eligibility

Summer 2024 - Project Descriptions

Gen AI 2024 - Amin (Project #1)

Project Supervisor:

Project Title:

Project Overview:

Opportunity for the Fellow:

Outcomes:

Keywords:

Gen AI 2024 - Amin (Project #2)

Project Supervisor:

Project Title:

Project Overview:

Opportunity for the Fellow:

Outcomes:

Gen AI 2024 - Bol

Project Supervisor:

Project Title:

Project Overview:

Opportunity for the Fellow:

Outcomes:

Keywords:

Gen AI 2024 - Brennan

Project Supervisor:

Project Title:

Project Overview:

Opportunity for the Fellow:

Outcomes:

Keywords:

Gen AI 2024 - Calmon

Project Supervisor:

Project Title:

Project Overview:

Opportunity for the Fellow:

Outcomes:

Gen AI 2024 - Chetty

Project Supervisor:

Project Title:

Project Overview:

Opportunity for the Fellow:

Outcomes:

Gen AI 2024 - Dell

Project Supervisor:

Project Title:

Project Overview:

Opportunity for the Fellow:

Outcomes:

Gen AI 2024 - Elani

Project Supervisor:

Project Title:

Project Overview:

Opportunity for the Fellow:

Outcomes:

Keywords:

Gen AI 2024 - Gaudet

Project Supervisor:

Project Title:

Project Overview:

Opportunity for the Fellow:

Outcomes:

Gen AI 2024 - Henrich

Project Supervisor:

Project Title:

Project Overview:

Opportunity for the Fellow:

Outcomes:

Gen AI 2024 - Holland

Project Supervisor:

Project Title:

Project Overview:

Opportunity for the Fellow:

Outcomes:

Keywords:

Gen AI 2024 - Janapa Reddi

Project Supervisor:

Project Title: