Research

A Hybrid Approach to Research

Academia + Industry

My research is an unusual combination of both traditional STEM and non-STEM academic research informed by myt diverse work experiences that includes computer security, healthcare, insurance, finance, intellectual property and more. I have patents and research published in both leading Science and Technology journals (virtual private networks, encryption, medical informatics, gene therapy) as well as the Social Sciences and Humanities (cultural analytics, narrative, philosophy and ethics, modern languages). My current research focuses on ML/AI, NLP, Large Language Models, eXplainable AI (XAI), Fairness-Accuracy-Transparency-Explainability (FATE), Multimodal Sentiment Analysis/Affective AI, Open Source AI Benchmarking/Metrics and Autonomous AI Agent/Agent Networks.

For the past decade, my research and teaching has included traditional statistical machine learning, data analytics and Deep Learning/AI including:

* AI End-to-End Pipelines: OpenAI API, LangChain, LlamaIndex, etc (eg. summarization, sentiment analysis, etc)
* Benchmarking: Individual models as well as ensembles of AI SOTA commercial vs Open-source models
* Data Synthesis and Augmentation (e.g. sentiment to anomalous time series)
* Dataset/Model Training Optimization: Small Open LLMs and Teacher-Student Distillation
* Prompt/Ensemble Optimization: (from simple prompt engineering to Mixture of Experts)
* Multimodality: text, image, video, timeseries, graphs, etc.
* Reduce/Minimize Hallucinations, Stochasticity, Factuality Errors
* Reasoning, autonomous agents, networks of collaborating autonomous agents
* Affective AI:, Emotions, Persuasion, and Influence (e.g. Narratives to Chatbots)
* Arising Enterprise AI Programming Paradigm (e.g. Microsoft Semantic Kernel, Stanford DSPy)
* Explainability, Fairness and Mechanistic Interpretability (e.g. trust, auditing, regulation compliance, risk/insurance)

OIn 2016 Katherine Elkins and I co-founded the world’s first human-centered AI curriculum and AI Lab at Kenyon College. Since then, we have mentored over 300 student research projects representing nearly every department across campus from Math department winners to creative writing scholarship winners. The AI Lab encompasses a broad range of applied state-of-the-art AI/LLM, GOFAI statistical ML and best software engineering practices from a wide variety of sectors including medicine, finance, law, journalism, art, literature, film, political science, sociology, philosophy, history, poetry, gender studies, etc. As of January 2024, this diverse and original research has been downloaded approximately 40,000 times by over 2700 institutions from 160 countries worldwide including Stanford, Berkeley, MIT, NYU, Columbia, Princeton, Oxford and Cambridge.

Current Research
Informed AI Regulation: Comparing the Ethical Frameworks of Leading LLM Chatbots Using an Ethics-Based Audit to Assess Moral Reasoning and Normative Values

ABSTRACT:

With the rise of individual and collaborative networks of autonomous agents, AI is deployed in more key reasoning and decision-making roles. For this reason, ethics-based audits play a pivotal role in the rapidly growing fields of AI safety and regulation. This paper undertakes an ethics-based audit to probe the 8 leading commercial and open-source Large Language Models including GPT-4. We assess explicability and trustworthiness by a) establishing how well different models engage in moral reasoning and b) comparing normative values underlying models as ethical frameworks. We employ an experimental, evidence-based approach that challenges the models with ethical dilemmas in order to probe human-AI alignment. The ethical scenarios are designed to require a decision in which the particulars of the situation may or may not necessitate deviating from normative ethical principles. A sophisticated ethical framework was consistently elicited in one model, GPT-4. Nonetheless, troubling findings include underlying normative frameworks with clear bias towards particular cultural norms. Many models also exhibit disturbing authoritarian tendencies. Code is available at https://github.com/jonchun/llm-sota-chatbots-ethics-based-audit.

Keywords: large language models (LLMs), AI safety, Human-AI alignment, AI regulation, ethics-based auditing (EBA), AI reasoning, autonomous AI, agentic AI

Talk: Multilayered Large Language Models Strategies for Generating Time Series Simulation Data

In the rapidly changing AI landscape, Large Language Models (LLMs) like OpenAI’s GPT4 initially may appear tangential to simulation technologies. However, with a closer look, the potential of LLMs becomes clear, presenting exciting opportunities for NAFEMS members. This presentation will deliver a hands-on exploration of leveraging OpenAI’s GPT4 and associated LLM frameworks to generate synthetic and enrich existing time series datasets, all within the context of physical simulations for anomaly detection—a key area of interest in the manufacturing industry.

We’ll use popular open training datasets used to train predictive maintenance models as both a ground truth reference and a source for augmentation. This will include both normal and abnormal time series for vibration, temperature, pressure and current/voltage measurements. We’ll use the ground truth reference time series datasets to evaluate both normal and abnormal time series generated and augmented using the best LLM strategy identified.Our performance metrics will be based on two statistical profiles tailored to two types of time series abnormalities: (a) a global regime/distribution shift type like those found in asset price trading bands, and (b) more localized feature anomalies that often predict impending failures.

We review an incremental progression of GPT4 utilization, illuminating its potential while addressing the inherent limitations. We’ll begin with basic GPT models, explore various prompt engineering strategies, then delve into Python Code Interpreter extension and OpenAI tool use via Langchain. We’ll explicitly address issues around hallucination, stale training data and innumeracy of LLM. The culmination will dive into OpenAI’s 0613 model updates, which introduce a new API function object specification that dramatically enhances the reliability of programmatically interfacing with GPT3.5 and GPT4 models.

The presentation will wrap with a comparative analysis of LLM data synthesis and augmentation techniques against traditional approaches, including both open source and commercial offerings like Gretel.ai. Listeners will be equipped with a practical and up to date understanding of the latest state-of-the-art GPT4 LLM and how to better utilize such generative LLM AI models for their generating or augmenting data for simulation or fine-tuning other AI models. We will close with an update on the recent and anticipated AI advancements, enabling you to better align future LLM applications to particular physical simulations.

The Crisis and Opportunities AI Presents for the Digital Humanities (Editors’ Featured Paper) (
International Journal of Humanities and Arts Computing (Oct 2023: Special Issue On Crisis in Digital Humanities Pedagogy)

ABSTRACT:

This article outlines what a successful artificial intelligence digital humanities (AI DH) curriculum entails and why it is so critical now. Artificial intelligence is rapidly reshaping our world and is poised to exacerbate long-standing crises including (1) the crisis of higher education and the humanities, (2) the lack of diversity, equity and inclusion (DEI) in computer science and technology fields and (3) the wider social and economic crises facilitated by new technologies. We outline a number of ways in which an AI DH curriculum offers concrete and impactful responses to these many crises. AI DH yields meaningful new avenues of research for the humanities and the humanistic social sciences, and offers new ways that higher education can better prepare students for the world into which they graduate. DEI metrics show how an AI DH curriculum can engage students traditionally underserved by conventional STEM courses. Finally, AI DH educates all students for civic engagement in order to address both the social and economic impacts of emerging AI technologies. This article provides an overview of an AI DH curriculum, the motivating theory behind design decisions, and a detailed look into two sample courses.

Keywords: Digital Humanities, Artificial Intelligence, Large Language Models, Generative AI, AI Alignment, AI Safety, AI Curriculum, AI Ethics, Computational Digital Humanities

eXplainable AI (XAI) for Large Language Model Story Analysis and Generation: A Novel Grey-Box Ensemble Method For the Newest Function-Enabled GPT4
(October 11, 2023)

ABSTRACT:

The recent development of Transformers and large language models (LLMs) offer unique opportunities to work with natural language. They bring a degree of understanding and fluidity far surpassing previous language models, and they are rapidly progressing. They excel at representing and interpreting ideas and experiences that involve complex and subtle language and are therefore ideal for Computational Digital Humanities research. This paper briefly surveys how XAI can be used to augment two Computational Digital Humanities research areas relying on LLMs: (a) diachronic text sentiment analysis and (b) narrative generation. We also introduce a novel XAI greybox ensemble for diachronic sentiment analysis generalizable to any AI classification data points within a structured time series. Under human-in-the-loop supervision (HITL), this greybox ensemble combines the high performance of SOTA blackbox models like gpt-4–0613 with the interpretability, efficiency, and privacy-preserving nature of whitebox models. Two new local (EPC) and global (ECC) metrics enable multi-scale XAI at both the local and global levels. This greybox ensemble framework extends the SentimentArcs framework with OpenAI’s latest GPT models, new metrics and a modified supervisory HITL workflow released as open source software at https://github.com/jon-chun/SentimentArcs-Greybox.

Keywords: GPT4, LLM, XAI, Greybox XAI, Ensemble Learning, GPT4 function calling, NLP, Sentiment Analysis, Story Generation/SentimentArcs-Greybox.

Comparing the Ethical Frameworks of Leading LLM Chatbots: A Novel Ethics-Based Audit to Assess Moral Reasoning and Normative Values
(Under Review, Submitted Summer 2023)

ABSTRACT:

This paper undertakes an ethics-based audit to probe the leading commercial and open-source Large Language Models including GPT-4. We assess explicability and trustworthiness by a) establishing how well different models engage in moral reasoning and b) comparing normative values underlying models as ethical frameworks. We employ an experimental, evidence-based approach that challenges the models with ethical dilemmas in order to probe human-AI alignment. The ethical scenarios are designed to require a decision in which the particulars of the situation may or may not necessitate deviating from normative ethical principles. A sophisticated ethical framework was consistently elicited in one model, GPT-4. Nonetheless, troubling findings include underlying normative frameworks with clear bias towards particular cultural norms. Many models also exhibit disturbing authoritarian tendencies. https://github.com/jon-chun/llm-sota-chatbots-ethics-based-audit

Keywords: Large Language Models (LLMs), AI Safety, Human-AI Alignment, Ethics Based Audit
(EBA), LLM Reasoning

(Upcoming: LLM Training Dataset Synthesis and Augmentation

.

Research & Patents

My academic interests are in applied ML/AI research projects that impact real-world applications. I have published patents, analysis, and research on AI Large Langauge Models (LLMs) like GPT2 and GPT4, NLP sentiment analysis, electronic medical records, gene therapy, computer security and semiconductor research. I have also given academic and conference presentations on AI and LLMs for Finance, Medicine, Manufacturing, Narratology, Higher Education.

The AI Lab

In addition to my personal academic research interests, much of my time is spent mentoring hundreds of our students’ original research projects for our AI Lab. This can range from brainstorming, architecting technical details, to assisting in implementation based upon my wide-ranging experiences. My experiences across the US and abroad in multiple organizations, roles and industry sectors enables me to share directly relevant insights with all students. Like every good leader, my goal is to identify, connect, and enable students to combine their backgrounds, talents and interests within a deeply meaningful ML/AI project with real-world impact. As of November 2023:

300+ Projects

Since 2017, we have mentored over 300 unique ML/AI research projects. These projects range across virtually every discipline using NLP sentiment analysis, social media, LLM/GPT4 and econometrics.

34k Downloads

After 5 years of publishing our research online we have more than doubled readership annually to 28 thousand. This represents organic growth without any media or pr outreach.

1700 Institutions

Leading academic, commercial and governmental agencies world-wide including Stanford, MIT, Berkeley, Princeton, Oxford, Amazon, Intel, HP, Starbucks and Kate Spade.

150+ Countries

Although the majority of our audience comes from the US and Europe, followers of our research come from all parts of the world from Cameroon to the Solomon Islands.