Keynote
Alan Turing Auditorium
Speaker
Prof José Hernández-Orallo
Jose H. Orallo is Director of Research at the Leverhulme Centre for the Future of Intelligence, University of Cambridge, UK, and Professor (on partial leave) at TU Valencia, Spain. His academic and research activities have spanned several areas of artificial intelligence, machine learning, data science and intelligence measurement, with a focus on a more insightful analysis of the capabilities, generality, progress, impact and risks of artificial intelligence. He has published five books and more than two hundred papers, from specialised conferences such as NeurIPS, ICML and AAAI, to generalistic venues such as Nature and Science. His research in the area of machine intelligence evaluation has been covered by several popular outlets, such as The Economist, WSJ, FT, BBC or New Scientist. He keeps exploring a more integrated view of the evaluation of natural and artificial intelligence, as vindicated in his book "The Measure of All Minds" (Cambridge University Press, 2017, PROSE Award 2018). He is a founder of aievaluation.substack.com and ai-evaluation.org. He is a member of AAAI, CAIRNE and ELLIS, and a EurAI Fellow.
Keynote Talk
Capabilities and Propensities That Explain and Predict AI (mis)Behaviour
Much is being said about the need for a Science of Evaluation in AI, yet the answer may simply be found in what any science should provide: explanatory power to understand what AI systems are capable of, and predictive power to anticipate where they will be correct and safe. For increasingly general and capable AI, this power should not be limited to aggregated tasks, benchmarks, or distributions, but should apply to each task instance, especially out of distribution. I will present a new paradigm in AI evaluation based on general scales that are exclusively derived from task demands and can be applied through both automatable and human-interpretable rubrics. These scales can explain what common AI benchmarks truly measure, extract ability profiles quantifying the limits of what AI systems can do, and predict performance for new task instances robustly. This new paradigm based only on capabilities, however powerful, is not enough. Propensities—tendencies for behaviours, such as personalities or values—should play an increasingly relevant role in explaining and predicting anomalous and dangerous behaviour, critical for AI safety and security. I’ll introduce a simple mathematical framework for propensities that can be used alongside capabilities to understand and anticipate AI (mis)behaviour.
Panel
Alan Turing Auditorium
Panelists
Daniel Mercader Rodríguez - Deputy to the Innovation & Technology Director Agencia Española de Protección de Datos
Simone Fischer-Hübner - Professor of Computer Science at Karlstad University and Chalmers University of Technology, Sweden
Josep Domingo Ferrer - Professor of Computer Science at Universitat Rovira i Virgili, Tarragona
Ruba Abu-Salma - Associate Professor of Computer Science at King's College London
Moderator
Rongjun Ma - Postodctoral Researcher at Valencian Research Institute for Artificial Intelligence (VRAIN), UPV
Workshop presentations
Alan Turing Auditorium
Workshop presetations are either short format (5 min) or long format (10 min), both followed by 5 min Q&A.
WS1: Security and Privacy in Human-AI interactions
Dr Yixin Zou - Max Planck Inst. on Security and Privacy
AI: Good, Evil, or Both? The Interplay Between Public Perceptions of AI and Narratives Around AI (long)
Prof Jose Such - HASP @ INGENIO (CSIC-UPV)
Malicious AI manipulates users to Reveal Personal Information (long)
Dr William Seymour - King's College London
Designing Novel Consent Mechanisms (short)
Dr Rongjun Ma - HASP @ VRAIN (UPV)
Privacy in custom and romantic AI (long)
Abstract
The AI landscape is rapidly evolving, introducing new forms of interaction through customized tools
and emotionally embedded applications. In this talk, I present two studies examining privacy in
emerging LLM-based contexts: custom GPTs and AI romantic partners. Drawing on interviews with GPT
users and creators (N=23) and individuals in AI romantic relationships (N=17), I explore how blurred roles,
shifting data flows, emotional intimacy, and platform affordances reshape privacy perceptions. Across both
contexts, participants navigate unclear boundaries of responsibility, disclosure, and control. Together,
these findings highlight the need to rethink privacy frameworks in an increasingly participatory and relational AI ecosystem.
Dr Sameer Patil - University of Utah
AI-Patient Confidentiality? Unpacking Privacy Aspects of Human-AI Interactions for Mental Health Support (short)
Dr Jennifer Pybus - Canada Research Chair in AI, at York University
Extraction-by-Design: Methods for Auditing Intimate Health Applications (long)
Abstract
This presentation summarises three of my recent publications that
advance distinctive methodological interventions for studying intimate health data routinely monetised through
mobile applications. First, using large language models, we develop an innovative mixed-method audit framework
to examine how baby-tracking and menopause apps collect, process, and circulate sensitive reproductive and
infant data. Second, we translate these audit findings into a participatory HCI intervention by adapting
Voros’s Futures Cone to the context of period-tracking applications, conducting workshops in the UK and
Canada to foster critical engagement with intimate data infrastructures. By pairing technical audits with
futures-oriented participatory methods, we demonstrate an integrated approach that links our empirical research
on mobile tracking infrastructures with collective imagination and data governance reform.
WS2: Security and Privacy in AI Models
Prof Josep Domingo Ferrer - Universitat Rovira i Virgili
Are privacy attacks against machine learning really dangerous? (long)
Abstract
In several jurisdictions, the regulatory framework on the release and sharing of
personal data is being extended to machine learning (ML). The implicit assumption is that
disclosing a trained ML model entails a privacy risk for any personal data used in training
comparable to directly releasing those data. However, given a trained model, it is necessary
to mount a privacy attack to make inferences on the training data. I will
examine the main families of privacy attacks against predictive and generative ML, including
membership inference attacks (MIAs), property inference attacks, and reconstruction attacks.
Most of these attacks seem less effective in the real world than
what a prima facie interpretation of the related literature could suggest.
Abstract
This short talk presents work in progress from the EU Horizon projects TRUMAN on
TRUstworthy and huMAN-centric AI.
Abstract
In this talk I will summarize some of my work related to untrusted data sources during my PhD at Imperial College London and at IBM Research. The first part of the talk will be on indiscriminate data poisoning attacks against supervised learning. I introduce a threat model for poisoning attacks against regression models, and propose a novel optimal stealthy attack formulation against regression models via multiobjective bilevel optimization, where the two objectives are attack effectiveness and detectability. I experimentally show that state-of-the-art defenses do not mitigate these stealthy attacks. In the second part of the talk, I will describe my work on factuality of LLMs, introducing FactReasoner and Granite Guardian. FactReasoner is a novel neuro-symbolic-based factuality assessor that employs probabilistic reasoning to evaluate the truthfulness of long-form generated responses. FactReasoner estimates the posterior probability that each part of the response is supported by the evidence retrieved from external knowledge sources. Our experiments demonstrate that FactReasoner often outperforms state-of-the-art prompt-based methods. Finally, I will also briefly mention Granite Guardian, which is a collection of models and LoRA adapters designed at IBM Research to judge if the input prompts and the output responses of an LLM-based system meet specified risk criteria, such as jailbreak attempts, profanity, factuality, and hallucinations.
WS3: Security and Privacy in AI Systems and Agentic AI
Dr Mark Coté - King's College London
CONTEXT: A Research Programme for Agentic AI (short)
Abstract
CONTEXT is a research programme that examines how the rapid shift from static,
developer-curated APIs to context-aware agentic AI (e.g., MCP-style stacks) is reshaping the privacy and security
surface of everyday and organisational computing, as agents dynamically discover tools, reason across services,
and retain “lived context” over time. We ask how to make this new operational surface safe by design: enforcing
least-privilege access, purpose/time limits, revocable consent, and auditable evidence trails so that agent behaviour
is observable, reconstructable, and accountable to users, engineers, and regulators. We propose a Public Context
Broker —a policy-enforcing intermediary (or a more decentralised solution) that applies least-privilege controls,
purpose/time limits, revocation, and audit-grade evidence trails so agent actions are observable, reconstructable,
and governable. Privacy and security are treated as socio-technical problems, validated through co-designed testbeds
spanning digital health (Type 1 diabetes Looper communities) and cross-service enterprise analytics.
Abstract
We present Compliance-as-Code for agentic LLM systems, turning high-level
policies into testable controls that run continuously as agents evolve. The framework automatically executes
policy-oriented (including tool-use and adversarial) scenarios and returns structured, auditable decisions
linked to concrete evidence such as transcripts, tool traces, and configuration snapshots. This enables reproducible
compliance reports, regression testing, and drift detection across versions, supporting risk-based governance and oversight
of real-world agent ecosystems.
Abstract
Traditional smart toys are interactive toys with IoT features like communication, computation, and sensing. The newest generation of smart toys incorporates generative AI, including chatbots and image generators. While these toys may offer children new options for entertainment and playful education, they come with potential costs to privacy and child safety. In this talk, I will present findings from recently completed and ongoing research on smart AI toys that sheds light on their privacy practices, biases, and child safety. I will also show how privacy-friendly alternatives can be built.
Dr Jide Edu - Strathclyde University
Assessing Security and Compliance Issues: from cloud-based apps to LLMs (long)
Juan Carlos Carrillo - HASP @ VRAIN (UPV)
Personal Data Flows and Privacy Policy Traceability in Third-party GPT Integrations (long)
Abstract
The rapid growth of platforms for customizing Large Language Models (LLMs), such as OpenAI’s GPTs, has raised new privacy and security concerns, particularly related to the exposure of user data via third-party API integrations in LLM apps. To assess privacy risks and data practices, we conducted a large-scale analysis of OpenAI’s GPTs ecosystem. Through the analysis of 5,286 GPTs and the 44,102 parameters they use through API calls to external services, we systematically investigated the types of user data collected, as well as the completeness and discrepancies between actual data flows and GPTs’ stated privacy policies. Our results highlight that approximately 35% of API parameters enable the sharing of sensitive or personally identifiable information, yet only 15% of corresponding privacy policies provide complete disclosure. By quantifying these discrepancies, our study exposes critical privacy risks and underscores the need for stronger oversight and support tools in LLM-based application development. Furthermore, we uncover widespread problematic practices among GPT creators, such as missing or inaccurate privacy policies and a misunderstanding of their privacy responsibilities. Building on these insights, we propose design recommendations that include actionable measurements to improve transparency and informed consent, enhance creator responsibility, and strengthen regulation.
Abstract
AI seems dominated by closed frontier models provided as a service, however, open source is present in AI and is crucial for AI security and safety. There are organizations that belive in a future where istead of big closed models that do everything, there will be many small models doing very specific tasks. Open source will be key in AI and AI security.
Abstract
In this talk, I will introduce my research conducted for the European Data Protection Board and the Council of Europe on privacy and security risks in Large Language Models and emerging agentic AI systems. The work develops a lifecycle-based risk framework to identify and assess model and system level risks, and proposes structured mitigation approaches. I will also briefly present ongoing research at the Dutch Data Protection Authority on agentic AI and chatbots.
WS4: AI for Security and Privacy Applications
Prof Pinar Yolum - Utrecht University
Mitigating Privacy Conflicts with Computational Theory of Mind (long)
Prof Juan Caballero - IMDEA Software
Unsupervised LLM-Based Classification of Victim Abuse Reports (long)
Abstract
Cryptocurrency abuse reporting services are a valuable data source about abusive blockchain addresses, prevalent types of cryptocurrency abuse, and their financial impact on victims. However, they may suffer data pollution due to their crowd-sourced nature. We propose an unsupervised LLM-based classifier to automatically identifying valid reports submitted to abuse reporting services, and classify them into abuse types. Our unsupervised LLM-based classifier clearly outperforms two baselines: a supervised classifier and a naive usage of the LLM. We show how the classifier results can be used for quantifying the financial impact of different cryptocurrency abuse types.
Abstract
I will summarize 2-3 papers on this topic.
