Projects

Automatic evaluation and optimization of hosted AI agents in Microsoft Foundry Agent Service.

Agent Optimizer is a service in Microsoft Foundry Agent Service that automates the agent improvement cycle through a closed-loop process: it evaluates baseline agent performance against defined criteria, generates optimized candidate configurations, tests and ranks candidates by score, and deploys winning configurations with a single command. It supports multiple optimization targets - instructions (rewriting system prompts), skills (generating reusable procedures), models (evaluating across LLM deployments), and tool descriptions (improving function-calling clarity). An 'eval init' command generates test datasets and scoring criteria from existing agent instructions, addressing the cold-start problem, and a portal interface supports browsing optimization runs and comparing candidates. Announced at Microsoft Build 2026 and available to hundreds of thousands of customers on Azure.

An open-source SDK and runtime for building, orchestrating, and deploying AI agents - the unification of Semantic Kernel and AutoGen.

Microsoft Agent Framework is an open-source SDK and runtime (Python and .NET) for building, orchestrating, and deploying AI agents and multi-agent workflows. It unifies Semantic Kernel and AutoGen into a single framework and is integrated into Azure Foundry Agent Service. I led the unification effort and designed the middleware/guardrails architecture that defines interception points for all function, tool, and agent calls across the SDK.

A research prototype of a human-centered web agent with co-planning, co-tasking, and action guards.

Magentic-UI is an open-source research prototype for developing and studying human-agent interaction. Built on a flexible multi-agent architecture, it supports web browsing, code execution, and file manipulation, and can be extended with tools via the Model Context Protocol (MCP). It implements six mechanisms for effective, low-cost human involvement - co-planning, co-tasking, multi-tasking, action guards, and long-term memory - and seeks user approval before executing potentially irreversible actions.

A generalist multi-agent system where an Orchestrator directs specialized agents - WebSurfer, FileSurfer, Coder and ComputerTerminal - to complete complex, multi-step tasks.

Magentic-One is a generalist agentic system for tasks that span the web, files and code. A lead Orchestrator agent plans the work and tracks it in a task ledger and a progress ledger, delegating to four specialized agents: WebSurfer (a Chromium browser), FileSurfer (local files), Coder (writes and analyzes code) and ComputerTerminal (runs programs and installs libraries); the Orchestrator re-plans when progress stalls. On the GAIA, AssistantBench and WebArena benchmarks it performs statistically comparably to the state of the art, and significantly better than GPT-4 alone. Released in November 2024 as a Microsoft Research technical report and open sourced in AutoGen, it has become a reference architecture for autonomous multi-agent systems.

A low-code interface for rapidly building, testing, and sharing multi-agent solutions.

AutoGen Studio is a low-code interface built on AutoGen, enabling developers to rapidly build, test, deploy, and share multi-agent solutions. It provides a user-friendly interface to create and customize agents with little to no coding required. AutoGen Studio allows users to rapidly author agent workflows via a user interface, interactively test and debug agents, reuse artifacts, and deploy workflows. Key features include the ability to choose from pre-defined agents, compose them into teams (workflows), customize agents with foundation models, prompts, and skills, and deploy workflows as APIs. Future plans include a drag-and-drop interface for workflow authoring and a community gallery for sharing workflows, agents, and skills.

Automatic Generation of Grammar-Agnostic Visualizations and Infographics.

Systems that support users in the automatic creation of visualizations must address several subtasks - understand the semantics of data, enumerate relevant visualization goals and generate visualization specifications. In this work, we pose visualization generation as a multi-stage generation problem and argue that well-orchestrated pipelines based on large language models (LLMs) and image generation models (IGMs) are suitable to addressing these tasks. We present LIDA, a novel tool for generating grammar-agnostic visualizations and infographics. LIDA comprises of 4 modules - A SUMMARIZER that converts data into a rich but compact natural language summary, a GOAL EXPLORER that enumerates visualization goals given the data, a VISGENERATOR that generates, refines, executes and filters visualization code and an INFOGRAPHER module that yields data-faithful stylized graphics using IGMs. LIDA provides a python api, and a hybrid user interface (direct manipulation and multilingual natural language) for interactive chart, infographics and data story generation.

UI interface for experimenting with multimodal (text, image) models (stable diffusion).

UI interface for experimenting with multimodal (text, image) models (stable diffusion).

A Deep Learning Library for Automatic Offline Signature Verification Tasks.

Existing studies in automatic signature verification frequently focus on representation extraction while ignoring adjacent, non-trivial tasks such as signature detection, normalization (thresholding, noise removal), representation explanations and scalable verification/matching that are frequently required in practice. In this work, we present the signver library with the goal of enabling researchers and non-technical domain experts efficiently address each of these sub tasks via an intuitive API set. We benchmark SignVer's performance and provide novel insights on the use of skip connection based models (ResNet and UNet) for representation extraction and noise removal tasks. Our experiments also characterize impact of noise removal on overall performance of a representation extraction. We show that SignVer is versatile by demonstrating its application in three use cases - signature based verification, signature based identification and signature based document retrieval. SignVer is open source and available for community use https://github.com/victordibia/signver.

A Usable Library For Question Answering On Large Datasets.

Existing tools for Question Answering (QA) have challenges that limit their use in practice. They can be complex to set up or integrate with existing infrastructure, do not offer configurable interactive interfaces, and do not cover the full set of subtasks that frequently comprise the QA pipeline (query expansion, retrieval, reading, and explanation/sensemaking). To help address these issues, we introduce NeuralQA - a usable library for QA on large datasets. NeuralQA integrates well with existing infrastructure (e.g., ElasticSearch instances and reader models trained with the HuggingFace Transformers API) and offers helpful defaults for QA subtasks. It introduces and implements contextual query expansion (CQE) using a masked language model (MLM) as well as relevant snippets (RelSnip) - a method for condensing large documents into smaller passages that can be speedily processed by a document reader model. Finally, it offers a flexible user interface to support workflows for research explorations (e.g., visualization of gradient-based explanations to support qualitative inspection of model behaviour) and large scale search deployment. Code and documentation for NeuralQA is available as open source on Github.

Visual Exploration of Representations Learned by Convolutional Neural Networks.

This demo allows you to explore visualizations of patterns learned by channels (groups of neurons) in each layer of a Convolutional Neural Network (pretrained on imagenet). To begin, click on a model, and a layer to view visualizations of selected channels in that layer.

Visualize, Train and Evaluate a Deep Autoencoder Model for Anomaly Detection Tasks (Tensorflow.js).

Deep Neural Networks excel at approximating complex mapping functions and have found extensive applications across multiple domains. However, they rely on complex mathematical functions and can be challenging to understand and effectively apply. In this work, we introduce Anomagram - an interactive visualization designed to help users build intuition on how a deep feed forward autoencoder model can be applied to the task of Anomaly Detection. To achieve these goals, Anomagram provides two main modules - an explainer module that provides a discussions of concepts, paired with interactive visualizations, and a extit{trainer} module where users can build, train and evaluate an autoencoder from scratch. Anomagram integrates a direct manipulation model composer interface for specifying the autoencoder, and provides interactive charts that illustrate model performance (loss, evaluation metrics, histogram of errors, ROC curve, internal state dimensions) as training progresses. Anomagram addresses the lack of realism that frequently limits interactive learning tools by integrating a real world dataset (ECG5000). To further aid the learning process, it also allows the user to craft specialized (adversarial) input and observe performance. Anomagram runs entirely in the browser with no installation required, which makes it an accessible learning tool to any user with a modern web browser

Semantic (content-based) image search with pretrained Deep Neural Networks.

Recent advances in the design of Convolutional Neural Network models (CNNs or ConvNets) have made them the extit{de facto} tool for solving hard image analysis problems. However, due to their complexity, developers with limited experience in machine learning still face challenges building up intuition required to successfully apply them. Further, the current landscape of interactive tools for supporting ML education focus on mathematical concepts and utilize artificial data scenarios which do not translate well to real world problems. In this work, we introduce ConvNet Playground, an interactive visualization tool for exploring ConvNets applied to a concrete task - semantic image search on real world datasets. ConvNet Playground offers multiple approaches for the user to make sense of model behaviour - first evaluating the impact of developer decision choices and model properties (e.g. model architecture, sub model architecture, model size, distance metric etc.) on search performance and secondly a visual exploration of the patterns learned within each model architecture.

A library for prototyping realtime hand detection (bounding box), directly in the browser.

A while ago, I was really blown away by results from an experiment using TensorFlow object detection api to track hands in an image. I made the trained model and source code available, and since then it has been used to prototype some rather interesting usecases (a tool to help kids spell, extensions to predict sign language, hand ping pong, etc). However, while many individuals wanted to experiment with the trained model, a large number still had issues setting up Tensorflow (installation, TF version issues, exporting graphs, etc). Luckily, Tensorflow.js addresses several of these installations/distribution issues, as it is optimized to run in the standardized environment of browsers. To this end, I created Handtrack.js as a library to allow developers quickly prototype hand/gesture interactions powered by a trained hand detection model.

Automatic Generation of Data Visualizations Using Sequence-to-Sequence Recurrent Neural Networks.

Rapidly creating effective visualizations using expressive grammars is challenging for users who have limited time and limited skills in statistics and data visualization. Even high-level, dedicated visualization tools often require users to manually select among data attributes, decide which transformations to apply, and specify mappings between visual encoding variables and raw or transformed attributes. In this paper we introduce Data2Vis, a neural translation model for automatically generating visualizations from given datasets. We formulate visualization generation as a sequence to sequence translation problem where data specifications are mapped to visualization specifications in a declarative language (Vega-Lite). To this end, we train a multilayered attention-based recurrent neural network (RNN) with long short-term memory (LSTM) units on a corpus of visualization specifications. Qualitative results show that our model learns the vocabulary and syntax for a valid visualization specification, appropriate transformations (count, bins, mean) and how to use common data selection patterns that occur within data visualizations. Data2Vis generates visualizations that are comparable to manually-created visualizations in a fraction of the time, with potential to learn more complex visualization strategies at scale.

Experiments in generating novel Africa Mask Art with Generative Adversarial Networks.

This work highlights interpretations of African Mask Art developed using a two-stage generative adversarial network (GAN). GANs are a class of deep neural networks (a subset of the general field of artificial intelligence) which can be used to model the properties of a given dataset and leverage this knowledge in recreating samples that are distinct but similar to the original dataset. In this work, the I first curates a dataset of 11,000 African mask art images, representing work dominantly from West, Central and South Africa. Next, a range of experiments are conducted where parameters are selected and used to train a GAN (learning to model the properties of images in the African Mask Art datast). The resulting GAN is then used to generate a set of novel images. These images are further curated, and a second stage GAN (superresolution) is applied in upsampling the final collection..

A maker kit (chipboard, raspberry pi, sensors) for prototyping with machine learning services.

TJBot is an open source, interactive robot designed to encourage people to build with cognitive services in a fun way. He is a paper robot, which can also be 3D printed, and comes with an initial set of recipes that bring him to life. Recipes are a combination of step-by-step instructions plus sample code that walk people through the assembly of the robot, its hardware components, and software code that connects him to Watson cognitive services. TJBot can be programmed to listen, speak, see and recognize, shine his LED, understand emotions, and wave his arm. TJBot was designed for two communities: makers, who enjoy the DIY aspects of building and programming novel devices, and students, who can learn about programming systems that integrate machine learning.

Wearable app to improve mental health through mindful meditation and focus techniques.

Foqus, is a smartwatch app designed to aid adults with mental health conditions like ADHD and mild forms of attention deficiency through two main routes – tools to foster extended focus and tools to reduce anxiety/stress. Using a user-centric design approach, three important features are identified, implemented and evaluated which aim to leverage the benefits of wearable devices: a flexible implementation of the Pomodoro time management technique, a tool for guided meditation, and positive message priming. Initial user test results suggest smartwatch-based interventions as a viable, ubiquitous tool for addressing mental health and stress related conditions.