Publications
For the most updated list of my publications, check my Google Scholar page.
2026
- ECCV Art GalleryAn Image Worth a Thousand PaintingsGeorge Cazenavette, Angelina Quan, Giannis Daras, Antonio Torralba, and Vincent SitzmannIn ECCV 2026 Art Gallery
Award Winner and Spotlight — Art Gallery, ECCV 2026
Dataset distillation synthesizes a small set of images that best approximate a real data distribution. We introduce a "drifting" method, building on the generative framework of Deng et al., that distills an artist’s entire body of work into a single image.
@inproceedings{thousand_paintings, title = {An Image Worth a Thousand Paintings}, author = {Cazenavette, George and Quan, Angelina and Daras, Giannis and Torralba, Antonio and Sitzmann, Vincent}, booktitle = {ECCV 2026 Art Gallery}, year = {2026}, } - CoRL RSS WorkshopAmbient Diffusion Policy: Imitation Learning from Suboptimal Data in RoboticsAdam Wei, Nicholas Pfaff, Thomas Cohn, Arif Kerem Dayı, Constantinos Daskalakis, Giannis Daras, and Russ TedrakeIn CoRL 2026 and RSS 2026 Workshops
Best Paper Award — Data-Centric Robotics Workshop, RSS 2026
Most Useful Practical Information Award — “It’s the Demos” Workshop, RSS 2026We propose Ambient Diffusion Policy, a simple and principled method for imitation learning from suboptimal data in robotics. High-quality, task-specific robot data is expensive and time-consuming to collect, while suboptimal datasets with lower-quality or out-of-distribution demonstrations are abundant. Existing methods that co-train on both data sources in robotics often fail to separate the meaningful and the harmful features in the suboptimal samples. In contrast, our method extracts only the useful features by introducing a new axis to co-training in robotics: noise-dependent data usage. Ambient Diffusion Policy restricts the contribution of suboptimal data during training to only the high and low diffusion times. To rigorously justify our approach, we first observe that robot action data exhibits a spectral power law. This induces two important properties on the optimal Diffusion Policy that we exploit: a global-to-local hierarchy and locality. We theoretically formalize this discussion using a simplified model. Our experiments validate Ambient Diffusion Policy on four types of suboptimal action data (noisy trajectories, sim-to-real gap, task mismatch, and large-scale data mixtures) across six tasks. The results show that it effectively learns from arbitrary sources of suboptimal data. Notably, it outperforms existing co-training baselines by up to 33% when scaled to Open X-Embodiment - a large dataset with heterogeneous data quality and unstructured distribution shifts. Overall, Ambient Diffusion Policy increases the utility of suboptimal demonstrations and expands the set of usable data sources in robotics.
@inproceedings{ambient_diffusion_policy, title = {Ambient Diffusion Policy: Imitation Learning from Suboptimal Data in Robotics}, author = {Wei, Adam and Pfaff, Nicholas and Cohn, Thomas and Dayı, Arif Kerem and Daskalakis, Constantinos and Daras, Giannis and Tedrake, Russ}, booktitle = {CoRL 2026 and RSS 2026 Workshops}, year = {2026}, } - ECDiffusion Based Auctions for Banner AdsLillian Sun, Henry Huang, Fucheng Warren Zhu, Giannis Daras, and Constantinos DaskalakisIn EC 2026
We propose auctions that fractionally allocate the creation of a banner to bidders according to their preferences. The mechanism elicits bids and textual prompts from advertisers, and composes them into a score function that drives a reverse diffusion process that generates the banner. It implements Monte Carlo sampling to calculate approximate VCG-based payments to incentivize high-welfare images. Extensive experiments on a diverse 20-prompt dataset with up to 3 agents demonstrate key economic properties: bid monotonicity, an efficiency improvement of up to 21% higher welfare than a single-winner VCG baseline, and approximate incentive compatibility, with average regret below 10% when deviating from truthful bidding, while preserving high image quality.
@inproceedings{diffusion_auctions_banner_ads, title = {Diffusion Based Auctions for Banner Ads}, author = {Sun, Lillian and Huang, Henry and Zhu, Fucheng Warren and Daras, Giannis and Daskalakis, Constantinos}, booktitle = {EC 2026}, year = {2026}, } - PreprintAmbient Physics: Training Neural PDE Solvers with Partial ObservationsHarris Abdul Majid, Giannis Daras, Francesco Tudisco, and Steven McDonagh2026
In many scientific settings, acquiring complete observations of PDE coefficients and solutions can be expensive, hazardous, or impossible. Recent diffusion-based methods can reconstruct fields given partial observations, but require complete observations for training. We introduce Ambient Physics, a framework for learning the joint distribution of coefficient-solution pairs directly from partial observations, without requiring a single complete observation. The key idea is to randomly mask a subset of already-observed measurements and supervise on them, so the model cannot distinguish "truly unobserved" from "artificially unobserved", and must produce plausible predictions everywhere. Ambient Physics achieves state-of-the-art reconstruction performance. Compared with prior diffusion-based methods, it achieves a 62.51% reduction in average overall error while using 125× fewer function evaluations. We also identify a "one-point transition": masking a single already-observed point enables learning from partial observations across architectures and measurement patterns. Ambient Physics thus enables scientific progress in settings where complete observations are unavailable.
@misc{ambient_physics, title = {Ambient Physics: Training Neural PDE Solvers with Partial Observations}, author = {Majid, Harris Abdul and Daras, Giannis and Tudisco, Francesco and McDonagh, Steven}, year = {2026}, } - ICMLAmbient Dataloops: Generative Models for Dataset RefinementAdrián Rodríguez-Muñoz, William Daspit, Adam Klivans, Antonio Torralba, Constantinos Daskalakis, and Giannis DarasIn ICML 2026
We propose Ambient Dataloops, an iterative framework for refining datasets that makes it easier for diffusion models to learn the underlying data distribution. Modern datasets contain samples of highly varying quality, and training directly on such heterogeneous data often yields suboptimal models. We propose a dataset-model co-evolution process; at each iteration of our method, the dataset becomes progressively higher quality, and the model improves accordingly. To avoid destructive self-consuming loops, at each generation, we treat the synthetically improved samples as noisy, but at a slightly lower noisy level than the previous iteration, and we use Ambient Diffusion techniques for learning under corruption. Empirically, Ambient Dataloops achieve state-of-the-art performance in unconditional and text-conditional image generation and de novo protein design. We further provide a theoretical justification for the proposed framework that captures the benefits of the data looping procedure.
@inproceedings{ambient_dataloops, title = {Ambient Dataloops: Generative Models for Dataset Refinement}, author = {Rodríguez-Muñoz, Adrián and Daspit, William and Klivans, Adam and Torralba, Antonio and Daskalakis, Constantinos and Daras, Giannis}, booktitle = {ICML 2026}, year = {2026}, } - PreprintDiffEM: Learning from Corrupted Data with Diffusion Models via Expectation MaximizationDanial Hosseintabar, Fan Chen, Giannis Daras, Antonio Torralba, and Constantinos Daskalakis2026
Diffusion models have emerged as powerful generative priors for high-dimensional inverse problems, yet learning them when only corrupted or noisy observations are available remains challenging. In this work, we propose a new method for training diffusion models with Expectation-Maximization (EM) from corrupted data. Our proposed method, DiffEM, utilizes conditional diffusion models to reconstruct clean data from observations in the E-step, and then uses the reconstructed data to refine the conditional diffusion model in the M-step. Theoretically, we provide monotonic convergence guarantees for the DiffEM iteration, assuming appropriate statistical conditions. We demonstrate the effectiveness of our approach through experiments on various image reconstruction tasks.
@misc{diff_em, title = {DiffEM: Learning from Corrupted Data with Diffusion Models via Expectation Maximization}, author = {Hosseintabar, Danial and Chen, Fan and Daras, Giannis and Torralba, Antonio and Daskalakis, Constantinos}, year = {2026}, } - NeurIPSAlign & Invert: Solving Inverse Problems with Diffusion and Flow-based Models via Representation AlignmentLoukas Sfountouris, Giannis Daras, and Paris GiampourasIn NeurIPS 2026
Enforcing alignment between the internal representations of diffusion or flow-based generative models and those of pretrained self-supervised encoders has recently been shown to provide a powerful inductive bias, improving both convergence and sample quality. In this work, we extend this idea to inverse problems, where pretrained generative models are employed as priors. We propose applying representation alignment (REPA) between diffusion or flow-based models and a DINOv2 visual encoder, to guide the reconstruction process at inference time. Although ground-truth signals are unavailable in inverse problems, we empirically show that aligning model representations of approximate target features can substantially enhance reconstruction quality and perceptual realism. We provide theoretical results showing (a) that REPA regularization can be viewed as a variational approach for minimizing a divergence measure in the DINOv2 embedding space, and (b) how under certain regularity assumptions REPA updates steer the latent diffusion states toward those of the clean image. These results offer insights into the role of REPA in improving perceptual fidelity. We integrate REPA into multiple state-of-the-art inverse problem solvers, and provide extensive experiments on super-resolution, box inpainting, Gaussian deblurring, and motion deblurring confirming that our method consistently improves reconstruction quality, while also providing efficiency gains reducing the number of required discretization steps.
@inproceedings{align_and_invert, title = {Align \& Invert: Solving Inverse Problems with Diffusion and Flow-based Models via Representation Alignment}, author = {Sfountouris, Loukas and Daras, Giannis and Giampouras, Paris}, booktitle = {NeurIPS 2026}, year = {2026}, }
2025
- NeurIPSAmbient Diffusion Omni: Training Good Models with Bad DataGiannis Daras*, Adrian Rodriguez-Munoz*, Adam Klivans, Antonio Torralba, and Constantinos DaskalakisIn NeurIPS 2025
Spotlight at NeurIPS 2025
We show how to use low-quality, synthetic, and out-of-distribution images to improve the quality of a diffusion model. Typically, diffusion models are trained on curated datasets that emerge from highly filtered data pools from the Web and other sources. We show that there is immense value in the lower-quality images that are often discarded. We present Ambient Diffusion Omni, a simple, principled framework to train diffusion models that can extract signal from all available images during training. Our framework exploits two properties of natural images — spectral power law decay and locality. We first validate our framework by successfully training diffusion models with images synthetically corrupted by Gaussian blur, JPEG compression, and motion blur. We then use our framework to achieve state-of-the-art ImageNet FID, and we show significant improvements in both image quality and diversity for text-to-image generative modeling. The core insight is that noise dampens the initial skew between the desired high-quality distribution and the mixed distribution we actually observe. We provide rigorous theoretical justification for our approach by analyzing the trade-off between learning from biased data versus limited unbiased data across diffusion times.
@inproceedings{ambient_omni, title = {Ambient Diffusion Omni: Training Good Models with Bad Data}, author = {Daras, Giannis and Rodriguez-Munoz, Adrian and Klivans, Adam and Torralba, Antonio and Daskalakis, Constantinos}, booktitle = {NeurIPS 2025}, year = {2025}, page = {/publication/ambient_omni/}, } - NeurIPSAmbient Proteins: Training Diffusion Models on Low Quality StructuresGiannis Daras*, Jeffrey Ouyang-Zhang*, Krithika Ravishankar, William Daspit, Costis Daskalakis, Qiang Liu, Adam Klivans, and 1 more authorIn NeurIPS 2025
Spotlight at NeurIPS 2025
We present Ambient Protein Diffusion, a framework for training protein diffusion models that generates structures with unprecedented diversity and quality. State-of-the-art generative models are trained on computationally derived structures from AlphaFold2 (AF), as experimentally determined structures are relatively scarce. The resulting models are therefore limited by the quality of synthetic datasets. Since the accuracy of AF predictions degrades with increasing protein length and complexity, de novo generation of long, complex proteins remains challenging. Ambient Protein Diffusion overcomes this problem by treating low-confidence AF structures as corrupted data. Rather than simply filtering out low-quality AF structures, our method adjusts the diffusion objective for each structure based on its corruption level, allowing the model to learn from both high and low quality structures. Empirically, Ambient Protein Diffusion yields major improvements: on proteins with 700 residues, diversity increases from 45% to 86% from the previous state-of-the-art, and designability improves from 68% to 86%. We will make all of our code, models and datasets available under the following repository: https://github.com/jozhang97/ambient-proteins.
@inproceedings{ambient_proteins, title = {Ambient Proteins: Training Diffusion Models on Low Quality Structures}, author = {Daras, Giannis and Ouyang-Zhang, Jeffrey and Ravishankar, Krithika and Daspit, William and Daskalakis, Costis and Liu, Qiang and Klivans, Adam and Diaz, Daniel J.}, booktitle = {NeurIPS 2025}, year = {2025}, page = {/publication/ambient_proteins/}, } - ICLRHow much is a noisy image worth? Data Scaling Laws for Ambient DiffusionGiannis Daras, Yeshwanth Cherapanamjeri, and Constantinos DaskalakisIn ICLR 2025
The quality of generative models depends on the quality of the data they are trained on. Creating large-scale, high-quality datasets is often expensive and sometimes impossible, e.g. in certain scientific applications where there is no access to clean data due to physical or instrumentation constraints. Ambient Diffusion and related frameworks train diffusion models with solely corrupted data (which are usually cheaper to acquire) but ambient models significantly underperform models trained on clean data. We study this phenomenon at scale by training more than 80 models on data with different corruption levels across three datasets ranging from 30,000 to ≈1.3M samples. We show that it is impossible, at these sample sizes, to match the performance of models trained on clean data when only training on noisy data. Yet, a combination of a small set of clean data (e.g. 10% of the total dataset) and a large set of highly noisy data suffices to reach the performance of models trained solely on similar-size datasets of clean data, and in particular to achieve near state-of-the-art performance. We provide theoretical evidence for our findings by developing novel sample complexity bounds for learning from Gaussian Mixtures with heterogeneous variances. Our theoretical model suggests that, for large enough datasets, the effective marginal utility of a noisy sample is exponentially worse than that of a clean sample. Providing a small set of clean samples can significantly reduce the sample size requirements for noisy data, as we also observe in our experiments.
@inproceedings{image_worth, title = {How much is a noisy image worth? Data Scaling Laws for Ambient Diffusion}, author = {Daras, Giannis and Cherapanamjeri, Yeshwanth and Daskalakis, Constantinos}, booktitle = {ICLR 2025}, year = {2025}, } - NeurIPS Creative AIOpt-In Art: Learning Art Styles Only from Few ExamplesHui Ren*, Joanna Materzynska*, Rohit Gandikota, Giannis Daras, David Bau, and Antonio TorralbaIn NeurIPS 2025 Creative AI Track
We explore whether pre-training on datasets with paintings is necessary for a model to learn an artistic style with only a few examples. To investigate this, we train a text-to-image model exclusively on photographs, without access to any painting-related content. We show that it is possible to adapt a model that is trained without paintings to an artistic style, given only few examples. User studies and automatic evaluations confirm that our model (post-adaptation) performs on par with state-of-the-art models trained on massive datasets that contain artistic content like paintings, drawings or illustrations. Finally, using data attribution techniques, we analyze how both artistic and non-artistic datasets contribute to generating artistic-style images. Surprisingly, our findings suggest that high-quality artistic outputs can be achieved without prior exposure to artistic data, indicating that artistic style generation can occur in a controlled, opt-in manner using only a limited, carefully selected set of training examples.
@inproceedings{opt_in_art, title = {Opt-In Art: Learning Art Styles Only from Few Examples}, author = {Ren, Hui and Materzynska, Joanna and Gandikota, Rohit and Daras, Giannis and Bau, David and Torralba, Antonio}, booktitle = {NeurIPS 2025 Creative AI Track}, year = {2025}, } - ICLRInfilling Score: A Pretraining Data Detection Algorithm for Large Language ModelsNegin Raoof, Litu Rout, Giannis Daras, Sujay Sanghavi, Constantine Caramanis, Sanjay Shakkottai, and Alexandros G. DimakisIn ICLR 2025
In pretraining data detection, the goal is to detect whether a given sentence is in the dataset used for training a Large Language Model LLM). Recent methods (such as Min-K % and Min-K%++) reveal that most training corpora are likely contaminated with both sensitive content and evaluation benchmarks, leading to inflated test set performance. These methods sometimes fail to detect samples from the pretraining data, primarily because they depend on statistics composed of causal token likelihoods. We introduce Infilling Score, a new test-statistic based on non-causal token likelihoods. Infilling Score can be computed for autoregressive models without re-training using Bayes rule. A naive application of Bayes rule scales linearly with the vocabulary size. However, we propose a ratio test-statistic whose computation is invariant to vocabulary size. Empirically, our method achieves a significant accuracy gain over state-of-the-art methods including Min-K%, and Min-K%++ on the WikiMIA benchmark across seven models with different parameter sizes. Further, we achieve higher AUC compared to reference-free methods on the challenging MIMIR benchmark. Finally, we create a benchmark dataset consisting of recent data sources published after the release of Llama-3; this benchmark provides a statistical baseline to indicate potential corpora used for Llama-3 training.
@inproceedings{infill_score, title = {Infilling Score: A Pretraining Data Detection Algorithm for Large Language Models}, author = {Raoof, Negin and Rout, Litu and Daras, Giannis and Sanghavi, Sujay and Caramanis, Constantine and Shakkottai, Sanjay and Dimakis, Alexandros G.}, booktitle = {ICLR 2025}, year = {2025}, } - ICLRAmbient Diffusion Posterior Sampling: Solving Inverse Problems with Diffusion Models trained on Corrupted DataAsad Aali, Giannis Daras, Brett Levac, Sidharth Kumar, Alexandros G. Dimakis, and Jonathan I. TamirIn ICLR 2025
We provide a framework for solving inverse problems with diffusion models learned from linearly corrupted data. Our method, Ambient Diffusion Posterior Sampling (A-DPS), leverages a generative model pre-trained on one type of corruption (e.g. image inpainting) to perform posterior sampling conditioned on measurements from a potentially different forward process (e.g. image blurring). We test the efficacy of our approach on standard natural image datasets (CelebA, FFHQ, and AFHQ) and we show that A-DPS can sometimes outperform models trained on clean data for several image restoration tasks in both speed and performance. We further extend the Ambient Diffusion framework to train MRI models with access only to Fourier subsampled multi-coil MRI measurements at various acceleration factors (R = 2, 4, 6, 8). We again observe that models trained on highly subsampled data are better priors for solving inverse problems in the high acceleration regime than models trained on fully sampled data.
@inproceedings{ambient_dps, title = {Ambient Diffusion Posterior Sampling: Solving Inverse Problems with Diffusion Models trained on Corrupted Data}, author = {Aali, Asad and Daras, Giannis and Levac, Brett and Kumar, Sidharth and Dimakis, Alexandros G. and Tamir, Jonathan I.}, booktitle = {ICLR 2025}, year = {2025}, }
2024
- NeurIPS D&BDataComp-LM: In search of the next generation of training sets for language modelsJeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, and 52 more authorsIn NeurIPS 2024 Track Datasets and Benchmarks
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters. As a baseline for DCLM, we conduct extensive experiments and find that model-based filtering is key to assembling a high-quality training set. The resulting dataset, DCLM-BASELINE, enables training a 7B parameter language model from scratch to 64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the previous state-of-the-art in open-data language models, DCLM-BASELINE represents a 6.6 percentage point improvement on MMLU while being trained with 40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and Llama 3 8B on MMLU (63% & 66%), and performs similarly on an average of 53 natural language understanding tasks while being trained with 6.6× less compute than Llama 3 8B. Our results highlight the importance of dataset design for training language models and offer a starting point for further research on data curation.
@inproceedings{dclm, title = {DataComp-LM: In search of the next generation of training sets for language models}, author = {Li, Jeffrey and Fang, Alex and Smyrnis, Georgios and Ivgi, Maor and Jordan, Matt and Gadre, Samir and Bansal, Hritik and Guha, Etash and Keh, Sedrick and Arora, Kushal and Garg, Saurabh and Xin, Rui and Muennighoff, Niklas and Heckel, Reinhard and Mercat, Jean and Chen, Mayee and Gururangan, Suchin and Wortsman, Mitchell and Albalak, Alon and Bitton, Yonatan and Nezhurina, Marianna and Abbas, Amro and Hsieh, Cheng-Yu and Ghosh, Dhruba and Gardner, Josh and Kilian, Maciej and Zhang, Hanlin and Shao, Rulin and Pratt, Sarah and Sanyal, Sunny and Ilharco, Gabriel and Daras, Giannis and Marathe, Kalyani and Gokaslan, Aaron and Zhang, Jieyu and Chandu, Khyathi and Nguyen, Thao and Vasiljevic, Igor and Kakade, Sham and Song, Shuran and Sanghavi, Sujay and Faghri, Fartash and Oh, Sewoong and Zettlemoyer, Luke and Lo, Kyle and El-Nouby, Alaaeldin and Pouransari, Hadi and Toshev, Alexander and Wang, Stephanie and Groeneveld, Dirk and Soldaini, Luca and Koh, Pang Wei and Jitsev, Jenia and Kollar, Thomas and Dimakis, Alexandros G. and Carmon, Yair and Dave, Achal and Schmidt, Ludwig and Shankar, Vaishaal}, booktitle = {NeurIPS 2024 Track Datasets and Benchmarks}, year = {2024}, } - ICMLConsistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy DataGiannis Daras, Alexandros G. Dimakis, and Constantinos DaskalakisIn ICML 2024
Ambient diffusion is a recently proposed framework for training diffusion models using corrupted data. Both Ambient Diffusion and alternative SURE-based approaches for learning diffusion models from corrupted data resort to approximations which deteriorate performance. We present the first framework for training diffusion models that provably sample from the uncorrupted distribution given only noisy training data, solving an open problem in this space. Our key technical contribution is a method that uses a double application of Tweedie’s formula and a consistency loss function that allows us to extend sampling at noise levels below the observed data noise. We also provide further evidence that diffusion models memorize from their training sets by identifying extremely corrupted images that are almost perfectly reconstructed, raising copyright and privacy concerns. Our method for training using corrupted samples can be used to mitigate this problem. We demonstrate this by fine-tuning Stable Diffusion XL to generate samples from a distribution using only noisy samples. Our framework reduces the amount of memorization of the fine-tuning dataset, while maintaining competitive performance.
@inproceedings{ambient_tweedie, title = {Consistent Diffusion Meets Tweedie: Training Exact Ambient Diffusion Models with Noisy Data}, author = {Daras, Giannis and Dimakis, Alexandros G. and Daskalakis, Constantinos}, booktitle = {ICML 2024}, year = {2024}, }
2023
- NeurIPSSolving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion ModelsLitu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alexandros G. Dimakis, and Sanjay ShakkottaiIn NeurIPS 2023
We present the first framework to solve linear inverse problems leveraging pre-trained latent diffusion models. Previously proposed algorithms (such as DPS and DDRM) only apply to pixel-space diffusion models. We theoretically analyze our algorithm showing provable sample recovery in a linear model setting. The algorithmic insight obtained from our analysis extends to more general settings often considered in practice. Experimentally, we outperform previously proposed posterior sampling algorithms in a wide variety of problems including random inpainting, block inpainting, denoising, deblurring, destriping, and super-resolution.
@inproceedings{psld, title = {Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models}, author = {Rout, Litu and Raoof, Negin and Daras, Giannis and Caramanis, Constantine and Dimakis, Alexandros G. and Shakkottai, Sanjay}, booktitle = {NeurIPS 2023}, year = {2023}, } - NeurIPSAmbient Diffusion: Learning Clean Distributions from Corrupted DataGiannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alexandros G. Dimakis, and Adam KlivansIn NeurIPS 2023
We present the first diffusion-based framework that can learn an unknown distribution using only highly-corrupted samples. This problem arises in scientific applications where access to uncorrupted samples is impossible or expensive to acquire. Another benefit of our approach is the ability to train generative models that do not memorize any individual training sample, since they never observe clean training data. Our main idea is to introduce additional measurement distortion during the diffusion process and require the model to predict the original corrupted image from the further corrupted image. We prove that our method leads to models that learn the conditional expectation of the full uncorrupted image given this additional measurement corruption. This holds for any corruption process that satisfies some technical conditions (and in particular includes inpainting and compressed sensing). We train models on standard benchmarks (CelebA, CIFAR-10 and AFHQ) and show that we can learn the distribution even when all the training samples have 90% of their pixels missing. We also show that when we finetune foundation models using small corrupted datasets (e.g. MRI scans with block corruptions), we learn the clean distribution without memorizing the training set.
@inproceedings{ambient, title = {Ambient Diffusion: Learning Clean Distributions from Corrupted Data}, author = {Daras, Giannis and Shah, Kulin and Dagan, Yuval and Gollakota, Aravind and Dimakis, Alexandros G. and Klivans, Adam}, booktitle = {NeurIPS 2023}, year = {2023}, } - NeurIPSDataComp: In search of the next generation of multimodal datasetsSamir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, and 27 more authorsIn NeurIPS 2023
Oral at NeurIPS 2023
Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms. To address this shortcoming in the ML ecosystem, we introduce DataComp, a testbed for dataset experiments centered around a new candidate pool of 12.8 billion image-text pairs from Common Crawl. Participants in our benchmark design new filtering techniques or curate new data sources and then evaluate their new dataset by running our standardized CLIP training code and testing the resulting model on 38 downstream test sets. Our benchmark consists of multiple compute scales spanning four orders of magnitude, which enables the study of scaling trends and makes the benchmark accessible to researchers with varying resources. Our baseline experiments show that the DataComp workflow leads to better training sets. In particular, our best baseline, DataComp-1B, enables training a CLIP ViT-L/14 from scratch to 79.2% zero-shot accuracy on ImageNet, outperforming OpenAI’s CLIP ViT-L/14 by 3.7 percentage points while using the same training procedure and compute.
@inproceedings{datacomp, title = {DataComp: In search of the next generation of multimodal datasets}, author = {Gadre, Samir Yitzhak and Ilharco, Gabriel and Fang, Alex and Hayase, Jonathan and Smyrnis, Georgios and Nguyen, Thao and Marten, Ryan and Wortsman, Mitchell and Ghosh, Dhruba and Zhang, Jieyu and Orgad, Eyal and Entezari, Rahim and Daras, Giannis and Pratt, Sarah and Ramanujan, Vivek and Bitton, Yonatan and Marathe, Kalyani and Mussmann, Stephen and Vencu, Richard and Cherti, Mehdi and Krishna, Ranjay and Koh, Pang Wei and Saukh, Olga and Ratner, Alexander and Song, Shuran and Hajishirzi, Hannaneh and Farhadi, Ali and Beaumont, Romain and Oh, Sewoong and Dimakis, Alex and Jitsev, Jenia and Carmon, Yair and Shankar, Vaishaal and Schmidt, Ludwig}, booktitle = {NeurIPS 2023}, year = {2023}, } - ICMLRestoration-Degradation Beyond Linear Diffusions: A Non-Asymptotic Analysis for DDIM-Type SamplersSitan Chen, Giannis Daras, and Alexandros G. DimakisIn ICML 2023
We develop a framework for non-asymptotic analysis of deterministic samplers used for diffusion generative modeling. Several recent works have analyzed stochastic samplers using tools like Girsanov’s theorem and a chain rule variant of the interpolation argument. Unfortunately, these techniques give vacuous bounds when applied to deterministic samplers. We give a new operational interpretation for deterministic sampling by showing that one step along the probability flow ODE can be expressed as two steps: 1) a restoration step that runs gradient ascent on the conditional log-likelihood at some infinitesimally previous time, and 2) a degradation step that runs the forward process using noise pointing back towards the current iterate. This perspective allows us to extend denoising diffusion implicit models to general, non-linear forward processes. We then develop the first polynomial convergence bounds for these samplers under mild conditions on the data distribution.
@inproceedings{restoration_degradation, title = {Restoration-Degradation Beyond Linear Diffusions: A Non-Asymptotic Analysis for DDIM-Type Samplers}, author = {Chen, Sitan and Daras, Giannis and Dimakis, Alexandros G.}, booktitle = {ICML 2023}, year = {2023}, } - NeurIPSConsistent Diffusion Models: Mitigating Sampling Drift by Learning to be ConsistentGiannis Daras, Yuval Dagan, Alexandros G. Dimakis, and Constantinos DaskalakisIn NeurIPS 2023
Imperfect score-matching leads to a shift between the training and the sampling distribution of diffusion models. Due to the recursive nature of the generation process, errors in previous steps yield sampling iterates that drift away from the training distribution. Yet, the standard training objective via Denoising Score Matching (DSM) is only designed to optimize over non-drifted data. To train on drifted data, we propose to enforce a consistency property which states that predictions of the model on its own generated data are consistent across time. Theoretically, we show that if the score is learned perfectly on some non-drifted points (via DSM) and if the consistency property is enforced everywhere, then the score is learned accurately everywhere. Empirically we show that our novel training objective yields state-of-the-art results for conditional and unconditional generation in CIFAR-10 and baseline improvements in AFHQ and FFHQ.
@inproceedings{cdm, title = {Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent}, author = {Daras, Giannis and Dagan, Yuval and Dimakis, Alexandros G. and Daskalakis, Constantinos}, booktitle = {NeurIPS 2023}, year = {2023}, } - TMLRSoft Diffusion: Score Matching for General CorruptionsGiannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G. Dimakis, and Peyman MilanfarTMLR 2023
We define a broader family of corruption processes that generalizes previously known diffusion models. To reverse these general diffusions, we propose a new objective called Soft Score Matching that provably learns the score function for any linear corruption process and yields state of the art results for CelebA. Soft Score Matching incorporates the degradation process in the network. Our new loss trains the model to predict a clean image, that after corruption, matches the diffused observation. We show that our objective learns the gradient of the likelihood under suitable regularity conditions for a family of corruption processes. We further develop a principled way to select the corruption levels for general diffusion processes and a novel sampling method that we call Momentum Sampler. We show experimentally that our framework works for general linear corruption processes, such as Gaussian blur and masking. We achieve state-of-the-art FID score 1.85 on CelebA-64, outperforming all previous linear diffusion models. We also show significant computational benefits compared to vanilla denoising diffusion.
@article{soft_diffusion, title = {Soft Diffusion: Score Matching for General Corruptions}, author = {Daras, Giannis and Delbracio, Mauricio and Talebi, Hossein and Dimakis, Alexandros G. and Milanfar, Peyman}, journal = {TMLR 2023}, year = {2023}, }
2022
- NeurIPS WorkshopMultiresolution Textual InversionGiannis Daras and Alexandros G. DimakisIn NeurIPS 2022, SBM Workshop
Oral at NeurIPS 2022, SBM Workshop
We extend Textual Inversion to learn pseudo-words that represent a concept at different resolutions. This allows us to generate images that use the concept with different levels of detail and also to manipulate different resolutions using language. Once learned, the user can generate images at different levels of agreement to the original concept; "A photo of S^(0)" produces the exact object while the prompt "A photo of S^(0.8)" only matches the rough outlines and colors. Our framework allows us to generate images that use different resolutions of an image (e.g. details, textures, styles) as separate pseudo-words that can be composed in various ways. We open-soure our code in the following URL: https://github.com/giannisdaras/multires_textual_inversion
@inproceedings{multires_textual_inversion, title = {Multiresolution Textual Inversion}, author = {Daras, Giannis and Dimakis, Alexandros G.}, booktitle = {NeurIPS 2022, SBM Workshop}, year = {2022}, page = {/publication/multires_textual_inversion/}, } - ICMLScore-Guided Intermediate Layer Optimization: Fast Langevin Mixing for Inverse ProblemsGiannis Daras*, Yuval Dagan*, Alexandros G. Dimakis, and Constantinos DaskalakisIn ICML 2022
We prove fast mixing and characterize the stationary distribution of the Langevin Algorithm for inverting random weighted DNN generators. This result extends the work of Hand and Voroninski from efficient inversion to efficient posterior sampling. In practice, to allow for increased expressivity, we propose to do posterior sampling in the latent space of a pre-trained generative model. To achieve that, we train a score-based model in the latent space of a StyleGAN-2 and we use it to solve inverse problems. Our framework, Score-Guided Intermediate Layer Optimization (SGILO), extends prior work by replacing the sparsity regularization with a generative prior in the intermediate layer. Experimentally, we obtain significant improvements over the previous state-of-the-art, especially in the low measurement regime.
@inproceedings{sgilo, title = {Score-Guided Intermediate Layer Optimization: Fast Langevin Mixing for Inverse Problems}, author = {Daras, Giannis and Dagan, Yuval and Dimakis, Alexandros G. and Daskalakis, Constantinos}, booktitle = {ICML 2022}, year = {2022}, page = {/publication/sgilo/}, } - NeurIPSMultitasking Models are Robust to Structural Failure: A Neural Model for Bilingual Cognitive ReserveGiannis Daras*, Negin Raoof*, Zoi Gkalitsiou, and Alexandros G. DimakisIn NeurIPS 2022
We find a surprising connection between multitask learning and robustness to neuron failures. Our experiments show that bilingual language models retain higher performance under various neuron perturbations, such as random deletions, magnitude pruning and weight noise compared to equivalent monolingual ones. We provide a theoretical justification for this robustness by mathematically analyzing linear representation learning and showing that multitasking creates more robust representations. Our analysis connects robustness to spectral properties of the learned representation and proves that multitasking leads to higher robustness for diverse task vectors. We open-source our code and models: https://github.com/giannisdaras/multilingual_robustness
@inproceedings{bilingual_robustness, title = {Multitasking Models are Robust to Structural Failure: A Neural Model for Bilingual Cognitive Reserve}, author = {Daras, Giannis and Raoof, Negin and Gkalitsiou, Zoi and Dimakis, Alexandros G.}, booktitle = {NeurIPS 2022}, year = {2022}, } - NeurIPS WorkshopDiscovering the Hidden Vocabulary of DALLE-2Giannis Daras* and Alexandros G. DimakisIn NeurIPS 2022 Workshop on Score-Based Methods
We discover that DALLE-2 seems to have a hidden vocabulary that can be used to generate images with absurd prompts. For example, it seems that \texttt(Apoploe vesrreaitais) means birds and \texttt(Contarra ccetnxniams luryca tanniounons) (sometimes) means bugs or pests. We find that these prompts are often consistent in isolation but also sometimes in combinations. We present our black-box method to discover words that seem random but have some correspondence to visual concepts. This creates important security and interpretability challenges.
@inproceedings{hidden_vocab, title = {Discovering the Hidden Vocabulary of DALLE-2}, author = {Daras, Giannis and Dimakis, Alexandros G.}, booktitle = {NeurIPS 2022 Workshop on Score-Based Methods}, year = {2022}, page = {/publication/hidden_vocab/}, }
2021
- PreprintSolving Inverse Problems with NerfGANsGiannis Daras, Wen-Sheng Chu, Abhishek Kumar, Dmitry Lagun, and Alexandros G. Dimakis2021
We introduce a novel framework for solving inverse problems using NeRF-style generative models. We are interested in the problem of 3-D scene reconstruction given a single 2-D image and known camera parameters. We show that naively optimizing the latent space leads to artifacts and poor novel view rendering. We attribute this problem to volume obstructions that are clear in the 3-D geometry and become visible in the renderings of novel views. We propose a novel radiance field regularization method to obtain better 3-D surfaces and improved novel views given single view observations. Our method naturally extends to general inverse problems including inpainting where one observes only partially a single view. We experimentally evaluate our method, achieving visual improvements and performance boosts over the baselines in a wide range of tasks. Our method achieves 30-40% MSE reduction and 15-25% reduction in LPIPS loss compared to the previous state of the art.
@misc{inv_nerfgans, title = {Solving Inverse Problems with NerfGANs}, author = {Daras, Giannis and Chu, Wen-Sheng and Kumar, Abhishek and Lagun, Dmitry and Dimakis, Alexandros G.}, year = {2021}, } - ICMLIntermediate Layer Optimization for Inverse Problems using Deep Generative ModelsGiannis Daras, Joseph Dean, Ajil Jalal, and Alexandros G. DimakisIn ICML 2021
We propose Intermediate Layer Optimization (ILO), a novel optimization algorithm for solving inverse problems with deep generative models. Instead of optimizing only over the initial latent code, we progressively change the input layer obtaining successively more expressive generators. To explore the higher dimensional spaces, our method searches for latent codes that lie within a small l_1 ball around the manifold induced by the previous layer. Our theoretical analysis shows that by keeping the radius of the ball relatively small, we can improve the established error bound for compressed sensing with deep generative models. We empirically show that our approach outperforms state-of-the-art methods introduced in StyleGAN-2 and PULSE for a wide range of inverse problems including inpainting, denoising, super-resolution and compressed sensing.
@inproceedings{ilo_full_version, title = {Intermediate Layer Optimization for Inverse Problems using Deep Generative Models}, author = {Daras, Giannis and Dean, Joseph and Jalal, Ajil and Dimakis, Alexandros G.}, booktitle = {ICML 2021}, year = {2021}, page = {/publication/ilo/}, } - NeurIPSRobust Compressed Sensing MRI with Deep Generative PriorsAjil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G. Dimakis, and Jonathan I. TamirIn NeurIPS 2021
The CSGM framework (Bora-Jalal-Price-Dimakis’17) has shown that deep generative priors can be powerful tools for solving inverse problems. However, to date this framework has been empirically successful only on certain datasets (for example, human faces and MNIST digits), and it is known to perform poorly on out-of-distribution samples. In this paper, we present the first successful application of the CSGM framework on clinical MRI data. We train a generative prior on brain scans from the fastMRI dataset, and show that posterior sampling via Langevin dynamics achieves high quality reconstructions. Furthermore, our experiments and theory show that posterior sampling is robust to changes in the ground-truth distribution and measurement process. Our code and models are available at: \url(https://github.com/utcsilab/csgm-mri-langevin).
@inproceedings{mri, title = {Robust Compressed Sensing MRI with Deep Generative Priors}, author = {Jalal, Ajil and Arvinte, Marius and Daras, Giannis and Price, Eric and Dimakis, Alexandros G. and Tamir, Jonathan I.}, booktitle = {NeurIPS 2021}, year = {2021}, }
2020
- NeurIPSSMYRF: Efficient Attention using Asymmetric ClusteringGiannis Daras, Nikita Kitaev, Augustus Odena, and Alexandros G. DimakisIn NeurIPS 2020
We propose a novel type of balanced clustering algorithm to approximate attention. Attention complexity is reduced from O(N^2) to O(N \log N), where N is the sequence length. Our algorithm, SMYRF, uses Locality Sensitive Hashing (LSH) in a novel way by defining new Asymmetric transformations and an adaptive scheme that produces balanced clusters. The biggest advantage of SMYRF is that it can be used as a drop-in replacement for dense attention layers without any retraining. On the contrary, prior fast attention methods impose constraints (e.g. queries and keys share the same vector representations) and require re-training from scratch. We apply our method to pre-trained state-of-the-art Natural Language Processing and Computer Vision models and we report significant memory and speed benefits. Notably, SMYRF-BERT outperforms (slightly) BERT on GLUE, while using 50% less memory. We also show that SMYRF can be used interchangeably with dense attention before and after training. Finally, we use SMYRF to train GANs with attention in high resolutions. Using a single TPU, we were able to scale attention to 128x128=16k and 256x256=65k tokens on BigGAN on CelebA-HQ.
@inproceedings{smyrf, title = {SMYRF: Efficient Attention using Asymmetric Clustering}, author = {Daras, Giannis and Kitaev, Nikita and Odena, Augustus and Dimakis, Alexandros G.}, booktitle = {NeurIPS 2020}, year = {2020}, page = {/publication/smyrf/}, } - CVPRYour Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative ModelsGiannis Daras, Augustus Odena, Han Zhang, and Alexandros G. DimakisIn CVPR 2020
We introduce a new local sparse attention layer that preserves two-dimensional geometry and locality. We show that by just replacing the dense attention layer of SAGAN with our construction, we obtain very significant FID, Inception score and pure visual improvements. FID score is improved from 18.65 to 15.94 on ImageNet, keeping all other parameters the same. The sparse attention patterns that we propose for our new layer are designed using a novel information theoretic criterion that uses information flow graphs. We also present a novel way to invert Generative Adversarial Networks with attention. Our method extracts from the attention layer of the discriminator a saliency map, which we use to construct a new loss function for the inversion. This allows us to visualize the newly introduced attention heads and show that they indeed capture interesting aspects of two-dimensional geometry of real images.
@inproceedings{ylg, title = {Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Models}, author = {Daras, Giannis and Odena, Augustus and Zhang, Han and Dimakis, Alexandros G.}, booktitle = {CVPR 2020}, year = {2020}, }