Unsupervised Domain Adaptation

General Information

Group project for the Deep Learning course of the Master’s Degree in Computer Science at University of Trento.

Description & Objectives

The main focus of this project is to build, train, and evaluate a deep learning framework on a standard setting of Unsupervised Domain Adaptation (UDA).

The test dataset on which the proposed methods are tested is the Adaptiope, a large scale, diverse UDA dataset with 3 domains: synthetic, product, and real-world data.

The implemented networks will be trained:

• in a supervised way on the source domain;
• in an unsupervised way on the target domain.

The gain, namely the amount of improvement we receive when we use the domain adaption framework compared to the outcome we get without any domain alignment approach, is used to assess the quality of the domain alignment strategy. The gain can be represented mathematically as:

This has been be done in both directions between the two selected domains, using as evaluation metric the validation accuracy on the target domain:

Where:

• $TP$ stands for true positives, namely cases in which the algorithm correctly predicted the positive class.
• $TN$ stands for true negatives, namely cases in which the algorithm correctly predicted the negative class.
• $FP$ stands for false positives, namely cases in which samples are associated with the positive class whereas they belong to the negative one. (Also known as a “Type I error.”)
• $FN$ stands for false negatives, namely cases in which samples are associated with the negative class whereas they belong to the positive one. (Also known as a “Type II error.”)

Eventually lower and upper bounds will be calculated training the networks only on one specific domain (source domain for a lower bound, target domain for an upper bound), enabling us to provide some considerations about the achievements of our architectures.

Domain Adaptation

Just to give a formal background, we leverage on the definition of Domain Adaptation provided by Gabriela Csurka et.al which in the following sub-section will be presented.

This process is known as transfer learning.

Furthermore, the Domain Adaptation field distinguishes between two scenarios:

1. Unsupervised Domain Adaptation, where labels are only available for the source domain.
2. Semi-supervised Domain Adaptation, where only a limited number of target samples are labeled.

Proposed Solutions

When addressing Deep Domain Adaptation problems, there are three primary approaches:

1. Discrepancy-based methods: These methods align the domain data representation using statistical measures.
2. Adversarial-based methods: They involve a domain discriminator to enforce domain confusion.
3. Reconstruction-based methods: These methods use an auxiliary reconstruction task to ensure a domain-invariant feature representation.

This repository includes solutions for each of these methods, as well as a custom solution that combines two of them: discrepancy-based and reconstruction-based (DSN + MEDM).

Due to the limited availability of examples, training a network from scratch has not yielded satisfactory results. Therefore, we start with a feature extractor pre-trained on the ImageNet object recognition dataset.

Additionally, to explore the proposed methods thoroughly, we provide the option to evaluate the domain adaptation method using two different backbone networks: AlexNet and ResNet18.

Network Architectures

To effectively address the domain adaptation challenge, we have implemented various network architectures adapted from relevant research papers and tailored to our requirements.

Our primary objective, as stated in the introduction, is to assess the performance (gain) of some of the most significant approaches in the domain adaptation field, integrated with two of the most prominent convolutional neural network architectures: AlexNet and ResNet.

In this repository, we cover:

• Deep Domain Confusion Networks
• Domain Adversarial Neural Network
• Domain Separation Networks
• Rotation loss Network
• Entropy Minimization vs. Diversity Maximization for Domain Adaptation
• Our attempt to combine MEDM with DANN
• Another approach to combine MEDM with DSN

Results

In this project, we explored various architecture categories for domain adaptation techniques, including discrepancy-based, adversarial-based, and reconstruction-based. Notably, the most effective approaches were DANN and MEDM.

MEDM is a versatile approach that can be seamlessly integrated into any architecture, featuring two straightforward entropy losses that result in stable training and impressive performance. In contrast, DANN is more susceptible to hyperparameter variations and often leads to unstable training.

During our experiments, we found that combining DSN and MEDM proved effective, while combining MEDM and DANN did not consistently yield improvement.

Adapting from the product domain to the real world is more challenging, but it offers a greater potential for improvement since there is more room between the baseline and the upper bound. Conversely, transitioning between real-world and product images ensures higher accuracy because the networks have been pretrained on ImageNet. Consequently, the gap between the baseline and the real world is narrower, making significant progress more challenging.

After testing both AlexNet and ResNet18 as backbone architectures, we discovered that not only did ResNet18 exhibit higher baseline and upper-bound values, but all domain adaptation strategies also generated substantially greater gains.

t-SNE plots clearly demonstrated that our final solution effectively extracts relevant features from both domains while maintaining accurate classification. The distributions closely resemble the optimal ones, represented by the upper bound ResNet18 model.

Distribution Alignment Plots

We have included plots that illustrate how features from the target and source domains are distributed using dimensionality reduction techniques.

For instance, we present the product-to-real-world domain adaptation, focusing on the best-performing model, namely DSN + MEDM. To provide an effective representation of distribution alignment and consolidate the most significant graph for our project, we present two graphs:

1. The distribution of features learned by the backbone neural network (ResNet18 in both cases), excluding the classifier or bottleneck layers.
2. The predicted class distribution.

Note: Both plots are generated based on the target domain (real-world), which the DSN+MEDM approach does not have access to.

Ideally, we aim for our domain adaptation model to achieve performance similar to that of the upper-bound model, which provides the best generalization for both source and target domain samples. Therefore, the model’s feature distribution is a valuable metric for assessing the quality of the domain adaptation technique.

Upper Bound Features Distribution on the Test Set:

Upper Bound Classes Distribution on the Test Set:

We expect both the classes and features distribution to align closely with the target distributions, as the model has achieved remarkable performance.

DSN + MEDM Features Distribution on the Test Set:

As evidenced by the class distributions below, the model’s classifier effectively clusters examples while maintaining high accuracy. Clusters are well-defined and separated, even for target domain examples, which are correctly classified.

DSN + MEDM Classes Distribution on the Test Set:

In both cases, PCA did not provide significant insights into the results, as a linear transformation may be too simplistic for reducing high-dimensional data to a lower-dimensional embedding.

Performance

Here are the results we achieved on the smaller version of Adaptiope:

Contributors

• Samuele Bortolotti
• Luca De Menego