Image classification - Assigning meaning to images via classes¶
The current tutorial will illustrate how to use Ipyannotator to classify images.
The task of identifying what an image represents is called image classification.
Ipyannotator allows users to explore an entire set of images and specific labels; manually create their datasets associating labels to images; improve existing annotations.
This tutorial is divided in the following steps:
Select dataset¶
For this tutorial you can select four different datasets:
Artificial Classification is a minimal dataset generated by Ipyannotator with 50 images in 3 classes to be labeled. It doesn’t require downloading and is used by default for this tutorial.
Cifar10 is a dataset with 60000 images in 10 classes of animals and passenger transporation vessels to be labeled.
Oxford102 is a dataset with 6149 images in 102 classes of flowers to be labeled.
Cub200 is a dataset with 6033 images in 200 classes of birds to be labeled.
You can choose between the datasets uncommenting the following cell.
dataset = DS.ARTIFICIAL_CLASSIFICATION
#dataset = DS.CIFAR10
# dataset = DS.OXFORD102
# dataset = DS.CUB200
You don’t need to download the data manually, it will be done automatically in the next step for datasets other than DS.ARTIFICIAL_CLASSIFICATION.
Setup annotator¶
This section will set up the paths and input/output data needed to classify the images.
The following cell imports the project file and directory where the images were downloaded (or generated). For this tutorial, we simplify the process using the get_settings function instead of hardcoding the paths.
# get special project settings for selected dataset
settings_ = get_settings(dataset)
settings_.project_file, settings_.image_dir
(Path('data/artificial_classification/annotations.json'), 'images')
Ipyannotator uses pairs of input/output data to set up the annotation.
The image classification annotator uses InputImage and OutputImageLabelas the pair to set up the annotator.
The InputImage function provides information about the directory that contains the images to be classified, and the images itself. The OutputImageLabel function provides information about the directory that contains the classes that can be associated with the images and labels itself.
input_ = InputImage(image_dir=settings_.image_dir,
image_width=settings_.im_width,
image_height=settings_.im_height)
output_ = OutputImageLabel(label_dir=settings_.label_dir,
label_width=settings_.label_width,
label_height=settings_.label_height)
input_.dir, output_.dir
('images', 'class_images')
The final part in setting up the Ipyannotator is the configuration of the Annotator factory with the pair of input/output data.
The factory allows three types of annotator tools: explore, create, improve. The next sections will guide you through every step.
anni = Annotator(input_, output_, settings_)
Explore¶
The explore option allows users to navigate across the images in the dataset using next/previous buttons. In case the dataset was already labeled, the labeling results can also be displayed. This function is used for data visualization only, improvement and addition of labels is done in the next steps.
explorer = anni.explore()
explorer
Sometimes the classes are not defined yet or incomplete. To explore the input images without worring about any classes you can use the NoOutput option on the annotator factory which is done in the following:
unlabel_factory = Annotator(input_, NoOutput(), settings_)
unlabel_factory.explore()
Create¶
The create option allows users to manually create their annotated datasets.
The next cell removes already created results for any dataset that can be chosen in this tutorial.
dirpath = ''
if dataset == DS.ARTIFICIAL_CLASSIFICATION:
dirpath = 'data/artificial_classification/create_results'
elif dataset == DS.CIFAR10:
dirpath = 'data/cifar10/create_results'
elif dataset == DS.OXFORD102:
dirpath = 'data/oxford-102-flowers/create_results'
elif dataset == DS.CUB200:
dirpath = 'data/CUB_200_2011/create_results'
if os.path.exists(dirpath) and os.path.isdir(dirpath):
shutil.rmtree(dirpath)
The next cell initializes the create option.
For this tutorial, a function was defined that imitates human work. You can choose between performing the annotation manually yourself or letting the function do the work for you.
creator = anni.create()
creator
The next cell imitates human work by automatically annotating all images randomly. If you want to manually annotate then skip the next step.
If you choose to annotate manually be sure to have some images incorrectly annotated. In this way you prepare a good dataset for the improve step below.
HELPER = Tutorial(dataset, settings_.project_path)
HELPER.annotate_randomly(creator)
The example above shows how to set up the image classification using already predefined labels. Occasionally, you may want to create a dataset with your own text labels, for example ‘Circle’ and ‘Rectangle’. You can create a dataset with new output labels like this:
output_label = OutputLabel(class_labels=['Circle', 'Rectangle'])
text_label_factory = Annotator(input_, output_label, settings_)
text_label_factory.create()
/home/runner/work/ipyannotator/ipyannotator/ipyannotator/storage.py:35: UserWarning: Error: Annotations file already exists in data/artificial_classification/create_results!
If you want to create annotations from scratch - use empty dir!
warnings.warn(f"Error: Annotations file already exists in {results_dir}!"
Improve¶
The improve feature allows users to refine the annotated dataset. This feature groups the annotated images according to their class and edits each class separately. This means that if your dataset has 3 labeling classes, 3 annotators instances are initiated to improve each class separately.
As before, for the purpose of the tutorial, a function can be used to performe the annotation and you don’t have to annotate manually. If you want to annotate manually then make sure to mark all errors (images, which belongs to different class).
If you chose to annotate manually don’t forget to click the SAVE button when finished with each class.
all_improvers = anni.improve()
Check the number of classes:
len(all_improvers)
3
Let’s select the first two classes to mark incorrectly labeled images:
all_improvers[:2]
[, ]
The next cell imitates the human work. If you chose to annotate manually make sure to skip this cell.
HELPER.fix_incorrect_annotations(all_improvers)
Now we obtain a list of all marked images that need to be reclassified:
reclasify_this = [[c for c, v in i.to_dict().items() if v['answer']] for i in all_improvers]
# Show 10 files with incorrect labels
for a in reclasify_this:
print(a[:10])
['data/artificial_classification/images/img_0.jpg', 'data/artificial_classification/images/img_23.jpg', 'data/artificial_classification/images/img_42.jpg']
['data/artificial_classification/images/img_0.jpg', 'data/artificial_classification/images/img_23.jpg', 'data/artificial_classification/images/img_42.jpg']
['data/artificial_classification/images/img_0.jpg', 'data/artificial_classification/images/img_23.jpg', 'data/artificial_classification/images/img_42.jpg']
Postprocessing¶
This section exemplifies how to process the data after the improve feature has been applied. By default, the improve feature creates a missed folder in your storage with a folder for every class available in the dataset.
The next cell loads one JSON file for a random class and displays the filenames of images marked as incorrectly labeled in the previous step.
from glob import glob
from random import sample
import pandas as pd
random_class = sample(glob(str(Path(
settings_.project_path) / settings_.result_dir / 'missed') + '/*'), 1)[0]
random_class_annotation = pd.read_json(Path(random_class) / 'annotations.json').T
anwered_missed = random_class_annotation[random_class_annotation['answer'] == True] # noqa: E712
random_missed = list(anwered_missed.index.values)
# shows selected random class and 10 files with incorrect labels within that class
random_class, random_missed[:10]
# result may be empty, if all annotations are correct
('data/artificial_classification/create_results/missed/red',
['data/artificial_classification/images/img_0.jpg',
'data/artificial_classification/images/img_23.jpg',
'data/artificial_classification/images/img_42.jpg'])