Creating a Mushroom Image Dataset (Update)
In this blog post I show how to create a mushroom image dataset using fastai v2 and DuckDuckGo.
This blog post is an update of an earlier blog post I wrote. In that ealier blog post I described how to create a custom dataset of mushroom images fetched from the internet using the fastai library to train a mushroom image classification model. Since I wrote this blog post, a new version of the fastai library has come out. This version is a complete rewrite of the library and it contains lots of changes compared to the old version. You can read more about the new fastai version here. In this blog post I describe how to create the mushroom image dataset using tools offered by this new fastai version.
As in my earlier blog post, I use Google Colab for model training and Google Drive to store the dataset. Thus, I need to mount Google Drive first of all.
from google.colab import drive
drive.mount('/content/gdrive', force_remount=True)
Next, let's load some magics (only useful if you use a Jupyter notebook or Google Colab).
%reload_ext autoreload
%autoreload 2
%matplotlib inline
Then, let's install the fastbook pip package. The fastbook pip package is offered by the fastai team as an easy way to use the fastai library. It was developed for the fastai MOOC and the fastai book. It includes all the tools that we need to create the dataset.
! pip install -Uqq fastbook
import fastbook
fastbook.setup_book()
Let's import the necessary libraries. Additionally, let's also print the version of the fastai and fastbook package as well as the python version, CUDA version and which GPU we use. I use this for all my notebooks now, since it makes it easier to replicate my results later.
import fastai
import platform
from fastbook import *
from fastai.vision.widgets import *
print('python version: {}'.format(platform.python_version()))
print('fastai version: {}'.format(fastai.__version__))
use_cuda = torch.cuda.is_available()
print('CUDA available: {}'.format(use_cuda))
print('cuDNN enabled: {}'.format(torch.backends.cudnn.enabled))
print('num gpus: {}'.format(torch.cuda.device_count()))
if use_cuda:
print('gpu: {}'.format(torch.cuda.get_device_name(0)))
print()
print('------------------------- CUDA -------------------------')
! nvcc --version
Let's specify where we want to store the dataset later.
root_dir = Path('/content/gdrive/My Drive/fastai/data')
data_dir = root_dir/'mushrooms'
Now, we are ready to download the images. In my earlier blog post I used Google Images for this. However, I had to manually search for images of each mushroom category and then get the image URLs using some JavaScript code. The fastbook package offers some functions to get the URLs for me, so that I don't need to do that manually anymore. But instead of Google Images fastbook contains a function using Bing and a function using DuckDuckGo. I decided to use DuckDuckGo here, since it is quite easy to use (e.g., in contrast to Bing it doesn't require creating an account).
fastbook package I found the jmd_imagescraper package. This package also uses DuckDuckGo to fetch images from the internet. Additonally, it also offers an own image cleaner. However, I haven’t used the package, yet. Thus, I don’t know how reliable and well maintained it is.
Before we can download the images, we need to decide which images we want to download. I decided to include the same 8 mushroom classes as in my earlier blog post. Thus, I use the same search terms as well. However, since I changed the search engine from Google Images to DuckDuckGo, it could be possible that we got better results by slightly adjusting the search terms (e.g., for DuckDuckGo we might need to exclude other keywords). But for simplicity reasons I keep it like this for now.
term_amanita_muscaria = "'amanita muscaria' -facebook -twitter -youtube -slideshare -reddit -apotheke -researchgate -sciencedirect -shop -music -kunst -kunstsammlung -illustration -cartoon -tattoo -christmas -weihnachten -halloween -filz -hat -shirt -fandom -modell -stamp -map -getrocknet -cloning"
term_boletus_santanas = "'boletus satanas' -facebook -youtube -amazon -soundcloud -spotify -shazam -bandcamp -necrocock -cremaster -researchgate -magazine -deviantart -comics -illustration -icon -reklamebilder -briefmarke -briefmarkenwelt -stamp -modell -shirt -map"
term_amanita_phalloides = "'amanita phalloides' -facebook -twitter -youtube -amazon -slideshare -soundcloud -spotify -heavenchord -journals -czechmycology -researchgate -sciencedirect -semanticscholar -slideplayer -deviantart -illustration -alice -icon -3dcadbrowser -stamp -hosen -map -fliegenpilz -chemistry -chemical -mykothek"
term_amanita_virosa = "'amanita virosa' -facebook -apple -twitter -youtube -amazon -bookdepository -animaltoyforum -slideplayer -review -table -soundcloud -spotify -shazam -recordshopx -album -deviantart -illustration -danbooru -character -stamp -map -microscope -chemistry -chemisch -chemical -fungalspores -asystole -mykothek"
term_agaricus_campestris = "'agaricus campestris' -facebook -twitter -tumblr -youtube -amazon -reddit -researchgate -japanjournalofmedicine -semanticscholar -slideplayer -untersuchung -shop -gamepedia -illustration -zeichnung -drawing -model -modell -clipdealer -clipart -cyberleninka -scandposters -fototapete -philatelie -stamp -map -candy -pferd -buy -pilzkorb -pilzgericht -mykothek"
term_morchella_esculenta = "'morchella esculenta' -facebook -twitter -youtube -reddit -amazon -alibaba -ebay -tripadvisor -researchgate -thesis -semanticscholar -sciencedirect -linkedin -dribbble -music -art -artsy -kunst -illustrated -clipartlogo -3D -plakate -stamp -monkstars -shirt -map -dried -porzellan -mykothek"
term_cantharellus_cibarius = "'cantharellus cibarius' -facebook -twitter -youtube -amazon -tripadvisor -researchgate -medical -semanticscholar -sciencedirect -dribbble -art -deviantart -clipart -cubanfineart -hiclipart -modell -sketchup -fandom -stamp -briefmarken -map -getrocknete -food -shirt -gericht -basket -korb -dried -porzellan -mykothek"
term_boletus_edulis = "'boletus edulis' -youtube -amazon -alibaba -ebay -tripadvisor -researchgate -semanticscholar -scientific -mikroskopie -russianpatents -onlineshop -illustration -modell -servietten -fandom -briefmarken -shirt -stamp -map -cutted -dried -getrocknet -salsa -sauce -samen -basket -korb"
Let's put the search term for each mushroom class into a dictionary.
search_terms = {
'agaricus_campestris': term_agaricus_campestris,
'amanita_muscaria': term_amanita_muscaria,
'amanita_phalloides': term_amanita_phalloides,
'amanita_virosa': term_amanita_virosa,
'boletus_edulis': term_boletus_edulis,
'boletus_satanas': term_boletus_santanas,
'cantharellus_cibarius': term_cantharellus_cibarius,
'morchella_esculenta': term_morchella_esculenta,
}
Now, we are ready to download the mushroom images. First, we need to search for images for each mushroom class using its search term via DuckDuckGo. DuckDuckGo gives us the image URLs as the result for each search term then. Second, we need to download each image using its URL.
To receive the image URLs from DuckDuckGo fastbook offers the function search_images_ddg. The function takes the search term and the maximum number of image URLs we want to receive as input, and returns a list of image URLs as output. However, there is currently a bug in the implementation of the search_images_ddg function. The bug is already fixed in the GitHub repository but the fastbook pip package hasn't been built with the fix, yet. As a workaround we can simply copy the function from the GitHub repository and use it instead of importing it from fastbook.
def search_images_ddg(key,max_n=200):
"""Search for 'key' with DuckDuckGo and return a unique urls of 'max_n' images
(Adopted from https://github.com/deepanprabhu/duckduckgo-images-api)
"""
url = 'https://duckduckgo.com/'
params = {'q':key}
res = requests.post(url,data=params)
searchObj = re.search(r'vqd=([\d-]+)\&',res.text)
if not searchObj: print('Token Parsing Failed !'); return
requestUrl = url + 'i.js'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0'}
params = (('l','us-en'),('o','json'),('q',key),('vqd',searchObj.group(1)),('f',',,,'),('p','1'),('v7exp','a'))
urls = []
while True:
try:
res = requests.get(requestUrl,headers=headers,params=params)
data = json.loads(res.text)
for obj in data['results']:
urls.append(obj['image'])
max_n = max_n - 1
if max_n < 1: return L(set(urls)) # dedupe
if 'next' not in data: return L(set(urls))
requestUrl = url + data['next']
except:
pass
When the fastbook package is build with the current code, we can remove the workaround again and important search_images_ddg from the package.
Next, let's search for images for each mushroom class using search_images_ddg, download them and store the images of each class in an own directory. I wrote a small function for this.
def download_dataset(search_terms, out_dir, max_images):
categories = search_terms.keys()
if not out_dir.exists():
out_dir.mkdir()
for c in categories:
dest = (out_dir/c)
dest.mkdir(exist_ok=True)
urls = search_images_ddg(search_terms[c], max_n=max_images)
download_images(dest, urls=urls)
print(f'{c} images downloaded')
else:
raise FileExistsError(f'Path {out_dir} exits already')
Let's call the function.
download_dataset(search_terms, data_dir, 200)
The images should be stored in Google Drive now. Let's load them using the get_image_files function.
fns = get_image_files(data_dir); fns
As we can see, we have downloaded 1375 images. However, before we use them to train the mushroom classifier, we should check all images if they are okay or if some of them are corrupted. We can check the files using the function verify_images.
failed = verify_images(fns); failed
There are 12 corrupted images. Let's remove them. We can do this by calling unlink on each of them.
failed.map(Path.unlink);
Then, let's load the images again.
fns = get_image_files(data_dir); fns
Now, we have 1363 images. Let's load them using the data block API. Since we have a single label image classification here, we need to specify ImageBlock as input and CategoryBlock as output. I decided to crop all images to a size of 224x224 pixel. Additionally, we should apply some standard data augmentation techniques on the images. We can use the function aug_transforms for this (check the default values of the function to see which data augmentations are applied). As target labels we simply use the name of each mushroom folder. This can be done by using the parent_label function. Finally, we split the dataset randomly into 80% training set and 20% validation set using the RandomSplitter.
mushrooms = DataBlock(
blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
splitter=RandomSplitter(valid_pct=0.2, seed=42),
get_y=parent_label,
item_tfms=RandomResizedCrop(224, min_scale=0.5),
batch_tfms=aug_transforms()
)
dls = mushrooms.dataloaders(data_dir)
len(dls.train.dataset), len(dls.valid.dataset)
We have 1091 images in our training set and 272 in our validation set. Actually, we should also have defined a test set to be able to report our final performance of our model at the end. However, since I only want to show how to create a dataset using images from the internet rather than creating a ready-to-use mushroom classifier, I will skip the creation of a test set here.
Let's look at some images.
dls.show_batch()
Now, we are ready to train the model using transfer learning. First, let's create a cnn_learner. As in my earlier blog post, I use a ResNet-50 as model architecturen. If you want to use another architecture, you can take a look here to see which pretrained models PyTorch offers.
learn = cnn_learner(
dls, resnet50,
metrics=accuracy
)
Before we start training, let's find a good learning rate using the learning rate finder that you can use via the lr_find function.
learn.lr_find()
We should pick the learning rate at one tenth of the minimum before it diverges. When looking at the plot, this should be 1e-2. For more detailed information about how to pick a good learning rate, you can take a look here.
Now, we can start training the model. To train the model I used the fit_one_cycle function in my earlier blog post, which performs training using the 1-cycle-policy introduced by Leslie N. Smith in his paper about Super Convergence. First, I only trained the custom head of the model while leaving the pretrained weights in the body of the model unchanged. Then, I unfroze the body and further trained the whole model. However, now I want to use the function fine_tune, which was introduced in fastai v2. The function does the same thing. First, it trains the head of the model for one epoch. Then, it unfreezes the body and further trains the whole model for a specified number of epochs. As a result, we do not need to perform these two steps manually anymore. The function fine_tune takes care of it for us. Additionally, it also tunes the learning rate and other hyper-parameters internally in a way that works pretty good for most cases (you can find the source code of fine_tune here). So, let's train our model via fine_tune for 6 epochs.
learn.fine_tune(6, 1e-2)
We receive an accuracy of approximately 93.38%. This is not bad. Let's look at the confusion matrix.
interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix(figsize=(12,12), dpi=60)
We can see that our model works quite well already. Furthermore, all classes seem to have a similar number of images (actually I should have checked that at the beginning to see whether we have an imbalanced dataset). However, there are still some problems, e.g. with the class boletus edulis. Let's look at some samples that our model is wrong about.
interp.plot_top_losses(6, figsize=(20,10), nrows=2)
interp.most_confused()
As we can see, there are some problems in our data. The sample in the top left and in the top middle are the same images. Furthermore, their target labels are wrong. I am not a mushroom expert, but I think it's clear that the images show an amanita muscaria and not an amanita phalloides. So, our prediction was actually correct but the target label is wrong. Finally, we also seem to have really strange images in our dataset as the one in the bottom right. The white circle might consist of mushrooms but it's not clear at all. We want images in our dataset that clearly show a mushroom. So, let's clean our images. But before doing that we should save the current version of our model.
learn.save('stage-1')
To clean the dataset we can use the ImageClassifierCleaner widget. The widget gives us a small GUI that displays the images of our dataset with the option to remove images or to change the target label of an image. Furthermore, it shows the images the model is most wrong about first, because these are the most likely ones that need to be corrected or removed, respectively. This way we can remove weird images that don't show mushrooms and correct the target labels if they are incorrect. So, let's start the widget.
cleaner = ImageClassifierCleaner(learn)
cleaner
However, the widget itself actually doesn't remove any images or changes any target labels. It just marks deletions and target label changes. We can remove images and adjust target labels using these information from the widget with the following code.
for idx in cleaner.delete(): cleaner.fns[idx].unlink()
for idx,cat in cleaner.change():
new_path = cleaner.fns[idx].parent / Path(f'from_{cat}_' + cleaner.fns[idx].name)
shutil.move(str(cleaner.fns[idx]), str(new_path))
shutil.move(str(new_path), data_dir/cat)
The ImageClassifierCleaner widget is also explained in the fastai book. However, I think there is a small bug when changing a target label. I corrected this in the code above.
Let's load the images again to see how many we still have left.
fns = get_image_files(data_dir); fns
We have 1094 images left. This should be still enough to train our toy model here. Let's load the dataset.
dls = mushrooms.dataloaders(data_dir)
len(dls.train.dataset), len(dls.valid.dataset)
Let's also look at some samples.
dls.show_batch()
There still seem to be strange images like the one in the top middle, but let's keep it like this for now. Next, we train our model once again with the cleaned dataset.
Let's create the cnn_learner and find the learning rate.
learn = cnn_learner(
dls, resnet50,
metrics=accuracy
)
learn.lr_find()
The plot is quite similar to the one we had before. So, let's stick to a learning rate of 1e-2 and train the model for 6 epochs again using fine_tune.
learn.fine_tune(6, 1e-2)
As we can see, the performance of our model has improved about 3% to an accuracy of 96.33%. Let's look at the confusion matrix.
interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix(figsize=(12,12), dpi=60)
This looks better now. Nevertheless, let's also see when our model is still wrong.
interp.plot_top_losses(6, figsize=(20,10), nrows=2)
interp.most_confused()
As we can see, our model still makes a few mistakes but our data seems to be better now. Let's save the model.
learn.save('stage-2')
Our model reaches an accuracy of 96.33%. This is less than the model in my earlier blog post where I reached 98.13%. However, with more (clean) data and more experimentations I think we can improve our performance back up to 98%. But the puprose of this blog post is to show how to create an image dataset and not to get a ready-to-use mushroom image classifier. Hence, I will keep the model like that for now.
Actually, to be able to use our model in practice we need to do a lot more. We need to include more mushroom types (I only included 8 here but there are way more in reality) and we need to test our model on way more data. Moreover, as I already mentioned in my earlier blog post, it would be good if our model could tell us when it is not sure. For instance, when we show it an image of a mushroom that is very different from the mushroom images it saw during training (e.g., a new mushroom type that we didn't consider or an image showing a mushroom in very weird lighting conditions that we didn't have during training), it shouldn't give us just a mushroom class but also a score expressing how sure it is that this is the correct class. Currently, our classifier would simply predict any of the 8 classes when it sees an unknown mushroom type. So, be cautious. Mixing up mushroom can be dangerous! However, since the focus of this blog post is on creating an own training dataset and not on how to create a ready-to-use mushroom classifier, we are done for now. In the fastai book is actually a chapter about how to get a model into production. I recommend checking that out.