31:Tbaf,The neural network is a simple feed forward one with some residual blocks ``` ============================================================================ Layer (type:depth-idx) Output Shape Param # ============================================================================ RotationNetwork -- -- ├─Sequential: 1-1 [1, 2] -- │ └─Conv2d: 2-1 [1, 8, 64, 64] 80 │ └─BatchNorm2d: 2-2 [1, 8, 64, 64] 16 │ └─ReLU: 2-3 [1, 8, 64, 64] -- │ └─EncoderBlock: 2-4 [1, 8, 32, 32] -- │ │ └─ResidualBlock: 3-1 [1, 8, 64, 64] 1,200 │ │ └─ResidualBlock: 3-2 [1, 8, 64, 64] 1,200 │ │ └─Sequential: 3-3 [1, 8, 32, 32] 1,048 │ └─EncoderBlock: 2-5 [1, 16, 16, 16] -- │ │ └─ResidualBlock: 3-4 [1, 8, 32, 32] 1,200 │ │ └─ResidualBlock: 3-5 [1, 8, 32, 32] 1,200 │ │ └─Sequential: 3-6 [1, 16, 16, 16] 2,096 │ └─EncoderBlock: 2-6 [1, 32, 8, 8] -- │ │ └─ResidualBlock: 3-7 [1, 16, 16, 16] 4,704 │ │ └─ResidualBlock: 3-8 [1, 16, 16, 16] 4,704 │ │ └─Sequential: 3-9 [1, 32, 8, 8] 8,288 │ └─EncoderBlock: 2-7 [1, 64, 4, 4] -- │ │ └─ResidualBlock: 3-10 [1, 32, 8, 8] 18,624 │ │ └─ResidualBlock: 3-11 [1, 32, 8, 8] 18,624 │ │ └─Sequential: 3-12 [1, 64, 4, 4] 32,960 │ └─EncoderBlock: 2-8 [1, 64, 1, 1] -- │ │ └─ResidualBlock: 3-13 [1, 64, 4, 4] 74,112 │ │ └─ResidualBlock: 3-14 [1, 64, 4, 4] 74,112 │ │ └─Sequential: 3-15 [1, 64, 1, 1] 65,728 │ └─Flatten: 2-9 [1, 64] -- │ └─Linear: 2-10 [1, 64] 4,160 │ └─BatchNorm1d: 2-11 [1, 64] 128 │ └─ReLU: 2-12 [1, 64] -- │ └─Linear: 2-13 [1, 2] 130 │ └─Tanh: 2-14 [1, 2] -- ============================================================================ Total params: 314,314 Trainable params: 314,314 Non-trainable params: 0 Total mult-adds (M): 50.64 ``` The training took less than 10 mins and the results were quite nice.32:[{"id":"ea1fb0c803576af833ff1d727d57653c0ec8c47e42438b976cb0179e08b895c675c8180615057f22f0cd6cb9ce0f6f58d361cc6cb73f34df03b319487cd23b8e","href":"https://static.observableusercontent.com/files/ea1fb0c803576af833ff1d727d57653c0ec8c47e42438b976cb0179e08b895c675c8180615057f22f0cd6cb9ce0f6f58d361cc6cb73f34df03b319487cd23b8e","name":"fig_cvat_1.jpg","type":"image/jpeg","lastModified":1621258546750.785,"size":282932},{"id":"19514727878dd6b2db0dc7aad58f7073412f5a01c1dd1b8f57b238f17548cd88288c2c0ca429ca6cf9fa31bd430d4a58dc543bf9cf428a82789694dcf6d591f2","href":"https://static.observableusercontent.com/files/19514727878dd6b2db0dc7aad58f7073412f5a01c1dd1b8f57b238f17548cd88288c2c0ca429ca6cf9fa31bd430d4a58dc543bf9cf428a82789694dcf6d591f2","name":"fig_coco_2.jpg","type":"image/jpeg","lastModified":1621259036763.077,"size":519068},{"id":"43ec121d899f344f04286c2695d7895a68ca93cf219683c324b677f21c21317f1219317e302b2c3e4355b59db467958aa4fb982aa52e2f119c82488390efab10","href":"https://static.observableusercontent.com/files/43ec121d899f344f04286c2695d7895a68ca93cf219683c324b677f21c21317f1219317e302b2c3e4355b59db467958aa4fb982aa52e2f119c82488390efab10","name":"fig_coco_1.jpg","type":"image/jpeg","lastModified":1621259036311.707,"size":572207},{"id":"06525691f6aa376079182f0eb2e0cd71c22b7e64621236d878960b6e8e70155d5c55739cc4858f1884a8f00cd03b06762271053e7d7d320b100585ad13b47dc2","href":"https://static.observableusercontent.com/files/06525691f6aa376079182f0eb2e0cd71c22b7e64621236d878960b6e8e70155d5c55739cc4858f1884a8f00cd03b06762271053e7d7d320b100585ad13b47dc2","name":"fig_rot_result.png","type":"image/png","lastModified":1621296885179.36,"size":159480},{"id":"b01af887965f1a1ab03c0c41eda0f4642758783bbb4ae2c1d02e1c5d594d71f9e9df6592c10163292f5cc8dcc24a09bba114307c428a619cd52f8df5697d2a8d","href":"https://static.observableusercontent.com/files/b01af887965f1a1ab03c0c41eda0f4642758783bbb4ae2c1d02e1c5d594d71f9e9df6592c10163292f5cc8dcc24a09bba114307c428a619cd52f8df5697d2a8d","name":"fig_rot_applied.jpg","type":"image/jpeg","lastModified":1621297205264.148,"size":1000668},{"id":"d09506a524103c74e4db489f5119680a5fa1dbaf8c25f938486cdbf72f016da1eded4f5c2af51439c08d14ec2722250b0f60a46b9ed1a3a1b28bca9ba2b48b59","href":"https://static.observableusercontent.com/files/d09506a524103c74e4db489f5119680a5fa1dbaf8c25f938486cdbf72f016da1eded4f5c2af51439c08d14ec2722250b0f60a46b9ed1a3a1b28bca9ba2b48b59","name":"fig_rot_label.png","type":"image/png","lastModified":1621296356106.133,"size":39810},{"id":"d81ff6f4b97d5cd4cf90ac635ed19917aa36ad47bc19e9c0585d2e2e08a51ca503949fe47adb3dd326f34f2415cd9970c41c788764c00d6c9f86c0cd9b1e6642","href":"https://static.observableusercontent.com/files/d81ff6f4b97d5cd4cf90ac635ed19917aa36ad47bc19e9c0585d2e2e08a51ca503949fe47adb3dd326f34f2415cd9970c41c788764c00d6c9f86c0cd9b1e6642","name":"fig_mask_center.png","type":"image/png","lastModified":1621296079906.116,"size":188328},{"id":"36eef5e8299e91c39638707f663f7315204d16ea50b88cc797209b6b32bb398bc97a2c5c536b07840a55e41d5731f994c57f1b7628aef14f04d2947ccf0a38d4","href":"https://static.observableusercontent.com/files/36eef5e8299e91c39638707f663f7315204d16ea50b88cc797209b6b32bb398bc97a2c5c536b07840a55e41d5731f994c57f1b7628aef14f04d2947ccf0a38d4","name":"fig_mask_train_best.jpg","type":"image/jpeg","lastModified":1621295826384.035,"size":217492},{"id":"e363369602aee18a98a80acd1550b8d11d4ad369e497adb59ab44ac7b941d67b78693a8392b32e28ed967faab2a4a8a673dd9c3ee348b4859a8a60ccef43a7de","href":"https://static.observableusercontent.com/files/e363369602aee18a98a80acd1550b8d11d4ad369e497adb59ab44ac7b941d67b78693a8392b32e28ed967faab2a4a8a673dd9c3ee348b4859a8a60ccef43a7de","name":"fig_mask_train_worst.jpg","type":"image/jpeg","lastModified":1621295826386.961,"size":237384},{"id":"be37e32f6fff4a5aa412e0e7c2137991ea5ce3a0d8c4ac8e839eb326aae864131fc9869a91b4590e95a014671108bc50b2f27c5bfe18dc682fa6938c26e46e13","href":"https://static.observableusercontent.com/files/be37e32f6fff4a5aa412e0e7c2137991ea5ce3a0d8c4ac8e839eb326aae864131fc9869a91b4590e95a014671108bc50b2f27c5bfe18dc682fa6938c26e46e13","name":"fig_mask_train.jpg","type":"image/jpeg","lastModified":1621295624494.408,"size":61789},{"id":"c90047fc43f60440a5cb5dfffe12831d236806a581332f3599d0289b4fd03dc2921f55f03dcde442816d86ea99e2be86ce12e61362a37538bd39bb7306dd1866","href":"https://static.observableusercontent.com/files/c90047fc43f60440a5cb5dfffe12831d236806a581332f3599d0289b4fd03dc2921f55f03dcde442816d86ea99e2be86ce12e61362a37538bd39bb7306dd1866","name":"fig_mask_refine@1.jpg","type":"image/jpeg","lastModified":1621295531426.022,"size":385284},{"id":"a7552fe37a7efb24fe6f97264f403fc1da27eb5fe86b51bb0da24791bd3d4f2b409bb9c9be0a446743f16c6afb75df82bb4306a9425821c4c727c63482a6e9a7","href":"https://static.observableusercontent.com/files/a7552fe37a7efb24fe6f97264f403fc1da27eb5fe86b51bb0da24791bd3d4f2b409bb9c9be0a446743f16c6afb75df82bb4306a9425821c4c727c63482a6e9a7","name":"fig_mask_opencv.png","type":"image/png","lastModified":1621294834690.591,"size":14706},{"id":"b19a4d1df3e284e664d4fed4914269ab95933bd9719d85b61a8b3210f59ad1fca0ed57cfe7658177cf757648f87d7506078df4c79b92b294a290ae7f518a22a7","href":"https://static.observableusercontent.com/files/b19a4d1df3e284e664d4fed4914269ab95933bd9719d85b61a8b3210f59ad1fca0ed57cfe7658177cf757648f87d7506078df4c79b92b294a290ae7f518a22a7","name":"fig_detect_best.jpg","type":"image/jpeg","lastModified":1621293890642.809,"size":986058},{"id":"408f1fc065f679d040046a6ae97eef66bd8b4c39c0ef2401e8d91d179efec6438ddeac6572d952e0c4393824b37625b49a43e7e6152ffe4f8001e7a206c101bd","href":"https://static.observableusercontent.com/files/408f1fc065f679d040046a6ae97eef66bd8b4c39c0ef2401e8d91d179efec6438ddeac6572d952e0c4393824b37625b49a43e7e6152ffe4f8001e7a206c101bd","name":"fig_detect_worst.jpg","type":"image/jpeg","lastModified":1621293890496.085,"size":919114},{"id":"f2e3d26fea0b5b4c80555d6e879935c63c2243538c655a022100b0787dbf24db4a76ce358b279c6c57baa57bea7a23a4208b7dd0e04e51a060b093b9dc4a8280","href":"https://static.observableusercontent.com/files/f2e3d26fea0b5b4c80555d6e879935c63c2243538c655a022100b0787dbf24db4a76ce358b279c6c57baa57bea7a23a4208b7dd0e04e51a060b093b9dc4a8280","name":"fig_detectron_2.jpg","type":"image/jpeg","lastModified":1621293889545.8,"size":738780},{"id":"922739b3ce93bcf120c0f190388678d6e0d7df48f09873d4bfaf60335fce30dba45c22d5e9c2402c8aedc1e7df1246072027136f9ce78d442a3d385f10350c49","href":"https://static.observableusercontent.com/files/922739b3ce93bcf120c0f190388678d6e0d7df48f09873d4bfaf60335fce30dba45c22d5e9c2402c8aedc1e7df1246072027136f9ce78d442a3d385f10350c49","name":"fig_detectron_1.jpg","type":"image/jpeg","lastModified":1621293888892.323,"size":612532},{"id":"87ba676d9261aabb9a7d1d11b7ad2394ebbe34477402ef1379eec29d6919706d60222489bcdf2ddec86ecd35183502edbba5888f2fb209df29ba9baa5360b0de","href":"https://static.observableusercontent.com/files/87ba676d9261aabb9a7d1d11b7ad2394ebbe34477402ef1379eec29d6919706d60222489bcdf2ddec86ecd35183502edbba5888f2fb209df29ba9baa5360b0de","name":"fig_cvat_0.jpg","type":"image/jpeg","lastModified":1621258546521.738,"size":305482},{"id":"e2df093d178175dff1cc140859b78e5f74418a9e37115f439fdcbb43d7de18c967821b58280365ed7c934dbba2895a8f96b916db687396f9131dbe63c3d5ed63","href":"https://static.observableusercontent.com/files/e2df093d178175dff1cc140859b78e5f74418a9e37115f439fdcbb43d7de18c967821b58280365ed7c934dbba2895a8f96b916db687396f9131dbe63c3d5ed63","name":"fig_flickr_album_1.jpg","type":"image/jpeg","lastModified":1621257457255.886,"size":873907},{"id":"c212afd50a7d42df3b3eec358c6ef1ffaf779ae6d3c25920131b8a87daa0a08bf0e8582d67ba8d17b4b9259577f9c00691fb94b348971330b1f694c8b0c718be","href":"https://static.observableusercontent.com/files/c212afd50a7d42df3b3eec358c6ef1ffaf779ae6d3c25920131b8a87daa0a08bf0e8582d67ba8d17b4b9259577f9c00691fb94b348971330b1f694c8b0c718be","name":"fig_flickr_album_0.jpg","type":"image/jpeg","lastModified":1621257455757.994,"size":600006},{"id":"7b98fd51a399cd1d55c35500eb2674b05a198dcb1df021712402e6ac15887d22378c07a1fe3eaea9227721673ff3aeb931d76ffddaf11d1b0cd9fae6c2f9f24b","href":"https://static.observableusercontent.com/files/7b98fd51a399cd1d55c35500eb2674b05a198dcb1df021712402e6ac15887d22378c07a1fe3eaea9227721673ff3aeb931d76ffddaf11d1b0cd9fae6c2f9f24b","name":"fig_parts_aligned_centered.png","type":"image/png","lastModified":1621298857731.877,"size":2490417},{"id":"071388abb54416002862dd1d99c30f53ed7b5e9e73944bbe0615b27f9b4f6e7f3b1c68211a5011524529d525e54587589fc8fe5b886922baadca22133d6302ee","href":"https://static.observableusercontent.com/files/071388abb54416002862dd1d99c30f53ed7b5e9e73944bbe0615b27f9b4f6e7f3b1c68211a5011524529d525e54587589fc8fe5b886922baadca22133d6302ee","name":"fig_stylegan2_w_interp.jpg","type":"image/jpeg","lastModified":1621483394788.373,"size":849520},{"id":"6c00367fa12e52ba34ab8fb0862fac17faacbe9db656978fa595ff9a71f7e14ccc243035a9cbb0fd56331cc203415388f2d0ad6b41db16178c390a67baee69fe","href":"https://static.observableusercontent.com/files/6c00367fa12e52ba34ab8fb0862fac17faacbe9db656978fa595ff9a71f7e14ccc243035a9cbb0fd56331cc203415388f2d0ad6b41db16178c390a67baee69fe","name":"fig_stylegan2_mix.jpg","type":"image/jpeg","lastModified":1621483294036.531,"size":1085426},{"id":"4984c2a708d348bae4dacad5e6fe977bcec6d81e41523cf7eeee8796d3126371a4337d416be78ab5638ff35bd95eea83c2ece987325d965eca17af4b276f16e1","href":"https://static.observableusercontent.com/files/4984c2a708d348bae4dacad5e6fe977bcec6d81e41523cf7eeee8796d3126371a4337d416be78ab5638ff35bd95eea83c2ece987325d965eca17af4b276f16e1","name":"fig_stylegan2_rand.jpg","type":"image/jpeg","lastModified":1621483292792.528,"size":744686}] 2f:{"title":"Paintings of Butterfly: Making Of","cells":[{"id":0,"value":"# Paintings of Butterfly: Making Of","pinned":false,"hidden":false,"mode":"md"},{"id":7,"value":"[Paintings of Butterfly](https://observablehq.com/@stwind/paintings-of-butterfly) is an experiment of generating butterfly drawings with StyleGAN2, in an attempt to mimic what scientists was doing with [CRISPR](https://entomologytoday.org/2020/01/09/crispr-cuts-through-layers-butterfly-wing-pattern-evolution/). To me there is always a certain sense of similarity between DNA manipulation and StyleGAN2 latent space navigation.\n\nThis is a documentation of the making process.","pinned":false,"hidden":false,"mode":"md"},{"id":20,"value":"## Data Collection\n\nThe [Biodiversity Heritage Library](https://www.biodiversitylibrary.org/) has a large collection of legacy literature of biodiversity, and they are available for open access. That's a good starting point. \n\nThere are about 6.5k photo albums on their [Flickr page](https://www.flickr.com/photos/biodivlibrary/albums/). I spent some hours to skim over the album covers, and picked out 200+ of them that I believed containing lots of butterfly drawings.\n\nThen I downloaded the albums and did a quick check of the thumbnails to throw out those obviously having no butterflies in it.","pinned":false,"hidden":false,"mode":"md"},{"id":44,"value":"html`

Die Gross-Schmetterlinge der Erde : eine systematische Bearbeitung der bis jetzt bekannten Gross-Schmetterlinge (1906)

\n\n

Nomenclatur und Beschreibung der Insecten in der Grafschaft Hanau-Münzenberg wie auch der Wetterau und der angränzenden Nachbarschaft dies und jenseits des Mains mit erleuchteten Kupfern

`","pinned":false,"hidden":false,"mode":"ojs"},{"id":59,"value":"## Annotating for Objects Detection\n\nIn order to extract the butterflies in the pictures, I randomly picked out few hundreds of them and used [CVAT](https://github.com/openvinotoolkit/cvat) to annotate the regions of butterflies.","pinned":false,"hidden":false,"mode":"md"},{"id":66,"value":"html`

\n\n

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":79,"value":"Then I exported the annotations as [COCO](https://cocodataset.org/) format.","pinned":false,"hidden":false,"mode":"md"},{"id":89,"value":"html`

\n\n

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":100,"value":"## Detecting Butterflies\n\nWith the annotation ready, I used [detectron2](https://github.com/facebookresearch/detectron2) to train an [Fast RCNN](https://github.com/facebookresearch/detectron2/blob/master/configs/COCO-Detection/fast_rcnn_R_50_FPN_1x.yaml) to find all the butterflies in the pictures. It took about an hour and the results are satisfying.","pinned":false,"hidden":false,"mode":"md"},{"id":105,"value":"html`

\n\n

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":113,"value":"Here are the best and worst cases, even the worst ones are good enough to contain a single butterfly.","pinned":false,"hidden":false,"mode":"md"},{"id":117,"value":"html`

\n\n

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":121,"value":"I kept the results with a score > 0.9 and with area larger than 256x256, then cropped out a square region for each of them.","pinned":false,"hidden":false,"mode":"md"},{"id":126,"value":"## Data Alignment\n\nThe data are not ready for StyleGAN2 yet, to get better results, they needed to be aligned: centering and rotation.\n\n### Finding Countours with U^2 Net\n\nFor centering, the idea was to find the contour of a butterfly with [U^2-Net](https://github.com/xuebinqin/U-2-Net) to do a salicency detection, and used the resulting mask to find the centroid.\n\nI picked out 1000 of them and used OpenCV to find a rough contour like this:","pinned":false,"hidden":false,"mode":"md"},{"id":154,"value":"html`

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":159,"value":"Then I used [Pixelmator](https://www.pixelmator.com/pro/) to do some manual refinements. Thanks to the [Applescript Support](https://www.pixelmator.com/tutorials/resources/advanced-automation-and-scripting-with-applescript/) of Pixelmator, this process can be largely automated. Masking hundreds of images took another few hours.","pinned":false,"hidden":false,"mode":"md"},{"id":165,"value":"html`

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":171,"value":"Now we are ready for the U^2-Net training. The model was trained for 30 epochs to reach an [IOU](https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/) of 0.96","pinned":false,"hidden":false,"mode":"md"},{"id":177,"value":"html`

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":181,"value":"Here are the best and worst cases. It is not perfect but good enough since we are only interested in finding centroids.","pinned":false,"hidden":false,"mode":"md"},{"id":185,"value":"html`

\n\n

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":189,"value":"Here is a subset of the centered masks.","pinned":false,"hidden":false,"mode":"md"},{"id":193,"value":"html`

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":197,"value":"## Rotational Alignment\n\nTo rotationaly align the images, we can train an neural network that outputs the rotated angles given the mask images.\n\nAgain I picked some of the masks, and built a simplistic custom labeling tool on Observable, to label rotation angles.","pinned":false,"hidden":false,"mode":"md"},{"id":205,"value":"html`

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":208,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":215,"value":"html`

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":221,"value":"Here is a subset of the rotated masks.","pinned":false,"hidden":false,"mode":"md"},{"id":225,"value":"html`

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":239,"value":"With the centroids and rotation angles, I can finally compile an aligned dataset of butterflies for StyleGAN2. There are still some inevitable bad cases but they are generally negligible.","pinned":false,"hidden":false,"mode":"md"},{"id":243,"value":"html`

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":246,"value":"## Training StyleGAN2\n\nWe are finally reach the fun part. I trained a StyleGAN2 using [stylegan2-ada](https://github.com/NVlabs/stylegan2-ada-pytorch), transferred from FFHQ512, on a Tesla V100 GPU for 22 hours (1000 kimg). Thanks to all the optimizations of NVlabs' implementation, this process was quite straight forward.\n\nHere are some quick result checks.","pinned":false,"hidden":false,"mode":"md"},{"id":277,"value":"html`

\n\n

\n\n

`;","pinned":false,"hidden":false,"mode":"ojs"},{"id":283,"value":"For more selected results, see [Paintings Of Butterfly](https://observablehq.com/@stwind/paintings-of-butterfly).","pinned":false,"hidden":false,"mode":"md"}],"files":"$W32","stdlib":"1","titling":"auto"}