The entire pipeline for producing digital excursions (known as DIGITOUR and proven in Determine 3) is as follows.

2.1 Tag Placement and Picture Capturing

Whereas making a digital tour for any real-estate property, it’s important to click on 360◦ photos from completely different property areas akin to bed room, lounge, kitchen, and so forth., then mechanically stitching them collectively to have a “walkthrough” expertise with out being bodily current on the location. Subsequently, to attach a number of equirectangular photos, we suggest inserting paper tags on the ground overlaying every location of the property, and inserting the digicam (in our case, we used Ricoh-Theta) in the midst of the scene to seize the entire website (entrance, again, left, proper and backside).

Furthermore, we be certain that the scene is obvious of all noisy components akin to dim lighting and ‘undesirable’ artifacts for higher mannequin coaching and inference. As proven in Determine 4, now we have standardized the tags with dimensions of 6” × 6” with two properties:

1. they’re numbered which is able to assist the photographer place tags in sequence and
2. they’re bi-colored to formulate the digit recognition downside as classification job and facilitate higher studying of downstream laptop imaginative and prescient duties (i.e. tag detection and digit recognition).

Please notice that completely different colours are assigned to every digit (from 0 to 9) utilizing the HSV coloration scheme and main digit of a tag has a black circle to tell apart it from the trailing digit as proven in Determine 4. The instinct behind standardizing the paper tags is that it permits to coach tag detection and digit recognition fashions, that are invariant to distortions, tag placement angle, reflection from lighting sources, blur circumstances, and digicam high quality.

2.2 Mapping Equirectangular Picture to Cubemap Projection

An equirectangular picture consists of a single picture whose width and peak correlate as 2 : 1 (as proven in Determine 1). In our case, photos are clicked utilizing a Ricoh-Theta digicam having dimensions 4096 × 2048 × 3. Sometimes, every level in an equirectangular picture corresponds to a degree in a sphere, and the pictures are stretched within the ‘latitude’ route. For the reason that contents of an equirectangular picture are distorted, it turns into difficult to detect tags and acknowledge digits instantly from it. For instance, in Determine 1, the tag is stretched on the middle-bottom of the picture. Subsequently, it’s essential to map the picture to a less-distorted projection and change again to the unique equirectangular picture to construct the digital tour.

On this work, we suggest to make use of dice map projection, which is a set of six photos representing six faces of a dice. Right here, each level within the spherical coordinate area corresponds to a degree within the face of the dice. As proven in Determine 5, we map the equirectangular picture to 6 faces (left, proper, entrance, again, prime and backside) of a dice having dimensions 1024 × 1024 × 3 utilizing python library vrProjector.

2.3 Tag Detection

As soon as we get the six photos akin to the faces of a dice, we detect the situation of tags positioned in every picture. For tag detection, now we have used the state-of-the-art YOLOv5 mannequin. We initialized the community with COCO weights adopted by coaching on our dataset. As proven in Determine 6, the mannequin takes a picture as enter and returns the detected tag together with coordinates of the bounding field and confidence of the prediction. The mannequin is skilled on our dataset for 100 epochs with a batch dimension of 32.

2.4 Digit Recognition

For the detected tags, we have to acknowledge the digits from the tag. In a real-world atmosphere, the detected tags may need incorrect orientation, poor luminosity, reflection from the bulbs within the room, and so forth. Because of these causes, it’s difficult to make use of Optical Character Recognition (OCR) engines to have good digit recognition efficiency. Subsequently, now we have used a customized MobileNet mannequin initialized on Imagenet weights, which makes use of coloration data in tags for digit recognition. Within the proposed structure, now we have changed the ultimate classification block of the unique MobileNet with the dropout layer and dense layer with 20 nodes representing our tags from 1 to twenty. Determine 7 illustrates the proposed structure. For coaching the mannequin, now we have used Adam as an optimizer with a studying fee of 0.001 and a discounting issue () to be 0.1. We now have used categorical cross-entropy as a loss operate and set the batch dimension to 64 and the variety of epochs to 50.

2.5 Mapping tag coordinates to the unique 360◦ Picture and Virutal Tour Creation

As soon as now we have detected the tags and acknowledged the digits we use the python library vrProjector to map the dice map coordinates again to the unique equirectangular picture. An instance output is proven in Determine 8. For every equirectangular picture, the detected tags type the nodes of a graph with an edge between them. Within the subsequent equirectangular photos of a property, the graph will get populated with extra nodes, as extra tags are detected. Lastly, we join a number of equirectangular photos in sequence primarily based on acknowledged digits written on them and the ensuing graph is the
digital tour as proven in Determine 2(b).

We now have collected knowledge by inserting tags and clicking equirectangular photos utilizing Ricoh-Theta digicam for a number of residential properties in Gurugram, India (Tier 1 metropolis). Whereas gathering photos we made positive that sure circumstances have been met akin to all doorways have been opened, lights have been turned on, ‘undesirable’ objects have been eliminated and the tags have been positioned overlaying every space of the property. Following these directions, common variety of equirectangular photos clicked per residential property was 7 or 8. Lastly, now we have validated our method on the next generated datasets (primarily based on background coloration of the tags).

1. Inexperienced Coloured Tags: We now have saved the background coloration of those tags (numbered 1 to twenty) to be inexperienced. We now have collected 1572 equirectangular photos from 212 properties. As soon as we convert these equirectangular photos to cubemap projection, we get 9432 photos (akin to dice faces). Since not the entire dice faces have tags (for e.g. prime face), we get 1503 photos with atleast one tag.
2. Proposed Bi-colored Tags (see Determine 4): For these tags, now we have collected 2654 equirectangular photos from 350 properties. Lastly, we acquired 2896 photos (akin to dice faces) with atleast one tag.

Lastly, we label the tags current in dice map projection photos utilizing LabelImg which is an open-source instrument for labeling photos in a number of codecs akin to Pascal VOC and YOLO. For all of the experiments, we reserved 20% of knowledge for testing and the remaining for coaching.

For any enter picture, we first detect the tags and eventually acknowledge the digits written on the tags. From this we have been capable of establish the true positives (tags detected and browse accurately), false positives (tags detected however learn incorrectly) and false negatives (tags not detected). The obtained mAP, Precision, Recall and f1-score at 0.5 IoU threshold are 88.12, 93.83, 97.89 and 95.81 respectively. Please notice that every one metrics are averaged (weighted) over all of the 20 courses. If all tags throughout all equirectangular photos of a property are detected and browse accurately, we obtain a 100% correct digital tour since all nodes of the graph are detected and related with their acceptable edges. In our experiments, we have been capable of precisely generate 100% correct digital tour for 94.55% of the properties. The inaccuracies have been as a result of presence of colourful artifacts that have been falsely detected as tags; and dangerous lightning circumstances.

Determine 9 demonstrates the efficiency of Yolov5 mannequin for tag detection primarily based on inexperienced coloured and bi-colored tags. Additional, experiments and comparability of fashions on digit recognition is proven in Determine 10.

We suggest an end-to-end pipeline (DIGITOUR) for mechanically producing digital excursions for real-estate properties. For any such property, we first place the proposed bi-colored paper tags overlaying every space of the property. Then, we click on equirectangular photos, adopted by mapping these photos to much less distorted cubemap photos. As soon as we get the six photos akin to dice faces, we detect the situation of tags utilizing the YOLOv5 mannequin, adopted by digit recognition utilizing the MobileNet mannequin. The subsequent step is to map the detected coordinates together with acknowledged digits to the unique equirectangular photos. Lastly, we sew collectively all of the equirectangular photos to construct a digital tour. We now have validated our pipeline on a real-world dataset and proven that the end-to-end pipeline efficiency is 88.12 and 95.81 by way of mAP and f1-score at 0.5 IoU threshold averaged (weighted) over all courses.

In the event you discover our work useful and put it to use in your initiatives, we kindly request that you simply cite it. 😊

@inproceedings{chhikara2023digitour,
title={Digitour: Automated digital excursions for real-estate properties},
writer={Chhikara, Prateek and Kuhar, Harshul and Goyal, Anil and Sharma, Chirag},
booktitle={Proceedings of the sixth Joint Worldwide Convention on Knowledge Science & Administration of Knowledge (tenth ACM IKDD CODS and twenty eighth COMAD)},
pages={223--227},
12 months={2023}
}

[1] Dragomir Anguelov, Carole Dulong, Daniel Filip, Christian Frueh, Stéphane Lafon, Richard Lyon, Abhijit Ogale, Luc Vincent, and Josh Weaver. 2010. Google avenue view: Capturing the world at avenue stage. Laptop 43, 6 (2010), 32–38.

[2] Mohamad Zaidi Sulaiman, Mohd Nasiruddin Abdul Aziz, Mohd Haidar Abu Bakar, Nur Akma Halili, and Muhammad Asri Azuddin. 2020. Matterport: digital tour as a brand new advertising and marketing method in actual property enterprise throughout pandemic COVID-19. In Worldwide Convention of Innovation in Media and Visible Design (IMDES 2020). Atlantis Press, 221–226.

[3] Chinu Subudhi. 2021. Reducing-Edge 360-Diploma Digital Excursions. https://www.mindtree.com/insights/sources/cutting-edge-360-degree-virtual-tours

[4] Glenn Jocher, Ayush Chaurasia, Alex Stoken, Jirka Borovec, NanoCode012, Yonghye Kwon, TaoXie, Jiacong Fang, imyhxy, Kalen Michael, Lorna, Abhiram V, Diego Montes, Jebastin Nadar, Laughing, tkianai, yxNONG, Piotr Skalski, Zhiqiang Wang, Adam Hogan, Cristi Fati, Lorenzo Mammana, AlexWang1900, Deep Patel, Ding Yiwei, Felix You, Jan Hajek, Laurentiu Diaconu, and Mai Thanh Minh. 2022. ultralytics/yolov5: v6.1 — TensorRT, TensorFlow Edge TPU and OpenVINO Export and Inference.

[5] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and LiangChieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE convention on laptop imaginative and prescient and sample recognition. 4510–4520.