1. Recipe Customization

Recipe customization is essential as a result of connection between meals, customs, and particular person preferences. Moreover, it turns into important when addressing allergic reactions or dietary restrictions. Surprisingly, regardless of the evident demand, present literature lacks devoted efforts in recipe customization. Our work goals to bridge the analysis hole by enabling personalised recipe customization, contemplating particular person style profiles and dietary restrictions.

To information future analysis on this space, we showcase the power of FIRE to help a recipe customization strategy that focuses on a variety of subjects (e.g., ingredient substitute, style adjustment, calorie adjustment, cooking time adaptation) to check few-shot efficiency completely. As proven within the purple a part of Determine 5, we take away components to trim the potatoes from the recipe. Two sentences associated to potatoes are deleted within the modified model, and one sentence is modified to make sure consistency. Particularly, we carry out ingredient addition to exchange ‘cheese’ with ‘cheddar cheese’ and acknowledge that it ought to be added earlier than baking, ensuing within the modified sentence ‘Sprinkle half every of cheddar cheese and onions.’

2. Producing Machine Code for Picture-based Recipes

Changing recipes to machine code permits automation, scalability, and integration with numerous present methods, thus lowering guide intervention, saving labor prices, and lowering human errors whereas getting ready the meals. To facilitate this process, we mix FIRE’s recipe era energy with the power of huge LMs to control code-style prompts for structural duties [14]. We present an instance strategy for producing Python-style code representations of recipes developed by FIRE, by prompting GPT-3 (please seek advice from orange half in Determine 5).

We launched FIRE, a technique tailor-made for meals computing, specializing in producing meals titles, extracting components, and producing cooking directions solely from picture inputs. We leveraged latest CV and language modeling developments to realize superior efficiency towards strong baselines. Moreover, we demonstrated sensible purposes of FIRE for recipe customization and recipe-to-code era, showcasing the adaptability and automation potential of our strategy.

We record three challenges that ought to be addressed in future analysis:

1. Current and proposed recipe era fashions lack a dependable mechanism to confirm the accuracy of the generated recipes. Standard analysis metrics fall quick on this side. Therefore, we wish to create a brand new metric that assesses the coherence and plausibility of recipes, offering a extra thorough analysis.
2. The range and availability of recipes are influenced by geographical, climatic, and spiritual elements, which can restrict their applicability. Incorporating data graphs that account for these contextual elements and ingredient relationships can provide various ingredient solutions, addressing this problem.
3. Hallucination in recipe era utilizing language and imaginative and prescient fashions poses a big problem. Future work would discover the state-tracking strategies to enhance the era course of, guaranteeing the manufacturing of extra life like and correct recipes.

I hope this overview has offered you the perception into the inspiration and growth of FIRE, our progressive device for changing meals photos into detailed recipes. For a extra in-depth exploration of our strategy, I invite you to take a look at our full paper, which is revealed within the IEEE/CVF Winter Convention on Functions of Pc Imaginative and prescient (WACV) — 2024. If our analysis contribute to your work, we might be comfortable should you cite it. 😊

Paper Hyperlink: https://openaccess.thecvf.com/content material/WACV2024/html/Chhikara_FIRE_Food_Image_to_REcipe_Generation_WACV_2024_paper.html

@InProceedings{Chhikara_2024_WACV,
creator    = {Chhikara, Prateek and Chaurasia, Dhiraj and Jiang, Yifan and Masur, Omkar and Ilievski, Filip},
title     = {FIRE: Meals Picture to REcipe Era},
booktitle = {Proceedings of the IEEE/CVF Winter Convention on Functions of Pc Imaginative and prescient (WACV)},
month     = {January},
yr      = {2024},
pages     = {8184-8194}
}

[1] Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, and Ramesh Jain. A survey on meals computing. ACM Comput. Surv., 52(5), sep 2019.

[2] Sutter Well being. Consuming Properly for Psychological Well being. https://www.sutterhealth.org/well being/diet/eating-wellfor-mental-health. Accessed on March 24, 2023.

[3] Kiely Kuligowski. 12 Causes to Use Instagram for Your Enterprise. https://www.enterprise.com/articles/10-reasons-touse-instagram-for-business/. Accessed on Might 12, 2023.

[4] Dim P. Papadopoulos, Enrique Mora, Nadiia Chepurko, Kuan Wei Huang, Ferda Ofli, and Antonio Torralba. Studying program representations for meals photos and cooking recipes, 2022.

[5] Sundaram Gunasekaran. Pc imaginative and prescient know-how for meals high quality assurance. Traits in Meals Science & Know-how, 7(8):245–256, 1996.

[6] Yoshiyuki Kawano and Keiji Yanai. Meals picture recognition with deep convolutional options. In Proceedings of the 2014 ACM Worldwide Joint Convention on Pervasive and Ubiquitous Computing: Adjunct Publication, pages 589– 593, 2014

[7] Amaia Salvador, Michal Drozdzal, Xavier Giro-i Nieto, and ´ Adriana Romero. Inverse cooking: Recipe era from meals photos. In Proceedings of the IEEE/CVF Convention on Pc Imaginative and prescient and Sample Recognition, pages 10453– 10462, 2019.

[8] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and era. In Worldwide Convention on Machine Studying, pages 12888– 12900. PMLR, 2022.

[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. A picture is value 16×16 phrases: Transformers for picture recognition at scale. arXiv preprint arXiv:2010.11929, 2020.

[10] Prateek Chhikara, Ujjwal Pasupulety, John Marshall, Dhiraj Chaurasia, and Shweta Kumari. Privateness conscious questionanswering system for on-line psychological well being threat evaluation. In The twenty second Workshop on Biomedical Pure Language Processing and BioNLP Shared Duties, pages 215– 222, Toronto, Canada, July 2023. Affiliation for Computational Linguistics.

[11] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the boundaries of switch studying with a unified text-to-text transformer. The Journal of Machine Studying Analysis, 21(1):5485–5551, 2020.

[12] Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzman, Luke Zettlemoyer, and Marjan ´ Ghazvininejad. Detecting hallucinated content material in conditional neural sequence era. In Findings of the Affiliation for Computational Linguistics: ACL-IJCNLP 2021, pages 1393–1404, 2021.

[13] Mehrdad Farahani and Kartik Godawat and Haswanth Aekula and Deepak Pandian and Nicholas Broad. Chef Transformer. https://huggingface.co/flax-community/t5- recipe-generation. Accessed on April 12, 2023.

[14] Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language fashions of code are few-shot commonsense learners. In Findings of the Affiliation for Computational Linguistics: EMNLP 2022, 2022