-1.3 C
New York
Monday, November 21, 2022

CodeToon: A tool to automatically create stories and comics – Tech Xplore


Forget Password?
Learn more
share this!
68
31
Share
Email
October 13, 2022 feature
by Ingrid Fadelli , Tech Xplore
Three researchers at University of Waterloo in Canada have recently developed a unique comic authoring tool that can automatically create stories and comics based on code. This system, called CodeToon, could open interesting possibilities for code-driven storytelling, the process of using computational code to create new stories.

Sangho Suh, one of the authors, came up with the idea for CodeToon and then worked on its development in collaboration with his colleague Jian Zhao, under the supervision of Prof. Edith Law. Their paper, set to be presented in a few weeks at the ACM Symposium on User Interface Software and Technology (UIST), the premier forum for innovations in human-computer interfaces, builds on some of their team’s previous efforts, including the development of a pedagogical tool called Coding Strip and another study leveraging generative conversational AI to teach computational thinking.
“We found that using comics is an effective means for teaching abstract programming concepts, but crafting the comics is tedious and time-consuming,” Jian Zhao, one of the researchers who carried out the study, told TechXplore. “We also identified the opportunities in advanced AI technologies for automating some steps of this task. Thus, we built CodeToon, a comic authoring tool that facilitates this -driven storytelling process.”
Coding Strip, the pedagogical tool previously created by Sangho Suh and his colleagues at University of Waterloo, was specifically designed to teach programming concepts using comics. CodeToon takes this idea one step further, by automatically creating code-based comics and stories that could be used to train computer science students.
“Inspired by the ladder of abstraction and its relationship to computational thinking, coding strip helps learners move between concrete (comic) and abstract (code) levels of abstraction,” Sangho Suh, one of the researchers who carried out the study, told TechXplore. “Creating coding strips, however, can be time-consuming and laborious, so the primary objective of this project was to make the authoring process more efficient and effortless.”
CodeToon can facilitate code-driven storytelling in two distinct ways. Firstly, it can automatically generate a story template using code. Subsequently, the tool can generate comics based on these code-based story templates.
“CodeToon is an interactive and intelligent comic authoring tool that supports creators to generate stories from code (code→story) and design comics from stories (story→comic),” Zhao and Suh said. “Since comics accurately represent the code semantics and execution process, it enables learners to learn the computational concepts and procedures in an engaging yet effective manner.”
The researchers evaluated the effectiveness of their comic authoring tool in a two-part user study. In the first part, 24 participants with a basic understanding of programming were asked to either use CodeToon or use a series of templates to create story ideas and generate matching comics. Subsequently, all the participants took part in a short interview, where they were asked what worked and what did not work for them.
In the second part of the evaluation, 20 participants who could read and understand Python code were asked to look at 8 coding strips—created using CodeToon or with a series of templates—and share their opinions about how well they mapped to code, illustrated coding concepts, and how useful they thought they might be for teaching.
Based on the interview responses, most participants felt that the 1-to-1 mapping of code onto stories and comics could be an effective teaching strategy, particularly when using comics to alternatively represent code. In addition, those who used CodeToon to generate their comics shared positive feedback about its performance in creating code-based stories and comics.
“Our findings confirm previous literature on multiple representational systems, which recommends making the mapping between multiple representations clear,” Zhao said. “Our insights also shed light on other research areas, such as data and stats comics, which can also leverage this approach to automatically generate comics for learning and teaching concepts in their domains.”
Notably, Suh, Zhao, and Law introduced a new way of looking at code that could have interesting implications. In the future, the same idea could perhaps be applied to other computational and mathematical languages. For instance, future studies could explore the possibility of generating comics and stories based on mathematical equations.
“The most notable achievement of our study, on the surface, may be the efficient process for generating comics from code,” Suh said “But on a deeper level, I think it is the notion of code-driven storytelling. In CodeToon, code drives storytelling: the code’s abstract structure embodying computational ideas such as if-else logic structure becomes the structure of the story and comic. In other words, code determines the sequence and overall plot of the story.”
The tool created by Suh, Zhao, and Law could soon be used in universities and colleges to make learning to code more engaging. In the future, the researchers would thus like to deploy their tool in real classroom environments to better assess its usefulness. In addition, they plan to investigate the possibility of using generative deep learning models, such as GPT-3 and Codex, to fill any gaps in stories and code.
“For example, based on the drawings on the canvas, the system could generate new drawings to add to the existing comic and provide design guidance in real-time,” Zhao explained. “Another possible research direction would be to enhance the authoring process with a more diverse set of sketches and resources, more customization and flexibility, as well as the support for more programming languages.”
In his upcoming studies, Suh would also like to investigate the possible use of mathematical languages to drive storytelling. This could ultimately improve the way in which lecturers and students make sense of abstract concepts, complex equations, and procedures.
“There are many exciting research directions we plan to explore in our next studies,” Suh added. “One is to further improve CodeToon, for example, by ensuring that it supports other programming languages and features. Another is to research how we can use CodeToon to integrate art into programming education.”


Explore further

A system that automatically generates comic books from movies and other videos


More information: Sangho Suh, Jian Zhao, Edith Law, CodeToon: Story ideation, auto comic generation, and structure mapping for code-driven storytelling. arXiv:2208.12981v1 [cs.HC], arxiv.org/abs/2208.12981
Explore further
edithlaw.ca/
edithlaw.ca/projects/codingstrip.html
ACM Symposium on User Interface Software and Technology (UIST) 2022. uist.acm.org/uist2022/program.html
© 2022 Science X Network
Facebook
Twitter
Email
Feedback to editors
Oct 14, 2022
0
Oct 13, 2022
0
Oct 13, 2022
0
Oct 12, 2022
0
Oct 11, 2022
0
Oct 14, 2022
Oct 14, 2022
Oct 14, 2022
Oct 14, 2022
Oct 14, 2022
Oct 14, 2022
Oct 13, 2022
Feb 18, 2021
Jun 16, 2020
Aug 12, 2021
Feb 05, 2019
Apr 01, 2019
Jul 14, 2022
Oct 14, 2022
Oct 13, 2022
Oct 12, 2022
Oct 12, 2022
Oct 11, 2022
Oct 11, 2022
Use this form if you have come across a typo, inaccuracy or would like to send an edit request for the content on this page. For general inquiries, please use our contact form. For general feedback, use the public comments section below (please adhere to guidelines).
Please select the most appropriate category to facilitate processing of your request
Thank you for taking time to provide your feedback to the editors.
Your feedback is important to us. However, we do not guarantee individual replies due to the high volume of messages.
Your email address is used only to let the recipient know who sent the email. Neither your address nor the recipient’s address will be used for any other purpose. The information you enter will appear in your e-mail message and is not retained by Tech Xplore in any form.

Daily science news on research developments and the latest scientific innovations
Medical research advances and health news
The most comprehensive sci-tech news coverage on the web
This site uses cookies to assist with navigation, analyse your use of our services, collect data for ads personalisation and provide content from third parties. By using our site, you acknowledge that you have read and understand our Privacy Policy and Terms of Use.

source

Related Articles

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Latest Articles