-1.6 C
New York
Wednesday, February 1, 2023

Cornell Tech – Programming Tool Turns Handwriting Into Computer Code – Cornell Tech

In our first-of-its-kind, immersive Studio curriculum, all master’s students will build solutions for industry partners and develop skills in end-to-end product development and entrepreneurship.
Find out what it’s like to study, work, and live on one of the most advanced campuses in the world.
Discover how our award-winning campus helps you think better and work smarter.
Explore our high-tech venues for your next NYC event.
Walk our campus from wherever you are in the world.
The Jacobs Technion-Cornell Institute is a place for dynamic experimentation, where the boundaries of academia are expanded.
2 West Loop Road, New York, NY 10044
News /
Categories 
Computer Science, Research
By Louis DiPietro, Cornell Ann S. Bowers College of Computing and Information Science
A Cornell team has created an interface that allows users to handwrite and sketch within computer code – a challenge to conventional coding, which typically relies on typing.
The pen-based interface, called Notate, lets users of computational, digital notebooks – such as Jupyter notebooks, which are web-based and interactive – to open drawing canvases and handwrite diagrams within lines of traditional, digitized computer code.
Powered by a deep learning model, the interface bridges handwritten and textual programming contexts: Notation in the handwritten diagram can reference textual code and vice versa. For instance, Notate recognizes handwritten programming symbols, like “n,” and then links them up to their typewritten equivalents. In a case study, users drew quantum circuit diagrams inside of Jupyter notebook code cells.
The tool was described in “Notational Programming for Notebook Environments: A Case Study with Quantum Circuits,” presented at the ACM Symposium on User Interface Software and Technology, held Oct. 29 through Nov. 2 in Bend, Oregon. The paper, whose lead author is Ian Arawjo, doctoral student in the field of information science, won an honorable mention at the conference.
“A system like this would be great for data science, specifically with sketching plots and charts that then inter-operate with textual code,” Arawjo said. “Our work shows that the current infrastructure of programming is actually holding us back. People are ready for this type of feature, but developers of interfaces for typing code need to take note of this and support images and graphical interfaces inside code.”
Arawjo said the work demonstrates a new path forward by introducing artificial intelligence-powered, pen-based coding at a time when drawing tablets are becoming more widely used.
“Tools like Notate are important because they open us up to new ways to think about what programming is, and how different tools and representational practices can change that perspective,” said Tapan Parikh, associate professor of information science at Cornell Tech and a paper co-author.
Other co-authors are: Anthony DeArmas ’22; Michael Roberts, a doctoral student in the field of computer science; and Shrutarshi Basu, Ph.D. ’18, currently a visiting assistant professor of computer science at Middlebury College.
Louis DiPietro is a writer for the Cornell Ann S. Bowers College of Computing and Information Science.
This story originally appeared in the Cornell Chronicle.
Our work shows that the current infrastructure of programming is actually holding us back. People are ready for this type of feature, but developers of interfaces for typing code need to take note of this and support images and graphical interfaces inside code.”
Ian Arawjo Doctoral Student
Stay in the loop on all things Cornell Tech
2 West Loop Road, New York, NY 10044
© 2020 Cornell Tech. All right reserved.

source

Related Articles

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Latest Articles