yining shi on approaching technology creatively and accessibly

This week we spoke to Yining Shi, one of the main instructors for the Bots & Machine Learning program, which takes place this July in Berlin. In this interview she reflects on her trajectory, the importance of educational tools, and what we can expect this summer.

Tell us a bit about yourself Yining, what brought you to where you are now?

Well, I did my undergrad in Electrical Engineering and Computer Science and my Masters at NYU’s Interactive Telecommunications Program, where I mostly focus on creative coding and machine learning. I like building tools to craft a better learning experience for people. I like solving complicated engineering problems to enable creative outputs. I also like the idea of making complex concepts and technology more approachable for beginners. Currently, I’m an adjunct professor at NYU where I teach a class called Machine Learning for the Web. I work at Sourcemap as a senior software engineer.

“I also like the idea of making complex concepts and technology more approachable for beginners”.

Where are you currently based?

I’m based in Brooklyn, New York City.

I see some of your works are based on data, physical computing and fabrication. What are some of the overarching themes in your work? What intrigues you?

Even though these projects use different technology, they are mostly based on the idea of visualizing data to express ourselves in a creative way. What intrigues me is exploring millions of different media to express our ideas. I’m always curious about new technologies and new experiences.

For example, Friendship.am is an expression of my digital persona that visualizes 10,000 of my messages. Every circle represents one message, and they are arranged around a clock. The size of each circle depends on the length of the message. I also color coded keywords in every message. For example, ‘What are you doing?’ is blue, and ‘haha’ is green. This was an abstract visual portrait of my digital communication habits and by looking at what messages I sent, when I sent them, and how they change over time, I got a lot of insights about my digital persona that I was previously unaware of. And so I also became very interested in personal data and how can I make them more meaningful in people’s everyday lives.

Puppy lamp is another example. It’s a playful experience that allows friends to express themselves in a new way. It is a digital interaction designed to work with a physical object. People can use the app to send color and text message to their friends and light up their friends’ Puppy Lamp. It is a way to express feelings to your friends with colors, lights and environmental ambience.

Can you tell us a bit more about p5.playground, and your interest in building educational tools?

p5.playground is my thesis at ITP, NYU. It’s an interactive programming tool for designers and beginners to understand drawing functions in p5.js.

It has two modes, live coding mode (when you change the code, the sketch will be updated in real time) and the playground code, when you manipulate the shapes on the canvas like you do in photoshop or illustrator, the code will be automatically updated.

If you move and resize the shapes on the canvas, the code that generates the shapes will be automatically updated.

I remember at the first class at Intro to Computational Media(ICM) class at ITP, we learned how to use Processing to draw shapes. I noticed that some people who are new to programming found it hard to draw complex shapes with code. After talking to them, I realized that it was because people might have difficulty understanding the coordinate systems: where is the origin, x,y axis.

p5 playground tries to use this WYSIWYG(What you see is what you get) interface to reinforce a cognitive link between the code and the shapes on the canvas. Now, p5.playground is used as a learning tool in the ICM class at ITP to help students understand how to draw shapes with code.

I love building educational tools because I really like the idea of making complicated concepts and technology more approachable to everyone who is interested in learning it. Not everyone needs to become a engineer when they grow up, but if anyone who wants to learn about a kind of technology, they should be able to learn it easily without any struggle.

Technology is everywhere in our life, and problem-solving skill is important to us, I want to make the learning experience a little easier to people especially students who don’t have access to a lot of educational resource. I also enjoy the moments when people told me that the tool is helpful to them or they made some projects using this tool. That’s the biggest reward moment for me.

“Not everyone needs to become a engineer when they grow up, but if anyone who wants to learn about a kind of technology, they should be able to learn it easily without any struggle”.

You are currently teaching Machine Learning for the Web at NYU, what are the topics which you most enjoy teaching?

Machine Learning for the Web is a class about using ml5.js and tensorflow.js to create interactive web applications. It starts with running pre-trained models and re-training models in the browser using high-level APIs from ml5.js, as well as explore the Layer APIs from TensorFlow.js to create models from scratch using custom data.

I really like teaching week 3: Build your own CNN model with tf.js. We start with using a pre-trained DoodleNet model to classify 345 doodle classes, and use transfer learning to customize our own classes, then we use tensorflow.js’s layer API to build our own model and train it in the browser. I also like teaching Style transfer and Pix2pix.

Now, a more human-oriented question! What is your all time favourite meal?

It has to be Sichuan chinese food. My favorite is a kind of dish is called hotpot. This is the hotpot my mom made.

How do you express yourself creatively through code?

Most of my recent projects have been all about using Machine Learning models to make some creative and interactive applications in the browsers.

Style transfer with ml5 is an interactive demo of style transfer, where it can recast the content of one image in the style of another image. people can either use the webcam, or upload an image, and choose a style from the artwork, and it will generate a new image with the style they chose. See live demo here.

Pix2pix Edges2Pikachu is an interactive drawing tool that automatically colors your Pikachu drawings in real time. It’s built with tensorflow.js. You can play with the live demo here.

KNN Image Classifier with ml5 is a an interactive demo that can recognize customized classes through webcam. For example, it can recognize rock, paper and scissors from your webcam.

DoodleNet is one of my most recent project. It can recognize your drawings. It’s trained on all 345 categories from Quickdraw dataset, 50000 images per class. You can play with live demo here.

You published Jumpstarting the Arduino 101, what do you like about this tool?

My journey with machine learning started with this book. It’s a book that helps you create projects with Arduino 101 using pattern matching & neural networks via the Web Bluetooth API.

It includes 4 projects, each project builds upon the concepts of the previous one. The last project is a gesture recognition media controller.

For example, people can train a complex gesture to start the music, and then train another gesture to stop the music. This is designed to be a starter example, which people can extend and make their own versions of it.

I really like Arduino 101 because for a board that cost $30, it offered a lot of features out of the box like Bluetooth LE and motion sensing, but my personal favorite feature was that the board had a dedicated machine learning chip and came with an open-source pattern matching library. Machine learning can be a classic example of a topic that sounds intimidating, but the ease with which beginners could do complex pattern matching and gesture recognition projects with this board demonstrated the effect accessibility has over learning.

Can we talk a bit about Bots & Machine Learning? What will you be teaching us here in Berlin and what do you hope students will get out of the program?

Yes! I’m very excited about teaching this course with Matthew Plummer Fernandez!

We will talk about common and popular machine learning models, how do they work, how to train these models, and their use cases in creative projects. The output of the class will be interactive ML applications.

The topics that will be covered are Image/Sound/Doodle Classification, Face/Pose Recognition, Image Style Transfer, pix2pix Image Transformation, and Image Synthesis. The techniques and neural networks we will use and build are Transfer Learning, Convolutional Neural Network, Generative Adversarial Network and more. We will use tools and frameworks like Runway, tensorflow.js, ml5.js and tensorflow.

I really look forward to meeting you in Berlin, explore all the ML models and technique, and build creative projects together!

What is the change you want to see in the world?

One thing that I really want to see is more people getting having easier access to education. There are a lot of facets to make this change but what I like focusing on is to make the process of learning new things more enjoyable and fascinating. I’d love to create more educational content and help beginners to learn. I am hoping to make more video tutorials about programming that people from any part of the world could watch and build their own projects based on the examples from the video.