This week we learned how to use the sequencer in Unreal and create planes with media textures for displaying video in the world. I brought in my favorite character from Mixamo to do some dancing for a virtual audience. I wanted to build cathedral like walls using Quixel assets as well. I had some technical hiccups this week but look forward to continue building more elaborate worlds as I get more comfortable with the software.
It took a minute to get up and running with Colab but I’m enjoying seeing the code and understanding the process a bit better. I started with style transfers again in order to get familiar and it dawned on me that I should start experimenting with some Cyanotype prints that I’ve made.
I think the first style transfer came out better than the second but I’m excited to keep experimenting with this, and making specific style inputs to make my own custom cyanotype transfer. The way that the foliage in the style transfer image translates is interesting and I’m going to start using photoshop to create floral patterns (I find they create an interesting pattern once converted in a cyanotype print).
I’m very excited to keep testing out models in different colab notebooks, as well as begin to create ideas for creating custom content based on the results I’m getting already.
My name is Will Politan and this is where I will be documenting the work I do for Mixed Reality Filmmaking. I am a second year ITP student and I got my BFA from NYU in filmmaking with a background as a cinematographer. I’m very interested in understanding these tools for the work I already do, but also to experiment in creating 3D assets and new worlds as well. I took Performative Avatars last semester so I have some basic understanding of avatar creation and the Unreal Engine, but I’m excited to learn more about the production workflow.
After getting set up in RunwayML I trained my first image of a black and white scene of ducks. I trained it on the Picasso model.
input – picasso model
output – picasso model
I was really excited to train different models on this photograph of the Jersey Devil at Lucille’s in New Jersey, especially the MUNIT model. I thought a menacing figure comprised of flowers would be a fun experiment. First I trained with two different models (sketch model & picasso) before changing the “style” function number for tons of variation on the MUNIT model.
input – for all following output images trained on different models
output- photo sketch model
output – picasso model
output – picasso model
output – munit model for subsequent images
After working with the MUNIT model I wanted to train a color photograph, so I used this beach sunset image to train the subsequent images.
input – for following output images
OUTPUT – KANDINSKY style transfer
output – cubist style transfer
output – hokusai style transfer
output – wu guanzho style transfer
I really loved what the Hokusai style transfer looked like and wanted to train it on this photograph of a lake at Cedar Bridge Tavern in New Jersey. It looks like the darkest areas of the image turned into just blurred light so in the future I would adjust the brightness to try and avoid that effect with this model.
input – hokusai style transfer
output – hokusai style transfer
Jude updated the questionnaire part of our project with a mock run through.
Irwin finished the card design and I printed them and cut them to begin working out how it feels in the physical space.
Card design by Irwin.
In the image below you can see that I chose virtual reality. I landed on some cards I thought I would need for a virtual reality production. This included investment, script writing, storyboarding, wireframing, animation, rigging, motion capture, testing, editing, marketing and screening. I really enjoyed how the extra descriptions can provide even more context for exactly what each step is requiring.
Actually printing out the deck was extremely helpful for getting a feel for how the system will work. While we didn’t have time to test on users yet I found in the short period of time that I was already realizing ways in which we can improve. Firstly, I think the use of real cards and numbers was very nice to see (also providing a dual purpose) but might slightly take away from the productivity of the game. If we wanted to keep it more fun, I’d say we could keep the traditional card numbers but if we wanted to make it more serious it might be better to brand the cards only on their categories.
To make diving into the brainstorming easier we could color code to only categories consisting of “channels, pre-production, production, post-production.” I found myself spreading out all of the cards in order to organize them to my liking. It was really great to see how we could start connecting the dots in order to visualize the process for a production. I feel like often the tools available to us for scheduling and many of these tasks are never visualized. It’s most often a to-do list, or excel spreadsheet with a timeline. It was really nice to piece a production together using the cards.
I am proud of the work we’ve done on this project so far. I think doing more user testing in person with the physical cards will be extremely valuable feedback. I also think getting feedback on the categories would be very helpful. We may need to adjust how many cards we have in each category to be more useful. Also, I think running users through both the website and cards back to back is necessary to see how seamless that process can be.
This week we continued to further our research into the best practices for choosing/preparing a visual story within the landscape of emerging media. We also identified the aspects of production we will be including for our card deck. In order to do this we broke down the corresponding details into four categories: pre-production, production, post-production, and channels.
- Script writing
- Recruitment / Casting
- Character design
- Financial / investment
- Location Scout
- Order production insurance
- Create equipment list
- Production planning
- Creation of assets
- Building infrastructure
- Filming / Cinematography / Photography
- User research/ User experience
- 3D modeling
- Drawing / illustration
- Motion Capture
- Sound design
- Color grading
- Traditional print media
- Small print materials
- Outdoor advertising
- Social media
- Physical Space
These will be turned into cards with vector graphics.
Mockup design created by Irwin.
This project is partially inspired by the production book I had to create and use in the undergraduate film program. It was extremely intensive to fill out and is necessary to obtain the student production insurance. While it is not necessary to show such documentation to purchase production insurance outside of school, it is extremely helpful and necessary to think through all of the same details. This process we are creating can be used by anyone to achieve two results: better understanding of the project they are embarking on and whether their current medium is best suited for that.
This culminated in our idea taking two forms, an online questionnaire and a card deck. The card deck will include all of the principles listed above. This is to help the producer/stakeholder plan out the various aspects of production. The questionnaire will serve as the guide for choosing which medium is likely the best to carry out.
Jude created a mockup of what the questionnaire will look like.
In the video you can see that this portion of the project will be focused on pairing down the most important details of the shoot that correspond to their ideal medium.
Example use case:
Jake: Independent filmmaker in his 30’s creating commercials for clients. Jake wants to take his filmmaking to the next level for social engagement that goes past traditional methods. He sees augmented reality potentially as a budding industry and great opportunity for more viewership/interaction. The problem is the pipeline for augmented reality is quite different than that for traditional filmmaking. Also, is augmented reality going to be the best medium? Or would virtual/mixed reality or something else be closer to his goals.
Jake starts with our questionnaire website:
He has to answer a number of questions ranging from how you want your product to be experienced, what is the budget, and who is your target audience.
Based on these questions we will make a “best case” suggestion. This will either solidify or potentially change his ideas up to this point. Once a decision is made on the target audience and medium, it’s time to plan out the shoot.
Jake must break his shoot down into our four categories of pre-production, production, post-production and channels. Here he can use the design deck to flesh out all of the details for the medium he chooses. He will pick the corresponding cards that fit the medium and begin to flesh out the shoot armed with more confidence and preparedness.
We still need to do more user testing and figure out how to make the process more informative and intuitive for our stakeholders.
I wanted to use this new tool to visualize some of the search terms on the topics my group is working on. We are making a tool for producers and storytellers to work through and design their experience in the correct new medium of their choice. This includes Virtual Reality, Augmented Reality, and Mixed Reality.
I started trying to connect some dots on this infographic program (InfraNodus) that scours the internet and creates connections between the nodes and their corresponding concepts/metadata. I tried inputting and searching for different terms. It was interesting to see the connections made by VR, AR, MR and Storytelling.
Some of these connections show the connected nature of these new mediums. For example blending reality with the virtual and the use of computers and their environment. Is this the perfect environment for all new stories? Definitely not, but our tool will help creators pick the perfect medium for the story they are telling.
The preliminary votes/responses from my group’s survey
Rani started this survey and we’ve gotten some prelimary responses already.
These responses are very interesting pertaining to the use of social media and AR. This seems to be the way to gather the largest audience for your work.
A key response here is the ease of use for AR. This is definitely a big advantage as it is definitely more accessible for the end user. The downsides are often the quality/accuracy of the experience itself.
We will be reaching out to experts in the field like the AR/VR association to gain some insights as well.
These are the main stakeholders we are targeting for our research project.
For the last part of my time capsule I decided to first start out sketching some potential scenes. I was having a hard time finding interesting revelations from whoever finds the time capsule. I was almost more interested in trying to predict what the world in 2145 would look like.
I’m not super confident in my drawing skills so I started looking online for comic creators to help me make something that would be enjoyable to read. I tried this site first but found the character editor wasn’t very good and you couldn’t edit their poses.
I was able to find the website Storyboardthat which turned out to be an awesome editor with just enough customization. I began building the scenes and working around some of the objects and backgrounds that they had available. It was honestly really fun to play around with and they give students a 14 day free trial so I could export everything.
I exported the final comic in multiple formats including a gif of the entire thing playing, I attached it at the bottom.
I decided to take this week to work on some of my physical computing skills in designing the enclosure for my time capsule. I happened to receive a mystery box of electronics that I thought would add a really fun element. Also I chose it because this box from Adafruit is something that represents myself and the time period we are in (electronics for hobbyists in 2020). I received a matrix portal in addition to a large 64×32 pixel display. I knew that it would be fun to attach the display to the enclosure. In researching what projects people made with these items, I decided I wanted to put “2020” along with a video of myself in Bitmoji form. Memojis are avatars that people create for use with Apple devices. I’m in a class called performative avatars and we’ve studied how this is another way people choose to represent themselves online. What could be a better way to introduce the viewer to myself than a pixel display I received this week displaying my memoji.
I hit many issues in creating this, but it started with setting up circuit python which was a new platform for me. Thanks to my past experience with Arduino I was able to work through the issues I encountered because I am used to updating libraries and troubleshooting. The thing that was the most confusing was a serial system they use called REPL, this allows you to run singular lines of code while the device is plugged in. In Arduino you usually run the program and can watch the serial monitor but not be able to interact with the device (hence stopping programs I was trying to run).
I first had to wire the matrix portal to the 64×32 pixel display and attach power.
In order to put the bitmoji of my face on the display, I needed to first record myself. I then exported and converted that video to an .mp4. This allowed me to edit it in Adobe Premiere Pro. Once in Adobe Premiere I made the sequence match the dimensions needed for the pixel display (64×32). I used a pixelated font to match the display as well.
It was hard to tell exactly how it would look because it changes at different distances. This was also my first time working with such a small canvas size in video.
I then needed to use a program called Aseprite, to turn my image sequence (I exported a JPG sequence from Premiere) into a vertical bit map. The vertical sprite sheet bitmap is what the circuit python template I was using already worked with. I did some tests to find the correct orientation.
If I had more time I would try and add some audio, along with building a better enclosure to hide the electronics a bit. I would also look into ways to make it more protected.
I was very excited to get started on my time capsule. While it seemed daunting to choose only three or five items, I loved the idea of curating a box for the future. I looked into Andy Warhol’s time capsules and was inspired by breath of items he included. It was a deep look into the mind and process of one of America’s greatest artists.
A theme I have been exploring during my work is the continuity of old and new. I’m fascinated by analog technologies and how they were driven out by digital competitors. I love to see the artifacts of past technologies either ingrained in newer society or fading away and ask why.
My family came to America in 1900 and lived in Newark, New Jersey for quite some time. During this era Newark was basically a little New York, immigrants either chose New York, Newark, or Pittsburg at the time. It had a bustling downtown and city center. I can still see the sparkle in my aunt’s and uncle’s eyes when they talk about the way Newark used to be. Today, Newark has been struggling to revitalize for the latter half of the century. Seeing old photos of Newark always fascinates me. I’ve read census documents, marriage records, and discovered old photos. It’s incredible to try and piece together the past. Today we have tools like google maps where I can revisit old addresses I find to see what the buildings of the past look like now. I know likely many things will be lost in translation for those who don’t have a sentimental attachment to the area. I think a time capsule is a unique opportunity to make a commentary on the past, the things that were important to you in your daily life, and the things you want to pass on. Upon this I feel a responsibility as well to honor the past.
Living in 2020 has been a pretty crazy experience to say the least. Much of our news and content we absorb online, which is another reason I love the idea of using physical items to express our condition.
9/11 Memorial rock
My first item is a rock that I bought from the 9/11 memorial. It states “NO DAY SHALL ERASE YOU FROM THE MEMORY OF TIME.” (Virgil) I really love this rock because it’s in reference to the installation at the 9/11 memorial. I was in NJ at the time while my dad was working in Manhattan. Everyone who was here during that time always remembers that day. It is a day that’s magnitude could not be overstated and is a symbol of rebuilding but honoring the victims. This was a moment New York had to come together and rebuild itself and will always leave its mark on us and our world.
pentax 110 auto camera & Minolta 16 Film
The Pentax 110 auto is my favorite little camera ever made. As I’ve said I’m someone who lives with the idea of the past closer to the forefront of my mind. I think that this camera says a lot of the kind of person I am and could be very interesting for someone to look back in the future. While we’ve had technological advances into digital photography, there is still a large group of people keeping film alive. Maybe in 50 years film will be completely obsolete (I truly hope not). I believe the underlying beauty of a camera like this is there is no getting away from the true art of photography and the photographic process. In digital cameras, abilities to do everything automatically, even sending data directly to a computer over wifi makes them extremely convenient. Sometimes, with so much convenience one loses out on that process. I believe there is a time and a place for both, but I absolutely love to hold onto this process for as long as I can. Also, being a 110 camera means it’s actually an outdated film in 2020. I hold on to this because I believe there is still a lot of inherent value in the smaller format. While attempting to differentiate myself as a photographer I think we have to look backwards as well.
The Minolta 16 film is another item that goes along with the Pentax 110 Auto. It is the same smaller format (110/16mm) and it was custom cut from a gentleman on Ebay. He even made his own packaging. I found this fascinating that someone would take the time to try and keep this old film alive. I surely treasure that there are people out there doing so.
Face mask with face scanned part
I recently used an application called Bellus3D to scan my face using photos, true depth data from the iPhone, and their wrapping algorithm. This resulted in a pretty accurate 3D model of my face. I chose a mask because obviously it’s very important in our present day Covid world. I chose to include it with the face scanned part to highlight a movement of people using modern technologies to help. I saw an extreme push from people all over the world flock to the maker community to find ways to help. This included sewing masks, people making 3d printed masks with air conditioning filters etc. This was extremely heartwarming and inspiring and it’s definitely something I want to be remembered in the time capsule. When we saw our doctors and nurses suffer from a PPE shortage, many jumped in to help in any way they could using their skills and what was available. I believe as 3D printing technologies get better and the consumer is able to print more homemade parts we will see this self sufficient part of society grow.
Proven skincare pamphlet and bottle
I think the products of 2020 in America are very specific and I wanted to find something that I thought embodied what that means. I chose Proven Skincare because I recently saw them on Shark Tank, another phenomenon that I think will be interesting to look back on. They came out with the idea of using an algorithm to create a perfect skincare product based on a number of key factors. I feel this represents capitalism in America, but also the idea that we trust machines/algorithms more these days to create personalized items. Data privacy and tracking are huge concerns these days, and carry both good and bad outcomes. We have created artificial intelligence to create profiles of our interests. I wonder how this will continue in the future, and I think this product is a really telling example of where capitalism and technology in America are right now.