Unity, C#, OSC, Blender, TouchDesigner, Python, Hand Tracking, physical interaction
The Art of Being is an immersive and interactive environment that explores how your senses augment your connection to your body and internal experience.
*best viewed with sound, full-screen*
CONCEPT
We often get caught up chasing after materialistic desires, causing us to overlook and not value core, internal experiences. For example, when the wind is strong, one often focuses on getting through it to get to their destination rather than immersing themselves in the sensory experience. The appeal of our goals, desires or the objects around us compels us to forget our bodily presence and how it affects the environment, and how changes in our environments affect our internal experience.
The Art of Being encourages people to reevaluate themselves, their values and lifestyles. It acts as a stimulus in pushing them to assess the way society functions and how they are expected to constantly chase after their next goal, their next desire, success and money, rather than connect with their own body to truly experience the present moment through all five senses.
The essence of life lies in sensory experiences like the sensation of water on skin or inhaling petrichor smell, and yet, we focus on chasing after the material to satisfy needs imposed onto us by society. As a means to help the viewer ground themselves, I am exploring how the senses can help you connect to your internal environment, in order to establish the difference between being and merely existing.
WHY A DIGITAL ENVIRONMENT?
When entering a digital environment, one often forgets about their own body and rather, engulfs themselves into the aspects of the space. By using a digital medium, I am exploring the relationship between the viewer and the work. I am actively questioning how to bring presence back in a digital space, as well as how external experiences﹣interaction with the environment﹣can affect internal experiences﹣sensory input. The use of a 360 environment in Unity, specifically, augments the feeling of immersion as a method in grounding the viewer in the present moment.
How can we design digital environments to evoke not only the visual and auditory senses, but also tactility, olfaction and gustation?
INTERACTIVITY
Through this environment, I am actively questioning how all five types of sensory experiences can be translated to a digital environment. How can we design digital environments to evoke not only the visual and auditory senses, but also tactility, olfaction and gustation?
The video above depicts the interactive aspect of the TouchDesigner visuals projected on the walls of the Unity environment. TouchDesigner uses your camera image to track motion. Your hand movements cause ripples in the animation, which parallels the way movement in water has a ripple effect. This brings tactility to the digital environment in an implicit manner. It also emphasizes the interesting concept of how you affect the environment as much as it affects you.
KEY ELEMENTS
What is the story?
Key Element 1: LIGHTING
In order to create an intended mood.
- When playing with lighting, I found that bright blues and clearness added energy and ecstaticness rather than peacefulness. I altered lighting in a way that made more mellow colors to appear, which allowed for calmness and peace.
- At first, I worked with more transparent water where light could seep in and refract, but it steered away from the ideas of internal reflection that I was exploring. Rather, it felt too close to replicating a pool experience, so I designed and coded an opaque water surface.
Key Element 2: VISUALS
In order to enforce the connection between your inner self (represented by heartbeat) and your visual perception.
- I experimented a lot in my process. I implemented a particle system as a test but found that it took away from the experience - and this gave me a better understanding of how minimalism vs. populating the environment with different elements affects mood. I played around with the balance of elements to achieve the intended mood.
- I went through several iterations of the animations to be mapped on the walls, because I wanted them to be interesting enough for the viewer to want to spend enough time in the environment, but not overwhelmingly apparent. If it were to become the main focus, it would take away from the internal experience I intended for the viewer to go through.
- I started out with an intricately designed pool, with tiling and gold lining, but remodelled a simpler, clean pool, as the former became a distraction from the experience.
Key Element 3: SOUND
In order to achieve immersion and amplify every beat.
- I overlaid heartbeat with the sound of being submerged, to amplify every beat. Compounding both of these sound sources together created a submersion-like distortion of sound, which added to the overall ambiance of the experience.
- I used sound as a key element to encourage reflection of internal experience, as hearing a heartbeat encourages you to internalize it, as if you’re in your own body, listening to your heart.
THE GUIDE
You are the narrator. The guide becomes implicit in the form of the senses and your heartbeat.
2 W’s AND AN A
The Art of Being is meant to encourage people to reevaluate themselves, their values and lifestyles. They are encouraged to evaluate the way society functions and how they are expected to constantly chase after material - their next goal, their next desire, to be successful and have money - rather than connect to their own body and truly experience the present moment.
WHO
The environment is aimed at those who are stuck in a loop, believing they are moving forward, when in fact, they are stuck, like hamsters, on a perpetually running wheel. Experiencing these core moments is what makes us feel alive, and yet, we focus on chasing after materialistic things to try to satisfy our “needs” - needs imposed onto us by society. The fact that we can feel the strong wind pushing through our hair and pulling our face back, the sensation of water on our skin, inhale the petrichor smell and let it pass through our bodies… nothing comes close to these core feelings and experiences.
WHAT I WANT THE AUDIENCE TO EXPERIENCE
I want the audience to be in touch with themselves and their feelings/emotions as they experience the environment. I’m not aiming for them to think of anything specific, just to truly be immersed in the moment through every sense.
❝ Exploring how your senses augment your connection to your body and internal experience. ❞
MAKING METHODOLOGY
Alexis Pauline Gumbs, Undrowned
American author (Durham, North Carolina)
Independent scholar, poet, activist
In Chapter “two: breathe” of Undrowned, she explores the process of evolution and immerses herself in its beauty. She explores interconnectivity and puts forth the idea that there is no difference or distance between 'you' and 'me' by showing the similarities between the evolution of sea mammals and humans. Our relationship to our environment has grown and shifted from what it was to our ancestors. Presence through breath allows us to immerse ourselves in the moment and evaluate how each breath affects our surroundings, and in turn informs how we navigate through life.2
Sara Ahmed, “Orientations”
British-Australian scholar
Area of Study: Intersection of feminist theory, lesbian feminism, queer theory, critical race theory and postcolonialism
In her paper, “Orientations”, she looks at what it means to be oriented and what difference it makes: how is it that we come to find our way in a world that acquires new shapes, depending on which way we turn? She focuses more on creating a dialogue between the concept of orientation and queer studies and looks at how bodies take shape through tending towards objects that are reachable, engaging in phenomenology.1
Aspects of this paper translate well into my work, as I am looking at a more abstract aspect of orientation, in terms of the implications of being oriented or lack of orientation to the senses and the metaphysics quality of it - being and existence. I’m also exploring the more materialistic quality of orientation, in terms of guiding the experiencer by stripping them of appeal of objects, and looking at how it shapes their inner bodily experience.
Notes from Sara Ahmed’s “Orientations”
TECHNICAL PROCESS
Environment Building
I used blender for 3D modeling
Worked with texture and shadergraph for the water, and C# code
Experimented and worked on making an aurora borealis effect with shader graph and put it in the pool instead of the sky in light blue, but it became too busy and steered my environment away from its intention
TouchDesigner <--> Unity
This was a super tough process because not many had done this before, so there isn’t much on it online, only with Unreal mostly
Initially, I was going to work with OSC to connect TouchDesigner to Unity, but this would only give me the sound input, so I was going to use those values to affect a shader and work with noise in Unity to create my visuals that way, even though my idea was to use TouchDesigner visuals.
This was a super complicated process and after 2 weeks of actively trying to figure out how to connect the two by trying to work with OSC and watching tutorials, I figured out that there was in fact a simple solution to my initial idea. There’s a syphon spout out component in TouchDesigner, and there’s a Syphon plugin for Unity that can connect it to various other software like TouchDesigner and MaxMSP.
I brought the TD animation in as a render texture, and mapped it onto my pool object, making sure the size matched the animation resolution
Because of the nature of the pool object, the animation stretched across the outside and inside of the pool, making the animation not visible. I only wanted the animation to cover the inner walls, so I just decided to have 4 planes as each wall with the animation. This became a limit that I hopefully will be able to sidestep in the future
OSC and Camera Movement
Even though this had a little bit more resources to work with, it was actually even tougher than the TD Unity animation rendering.
The camera rotation in Unity is controlled with mouse movement because I implemented a script on it. I wanted these camera rotation values to be passed to the OSCin CHOP in TouchDesigner.
I worked with the extOSC plugin available in the Unity Asset Store, but unfortunately, it had extremely little documentation on it so I had to really experiment with it myself. I went from trying to surf online to find any kind of help, but coming up short, to looking at the example folder to see what they had done. And in this, when I opened the example scene, TouchDesigner actually caught the values with OSCin, but then when I was manipulating the script to get camera rotations, I kept getting errors or it wouldn’t work so I’d keep going back and forth between testing and changing aspects of the code. I iterated as much as I possibly could based on whichever ideas came to me in how to achieve the result, but it still didn’t work. And then when i tried rerunning the original example code, that didn’t work either. I then tried to make my own script files and work in this way, but didn’t manage to figure it out.
I then switched over and retried working with the UI transmitter component and manipulating it to get it to feed data to TD. After some debugging, Unity was able to transmit and receive data within itself, but it just wouldn’t go to Touch. I tried many different solutions and also restarting the softwares and my computer, but as Clareese and I realized together, it was a jam with the port that we couldn’t get fixed.
Even though I didn’t figure it out in the end, this process was extremely rewarding as it helped me get better insight on how to read errors, how to debug, how to use tools online for help, and even though there was very little documentation on it, I was able to deep-dive and get a better understanding of how the plugin works. And i did get quite close to getting it to work, so it’s definitely something I’d like to explore further in the near future.
FUTURE STEPS
(1) Exploring the implementation of more senses
It was interesting to come up with ways to implement the different senses in a digital environment, because one cannot, for example, physically feel the texture of different objects.
I planned to have mouse movement affect the visuals with OSC, as I mentioned, but due to the complexity of this process, lack of documentation online and time restriction, I now plan to have it as a future step. Your movement in the environment affecting the visuals in a fractal manner, by mouse movement, has a parallel with the way that movement in water has a ripple effect. In a way, this brings in tactility. It also brings in the interesting concept of how you affect the environment as much as it affects you.
It’ll be interesting to explore more of the senses, especially since they will be implicit through materiality and senses that evoke olfaction. This will strengthen the viewer’s connection to the experience.
(2) Heartbeat sensor and Arduino, connected to Unity
This aspect is inspired by Naziha Mestaoui’s One Beat One Tree, where a smartphone detects the user’s heartbeat to plant a digital tree on a monument in Paris that grows in rhythm with the user’s heart . It “bridges the virtual and reality, technology and nature, as well as what is visible and invisible.” 3
Due to the timeline of the project, I wasn’t able to implement the heartbeat sensor into my project, but plan to have it as a key future step. It is important because it steers the viewer away from the materialistic appeal of the experience and makes them more aware of their bodily presence, and connects them to their inner selves. It forms a clear connection between the viewer and the environment, and personalizes the experience.
References:
1 Ahmed, Sara. “Orientations: Toward A Queer Phenomenology.” Thesis, Project Muse, n.d.
2 Gumbs, Alexis Pauline. "Breathe." Undrowned: Black Feminist Lessons from Marine Mammals, AK Press, 2019, pp. 42-60.
3 “Naziha Mestaoui Projects Virtual Forests Growing onto Paris' Monuments.” Designboom, 31 Oct. 2014.
1 Ahmed, Sara. “Orientations: Toward A Queer Phenomenology.” Thesis, Project Muse, n.d.
2 Gumbs, Alexis Pauline. "Breathe." Undrowned: Black Feminist Lessons from Marine Mammals, AK Press, 2019, pp. 42-60.
3 “Naziha Mestaoui Projects Virtual Forests Growing onto Paris' Monuments.” Designboom, 31 Oct. 2014.