Final year project in computational art

MODULE  – ML Speech to Command.

Hello world,

This element of the artwork is testing speech commands. A “Machine Learning” technique for sound classification. It is done by analysing the sound fed through the ML model and making judgements on the voice of the detected subject. Estimating the words received through the microphone into the model, which then compares it for the best match with the words trained beforehand.
I was fiddling with the out of the box model from Google, but I will probably train my own model if I decide to use this technique in my final artwork. Google’s model is good enough for basic things, but biased and trained on other people, so the accuracy is not as great and personalised as it could get.
I was exploring the utility of giving commands via words in the context of this project. I imagine I will possibly assign some visual elements, activate modes of operation, give commands, change variable values or other functionality not foreseen at this point.
Thes visuals not resembling the aesthetic direction of the project, but simply illustrating an exercise.

MODULE  – Pose estimation with PoseNet.

Hello world,

This element of the artwork is all about pose estimation. A “Machine Learning” technique for tracking human movement. It is done by analysing the image fed through the ML model and making judgements on the detected subjects skeleton’s position. This is done by estimating the position of 17 “key points” on the body.
I was exploring the meaning and utility of those key points in the context of this project. I imagine, I will assign some visual elements to those points as a means of aesthetic  expression, but it will not limit to it.
In addition to manipulating the position of “key points”, I taught the model with ML technique of regression to react on certain poses with certain action(e.g. changing the background colour when some skeleton pose is detected). This can be very handy, as I can choose to influence some elements of the artwork with movement – as a means of giving commands.

While testing the functionalities of poseNet,  took an opportunity to experiment with visuals. Not sure which aesthetic direction I will decide on, as it is way early for that to solidify.

PROJECT PROPOSAL

Hello world,
After all that brainstorming, ideasitation and prototyping, I assambled a proposal for this project.
Knowing myself, this does not mean that the project can not evolve into something more or less than this proposal, however this is more like a framework for my creative process.
A proposal of this type is a tool for me to develope and manifest this piecev – to focus my imagination and resourses.
Setting approximate scope and timeline for the process, helps to keep perspective where I am at and what needs to be done at a given time.
What follows is the development of individual modules and testing them out.

 Creative Project Proposal gpasz001

MORE EXPERIMENTATION

Hello world,
Here is some more experimentation, before I summon up a  proposal.
I’m still playing with visual and conceptual elements at this stage, so I can, later on, draw on a rich material when I start bringing things together.
Here is the original photograph I took for this sketch. I named it “meditation #044“.
Let me know what you think there.

EXPERIMENTATION

Hello world,

This part of the process is getting the ideas and concepts aired out, to get a better understanding of the project’s direction.

I took the smoke element from the previous creative session and added a digital shadow to it.
This “digital shadow”  or ” digital gaze” will be an overlay on the output media.
It is made of a few machine learning type algorithms, like face recognition/detection, poseNet(pose and movement recognition) and some pixel manipulation algorithm.

Conceptually, it is very interesting for me the idea of the real experience and representation of it in the form of visual data.
For example, I think I will use a dancer to explore movement, however, instead of displaying the dancer image/video directly, I intend to reveal some machine reading of the same, like the movement itself, or the data of the person, or a combination of all in intervals.

The images displayed here are highlighting the contrast in aesthetics of the real smoke and the machine’s gaze.
With this element, I’m exploring the phenomenon of classifying experience into data of any kind and calling – mistaking it for the experience itself.
Can we boil down, or represent things in full with data? – Even infinite amount of data?
There is a thing in sound synthesis, that if you have enough sin wawes, you could theoretically recreate any kind of sound or music, however, does all that amount of data retrospectively make you enjoy the music it’s trying to reproduce or represent?
Can data recreate any experience in full?
What is missing in translation?

While contemplating on this subject, I recognised that there is another side to this idea, which is to rather to trying to replace, we can enhance. I will create a user interface with the machine to move along with the human subject for performance sake.
So instead of either-or, I will aim for best from both.

Flow

PROTOTYPE – A

Hello world,
with this post, I’m heading towards getting a clearer concept and direction of this project.
This is PDF below represents the rough idea of the project ahead. I deliberately left it with the minimal description as it will probably change during the process.
I sketched out some visuals which represent ideas about the visual style. I aim to make it minimalistic, however, that is always a challenge for me, so we will see at the end.
I will use multiple codes running in real-time communicating with each other and along with this, there will be some pre-recorded media feeding into the codes.
The whole process will then feed into Resolume Arena via ManyCam and OBS Studio. From there I will project it on some surface – most probably a semi sculptural background.
I’m thinking to use these visual and conceptual elements mentioned above to track a dancer with computer vision and face detection(recognition) to create a short performance art. 
I’m still undecided, whether I will conduct this live, or assemble it from prerecorded elements. This desition will be made when I see more of the project evolved down on the line as I can’t foresee the complexity and complications just yet.

Prototype A workflow map

EXPERIMENTATION

Hello world,

this part of the process is still about inspiration and experimentation.
I was observing the smoke for hours. I had a feeling that it was dancing just for me and had something to communicate.
I like the smoke’s elegance and fluidity of movement, with the minimalistic aesthetic and I would like to aim for this kind of look for the final project, where less is more.
With this kind of form, the artwork could be more open for personal interpretation, and the imagination could fly higher and further with the experience.
This session inspired me to make some code snippets on the subject of thresholding, where the smoke is going to be “isolated” and manipulated.
It could be a direction for the project or a dead-end –  will see, however, I have a feeling that some part of the code will contribute to the final piece.

Flow

EXPERIMENTATION

Hello world,

In this session I experimented with some ideas and concepts.
I used mixed media to make a collage so to illustrate a possible workflow for my final project.
Programs used in this gig were Arena5, ManyCam, OBS, Adobe PS BR PR, VLC.
To achieve these visuals, I plugged in old works of mine and some raw materials, however in the final project, one or more code will run live and directed to a mixer application, where sound will be added to it.
I intend to repeat this session with some variations in equipment and media – 360 live feed for example.

GALLERY VISIT – INSPIRATION GATHERING

Hello world,

In the process of making art (or a project), one of the first things to do is to gather inspiration and doing research.
As part of this process, I paid a visit to Pace gallery, where I saw Trevor Paglen’s, Bloom. 

I left home without doing background research for this artist and the exhibition. I intended to see the “art” without influence and context.  As a result, I left the scene with a mixed bag of takeaways.

Takeaway No.1:  The display of the pieces did not work with the gallery. Space swallowed the art with complementary colours, shapes and sterile finish. The whole exhibition felt like a studio flat, from which the tenants are moved out along with the furniture, but left some art and flowery wallpapers behind along security personnel to guard it.

Takeaway No.2: The nature of the prints and the surrounding influenced me to perceive the flowery prints as wallpapers. Walking back and forth, trying different angles did not help me to enjoy the art, as I had a feeling that I’m missing out on something unknown to me. Also displaying some of it behind a glass gave a poster-like effect, to which I couldn’t connect.
Note: Perhaps, dimming a light a bit, would make these prints in the focus, instead just another thing on the wall.

Takeaway No.3: Not displaying the concept at the exhibition could be very beneficial for the viewer, since it could trigger the imagination and personal connection to the art, however, in this case, I think it took away from the experience.
The art displayed as something grand, but not looking like one, made me searching for the missing clues, instead of just being in the presence of the art itself.

Takeaway No.4: The cameras installed around the gallery were kind of interesting, but at the same time disappointing, since they didn’t do anything at all on the spot.
I learned later on the internet version of the exhibition, that those cameras had a function of broadcasting the show in real-time.
Looking it now from the browser, I ended up liking it more than the actual visit. It looks more real art somehow.

Reading about Trevor retrospectively, definitely adds to the work, however one cannot but wonder, is this how he intended to work and is this how art supposed to be viewed – contextually?