| Posted by Andrew - January 16, 2007 - 16:52 |
Inspect the UnexpectedThis morning for our second presentation we had a surprise: the System behaved in an totally unaccustomed way, degenerating into a most infernal rusty-flames and shadows thing that we've never seen before -- and which we liked better than anything we'd achieved deliberately to date! This is actually encouraging, since the whole idea of the project is a system which does the unexpected and evolves.Recent Cosmosis images can be seen here, where they will continue to accumulate. To recap on some of the points raised in discussion:
Although we felt much better about this presentation that the first one at Banff, we were still at a considerable disadvantage in that we don't have input working, we don't have tiled display working, and the sound needs to be custom (cellular automata and audio texturing) and exploit surround (OpenAL). Our goal is to have a decent solution to these problems by our third presentation. A shout out to my man David for awesome particle-physical code and a solid grasp on the future of the project, yaar. |
| Posted by Andrew - January 10, 2007 - 19:44 |
Rude Posey
I'm trying to keep in the habit of saving occasional images of the system.
It's important when developing graphics to have a convenient way to get a window capture without interrupting the work flow too much.
Can't say we have that at the moment, but I have been able to capture some images.
There are some charming artifacts which I cannot yet post because the screen grabber I've always used suddently ("bitrot") turns corrupt!!
On the right is shown a portion of a corrupt capture.
I think we can fix these...
The following represent only about one day's development, so the images are not particularly diverse. Rather than clog the blog with these in the future, I'll redirect to another page where I'll be trying to organise a visual history of development. 
|
| Posted by David - January 4, 2007 - 22:44 |
The Universal Question:"So what's it gonna look like?" ...Or more sophisticated variations of the above seem to have been asked every time that I've had the pleasure of presenting our project. I suppose it is only natural with all the talk of agents and ecosystems and things flying around, sticking to one another, interacting, and building complexity, but ... to me, at least, what is so amazing (and what sustains me through the periods of "dry" development) is the concept. That is, the theoretical model that novel and intricate structures and systems can construct themselves from very simple forms. No matter what the aesthetic of the rendering is, just the idea of this process is fascinating to me. Anyway, our simplest answer to the question is: "We don't know." While we choked on that (though I exaggerate) Alan was helpful enough to point out that what we have shown is very much in-progress (and really, more fresh a realization than anyone probably knew), so he asked instead for us to talk about our -process- of building an aesthetic. The comparison was made to painting -- a painting is not just a transferance of image from the mind to the canvas, it's often a series of experiments as well as, well, playing with the paint. Just so we find aesthetic forms by playing with the parameters and functions of the simulation and with OpenGL! Sure, I can (speak for myself and) say that I have vague images in my head, but I accept that the final product will have to simply be inspired by those but be realized by the process of the making of the system. Along similar lines I've accepted that I can never draw from the images in my head, what I make is always something quite different, but perhaps equally (or more?) valuable. And that's my answer to the question. (And I must apologize to Andrew for posting so much; I'm not trying to make him look bad. On the contrary, I'm trying to make myself feel good as I consider the amazing amount of coding he has pushed himself to do. So there!) |
| Posted by Andrew - January 4, 2007 - 15:36 |
Some Thoughts about DimensionsI think our main reason for wanting to keep the graphics perspective fixed is to avoid the frustrations associated with navigation in a 3D space using free gestures. However, it would be misleading to say that we are committed to a 2D environment. There was never any intention to use stereo imaging, so the graphics are always 2D, even when the models and graphics engines are supposed to simulate 3D. If we use layered 2D OpenGL, but use slight perspective rotations, this can be an efficient way to achieve enhanced perception of the layering at no cost, although this effect could also be achieved laboriously by perturbing the coordinates of the agents in software.There are single unbroken 1D curves which fill space. Fractals are so-called because their dimension is not an integer, but is fractional (for example, lying between 2D and 3D). There are definitions of dimension which give different dimension values for the same models. The gestural space is 3D, but the perspective of the camera projects this to 2D, and the gestures themselves are more like 1D curves. The abstract ecosystem space inhabited by the agents could end up being 4D or higher. Some agents may manifest visually as fractal textures, so that they have fractional dimension. The audio waveforms are essentially 1D, but we are using 5.1 surround so there is a 3D spatiality to the sound as well. Also, we are considering using 2D cellular automata to generate audio patterns, so in this sense the audio is not 1D. Furthermore, repeated motifs form a sort of aural tapestry which is more like a textile or a landscape than like a string of beads. |
| Posted by David - January 3, 2007 - 13:44 |
In the GrindI know what we did in mid-December... 
EyesWeb Video Interpretation Fig. 1: EyesWeb running our video interpretation patch Webcam video input is clipped and shrunk to a 90x90 pixel square to decrease the processing necessary with acceptable losee of image quality because we will only deal with broad movements at this point and the video input is not meant to be viewed, just interpretted. It's then greyscaled, crudely background-subtracted, and thresholded so that movement produces white pixels. Subfield Sampling Fig. 2: nine points of motion being tracked on webcam video. We expanded on our small Eyesweb "Conductor" project to allow multiple points of motion sampling. The webcam video input is cut into a nine-piece grid of subfields. Each subfield is sampled for the amount of motion (# of white pixels) and for the center (in X and Y coordinates) of the area of motion. (I should mention that this is an extremely crude way to try to interpret multiple points of motion from video because, well, it's limited to nine fixed subfields which output just one center of motion each. Not only is this a very small amount of information, it is inaccurate, especially if motion crosses over multiple fields. Eyesweb doesn't /easily/ support really nice interpretation for our purposes. Sure, there is a function meant to track a human body, but it only tracks -one- human body, and if Eyesweb doesn't have clear video it will give some very strange data. We'll ultimately have to output simplified video from Eyesweb and write our own interpretter, I think. But for now, this makes things run.) Three numbers (X position, Y position, and magnitude) are taken from each of the nine subfields and assembled into a 1x27 matrix (because Eyesweb 3 has very limited arrays) which is sent to a proxy program running on the same machine that, in turn, sends this information over the network to the machine running the ecosystem/renderer. OpenGL Rendering Fig. 3: a simple rendering of the nine points sent from EyesWeb to the renderer The ecosystem/renderer receives data from the network and, at this point, renders it pretty much as-is, without a simulated ecosystem. But the point is that it all works; The data flows! A Modest Proposal for Simple Simulation 
Fig. 1: Particle Decay A point will perhaps be the simplest form of agent in the multi-agent ecosystem, and for now are the basic form of input. Points may decay over time with regard to how interesting they are, and if they are given a magnitude based on the sampled sub-field's area of motion, then clearly points created with more motion will have more magnitude and more life. Fig 2: Field of Interaction A point-agent should interact within a certain field. This interaction can be a bounce, or it could combine the properties of the two point-agents into one more interesting agent. From the standpoint of reducing processing time it is useful to eliminate agents while creating interest. Fig 3: Interactions Again, to reduce processing, we may allow only energetic "hot" particles to detect collisions and produce interactions. So "hot" can interact with "hot" and "cold", but "cold" particles won't even bother running the collision calculations. Fig 4: Up the evolutionary ladder... From simple particles we can build up primitive shapes and behaviours, hot particles will collide to form "larger" particles which may combine to form line segments, which may collide and collect to form polygons, which will act in more complex ways, and so on... |
| Posted by David - December 7, 2006 - 18:29 |
Networking with EyesWeb, Staring into the Abyss
Andrew appears to discuss something important in this entirely spontaneous picture. 
No, I'm not trying not to laugh - that's just what I look like when I'm thinking deeply. 
Simple sample output of, what was it?, 1000 sets of 40 integers or so .. as colored lines! drawn between points of a grid. We've been working in the VR room (where else?) with the additional small pressure of my upcoming MADT 401 revue, then need to have something displayable in Banff. Small EyesWeb Project Our small EyesWeb project (Webcam tracking to MIDI output) was a bit lacking in presentation, I think, when we showed it at the U of C. The lighting and background conditions were entirely different; the VR room is painted black, has a grey floor, and a white screen; simple. The interactions lab has computers, lights, posters all over the place (not to mention a crowd of onlookers), so our previously clean-cut motion detection was filled with noise. Clearly our environment in the VR room is insufficiently hostile to produce the motivation for an extremely adaptable background-subtraction system. This is good because it means our job (Cosmosis) is easier in the controlled environment, but it makes me want to invest effort in doing a better job at interpretting the video to extract motion. Time to re-read that paper! And what's it mean to Cosmosis? Which brings us to the topic of what exactly we're going to get EyesWeb to send to the various Cosmosis components -- in short, it looks to be arrays of integers. How do you transform video into a meaningful array of integers? And how do you reduce distortion produced by image artifacts - like someone's shirt which happens to be the same tone as the carpet, or the back wall. Perhaps there is some way to make sure that whatever people walk in with is -always- different from the environment. Decoding EyesWeb Output To illuminate the undocumented gloom of EyesWeb network output Andrew used his lovely "Loom" program to examine the raw data spit out by Eyesweb. The opening shot of the connection is a bunch of mysterious junk (but did Uta find documentation on it?), with sets of 4 bytes following. An int: 8,0,0,0 / x,x,x,x / 0,0,0,0 The 8 tells how many bytes will follow the initial set of four bytes, X is the integer (reading from left to right, mind you, so if the int is "2", the set is "2,0,0,0"... but in hex or something), and that last set of bytes is just zero. Maybe it's what always ends a packet of data, maybe it's used for something else. I'm pretty sure we didn't find out, but I'd have to check with Andrew. An array with 3 ints, [x,y,z]: 12,0,0,0 / x,x,x,x / y,y,y,y / z,z,z,z So there are 12 bytes following the initial 4, and no trailing set of 0's. I think. Fascinating, 'eh? Network Instability Networking programming is a pain. Or maybe I should say that getting it to work without messing up is a pain. Is there an issue with ACAD's network security? We don't think so, and don't really know. I'm not sure how positively the network people would respond if we told them about what we were doing because institutions love to be network "security" fascists. Do we fully understand how socket networking operates -- in Window? Not really. But we'll learn. What's clear is we don't trust EyesWeb all that much, so the plan is to have EyesWeb to send data to a program (the "Repeater") running on its localhost, and that sends data across the network to the (presently combined and bare-bone) Ecosystem/Display program. It works with simulated EyesWeb output! Short-term Goals
I think that covers it. |
| Posted by David - November 29, 2006 - 22:09 |
Here is the flowchart Andrew mentioned:
If I were to do it again (and I will) I would make it clear that processing would be taking place over a number of networked machines. It is difficult to get OpenGL working with Python - on Windows. There is always another pile of errors and some kind of module missing. And documentation is poor to non-existent. Speaking of poor documentation - try EyesWeb! It's actually kind of fun to puzzle out exactly what does what. Well, it's frustrating too, depending on my mood. IR may not be necessary considering the silly things we can do with adaptive background subtraction (which makes it sound a lot more clever than the actual effect), but light levels should be changing quite a lot throughout the Cosmosis process, so IR is definitely desirable. We'll work on it tomorrow. I should like to get a simple data pipeline working by next week for the MADT 401 reviews, just any kind of input from EyesWeb to a Python program to a display (or sound?) will do. And from that skeleton, we'd simply build up the complexity! |
| Posted by Andrew - November 29, 2006 - 13:51 |
Ah, mustn't neglect the blog.David and I visited the VR lab at ACAD together last week, where we found to our relief that my webcam is supported without the driver disc (which I had forgotten). In fact, since our tour of the facilities with Alan Dunning, the lab has been equipped with its own (face-tracking!) webcam, and also ample USB extension cords which we will definitely be needing if we place the camera in front of the participants. David may wish to say more about this experience, but basically it was an important and somewhat painful first step where we familiarized ourselves with the equiptment and tried to get each thing working independently -- with middling success. One thing we have yet to determine is whether it will be possible to use infrared illumination with the webcam -- we tried using the IR lights in the lab, but it seems that at least those frequencies are blocked by both cameras.So far I've gotten Python OpenGL rendering working on a linux platform, and python networking (server/client) betweem remote machines. Python is a fine programming system and I've no regrets about abandoning initial Java plans in favour of it! Also, Chromium is written in python which should make our interactions with it that much easier. Chromium is a system for distributed rendering on multiple workstations, capable of targeting multiple displays or projectors. This site also refers to a SIGGRAPH paper which should be in our reading list; Figure 2 illustrates the configuration appropriate for our system. We aim to have the basic system (I hope David will post the image of our system diagram from his MADT proposal), sans any sophistocated multi-agent ecosystem, functioning before we adjourn for Christmas break. |
| Posted by David - October 27, 2006 - 22:19 |
Poetry, Programming, and LinksTo consider in terms of a particle mode of generative aesthetics:From Lederman's The God Particle, pg 342: And the Lord looked upon Her world, and She marveled at its beauty - for so much beauty there was that She wept. It was a world of one kind of particle and one force carried by one messenger who was, with divine simplicity, also the one particle. I've been working on a personal project to learn Python and programming in general; I've worked for hours and hours for the last three days and I think I understand the rush Andrew was talking about. You know, the agony and ectasy of coding. Here's a screenshot for the interested. If you really break it down, it's basically a cellular simulation. I found a few links on/through Gamedev.net; Some of this stuff takes some really serious math: Genetic Programming in C/C++ (Hans Kuhlmann / Mike Hollick) Application of Genetic Programming to the "Snake Game" (Tobin Ehlis) Neural Network FAQ (Warren S. Sarle) Neural Netware (Andre' LaMothe) And it seems we've got to pull some kind of "phidget" system out of our... hats. I don't think they could apply to our project. Andrew did suggest something about controlling lights or something, but how could a simple moving/changing light compare to the real lightshow going on in the screens? We'll see. Also, we checked out the ACAD VR room and it looks like we're aiming to use it for the installation of the project. As a venue it's terribly isolated and no one who doesn't already go to ACAD would probably see it, but the hardware is amazing, if perhaps a little clunky. Of course our goal is to make the program scalable to all sorts of systems, from a personal laptop with headphones to a projector/tower/speakers system to the powerful setup in the VR room. VR room first, of course, we've got a deadline. Somewhere. |
| Posted by Andrew - October 16, 2006 - 22:51 |
More about the Science (and the Art)...If the Goal of Science can be summarised in a sentence, it might be the collating and refining of cognitive models of reality, formalised rigorously with careful technical language (preferably mathematical), and corroborated by experiment, for the purpose of increasing our understanding about the true nature of our World. Paul Feyerabend, an influential philosopher of science, in his seminal work Against Method (1975) argued that science is fundamentally anarchistic, and rejected the notion of any universal scientific method, although certain methodologies have passed in and out of vogue through history.Science was not originally practised for benefit of its fruits, in contrast to our present times, but rather for very joy of it. One is tempted to speculate that what gives pleasure correlates with truth and usefulness. This is a personal philosophy I hold, that beauty leads us to truth. I take exception to the notion of an "ugly fact" -- a fact is only ugly if the beholder of the fact has previously committed themselves emotionally to an erroneous view. Hence, the scientific content of our proposal, aside from the relatively banal technical issues (and technique is as much an Artist's problem as a Scientist's), is an enquiry into empirical intentionality, a concept which will be illuminated over the course of the project, but which will involve investigating human aesthetic response in the context of an interactive generative system. This is cognitive and behavioural science more than it is physical science or engineering, despite the prominent dependence of our system design on the latter. The transferable value of these sorts of potential insights include at least two important contemporary areas:
|