And, while I'm catching up on blogging, Andrew Brown suggested I check out Jared Tarbell, who has put together some musical visualisation works. The levitated.net site has a lot of information on these, although it's a little difficult to find some of it!

I like the idea of using podcasts in student assignments. Paul Draper & Matthew Hitchcok gave a presentation on their work doing this with music technology students. More complex in the subjects I teach. XP is probably the most obvious candidate for this but I worry about the increased workload for the students. I guess make it optional to either present to class or create podcast? In some ways I think this could be good for students with poor English, but on the other hand they may just see it as more 'busy work'. .NET is more difficult. Can't think of a smart way to use it...

The stuff that Andrew Brown, Andrew Sorenson and Steve Dillon are working on - the new version of Jam2Jam - looks really cool. The programming language that Andrew S has designed, Impromptu, is pretty amazing. Makes me feel like getting into Lisp - nice way to fill in all that spare time! Andrew B is keen to get the Spheres stuff running on their 'roadshow' where they get kids to play with the various toys they're designing. Would be a great opportunity to get some further investigation of the physical model interfaces with a different set of users.

Also, Steve Dillon mentioned a journal article, something along the lines of "Software Development as Research". Sounds very appropriate.

1. Open iMovie and import the clip you want to analyse. Important: Use a local disk to save the project - external drives are problematic. (Lots of bugs in iMovie 3 apparently.)
2. 'Share' the movie; choose File -> Share, Quicktime and export as a full DV file (with extension .dv).
3. Use the ffmpegX program to convert from DV format to mpeg1 format. The format is called VCD (Video CD) in ffmpgX. Note, in the 'options' tab, deselect 'split mpg' and 'Author as' options so you just end up with one big .mpg file. You can use either the mpeg2enc or ffmpeg encoders - as far as I know it makes no difference. (I've definitely used mpeg2enc and it's worked.)

EDIT (1/3/7): You can skip step 2 above - just use iMovie to import the clips from tape, then exit (after saving). This produces a .mov file which has separate files for each clip that it contains, but this doesn't matter. ffmpegX can read this file and can convert to mpg1 directly - saves heaps of time! Remember to deselect the 'split mpg' and 'Author as' options.

Once the above is complete - and it can take a while - you have an mpeg1 file which Transana can use without any trouble.

"Colour and sound do not admit to being compared together in any way, but both are referable to a higher formula, both are derivable, though each for itself, from this higher law. They are like two rivers which have their source in one and the same mountain, but consequently pursue their way under totally different conditions in two totally different regions, so that throughout the whole course of both no two points can be compared."

Naively, I agree. (In googling this I also came across Niels Hutchison's page on colour music, which appears to be a comprehensive look at this topic.) Cook goes on to point out Kandinsky's application of this basic approach in his work: "Kandinsky was much influenced by theosophical thought, and the starting-point for his theory is that the universe consists essentially of the play of vibrations and that these vibrations have a fundamentally spiritual significance." (p. 46) Later on the same page:

"Another way to visualize this would be a triangle, with Kandinsky's 'spiritual' (Goethe's 'higher law') at the apex; understood this way, sound and colour do not relate directly to one another, but relate indirectly through a common relationship with the spiritual. There is no question, then, of mapping the structures of sound and colour on to one another, in the manner of Skriabin's mapping of the colour wheel onto the circle of fifths. Instead, sound and colour correspond to one another in so far as they embody the same ultimate meaning."

Playing different notes spawns more balloons??

Damien got good crowds to both performances, and we had a little mention in the Sydney Morning Herald, although no reviews as yet. Apparently someone from RealTime was there so there could be one shortly.

Also of course, we had the install and launch of the spheres-in-a-circle piece (confusingly known as 'Spheres of Influence') at beta_space in the Powerhouse Museum. I was a bit apprehensive about the launch - not feeling totally comfortable with art lingo - but on the night it was really enjoyable experience. I'm always surprised by just how much having to present your work to others helps clarify just what you're on about- to me anyway.

This might be a good place to say a big thankyou to Deborah, Matthew Connel and Greg for doing so much work to get everything set up, publicised and generally looking tickety-boo. And of course all the CCS volunteers who came along and helped out on the night- especially Roman for doing the video. Thanks!

The article also links to the rehearsal joypads, which are mock-ups of instruments that connect to software that gives feedback on correct fingering/slide position, etc. When you play phrases correctly an on-screen band marches in time and stays together. If you don't they walk off in disgust! This could be great for beginner students...

From what I gather, scanned synthesis generally uses simpler models updated at much lower frequencies. The models don't produce sound directly, but are 'scanned' at regular intervals to produce the waveforms. The idea is that a performer can manipulate the models in performance to produce musical/natural-sounding effects. From the above site:

"The psychophysical basis for Scanned Synthesis comes from our knowledge about human auditory perception and human motor control abilities. In the 1960's Risset showed that the spectra of interesting timbres must change with time. We observe that musically interesting change rates are less than about 15 hz which is also the rate humans can move their bodies. We have named these rates Haptic rates."

What I find interesting is that in my work I've been using live instruments to affect a low-frequency physical model, which in turn produces audio. I think this is an extension of scanned synthesis that I have not come across before - but that's not to say it's not been done!

Later: paper describing scanned synthesis is here.

This approach offers a lot of interesting possibilities. For example, you could specify that your virtual trumpet be made of wood (and Brass apparently lets you do this), or make a virtual trombone out of play-doh.

I find it curious though that these types of products are often marketed as, "Now you have all the control of the experts - but without the need for years of training!" The problem is, of course, that the more accurate your model the closer the amount of training required will approach that of the real thing. Oh well, it's only marketing anyway.

The web page also has a description of some of the research behind Brass which was conducted with IRCAM.

Later: Online book by the guru of waveguide synthesis, Julius O. Smith III - Physical Audio Signal Processing For Virtual Musical Instruments and Audio Effects.

Still later: The Computer Music Journal has a great article by Ali Momeni and Cyrille Henry: "Dynamic Independent Mapping Layers for Concurrent Control of Audio and Video Synthesis". Dry title, but fantastic overview of the possibilities of using physical models to mediate between musicians, visuals and sound. These are the guys who developed the Physical Modelling for Pure Data objects I've been using. The article also mentions another set of objects which may be better for really large mass-spring-damper systems: MSD.

Exerting force on the spinning protons/neutrons by playing notes causes the particle associated with that note to spin out in a bigger orbit. Playing lots of different notes causes the whole structure of the atom to go nuts. This was fun to play with and to me appears to have a lot of possibilities. Ben seemed to like it when I demo'd it via iChat videoconference and he has a copy to play with now.

I've got a good feeling about this movement now. Conceptually it ties in nicely with the first movement, but in effect is quite different. The first one is slow and dreamy whereas this one seems on the edge of chaos, with much more rhythmic audio. I'm looking forward to seeing what Ben can do with it.

In terms of the creative process, it seems that Ben is now working to write a piece to go with the new spinners. That is, the music he wrote for the 'imagined' version of the software is not really suitable for the 'realised' version and so it (the music) will need significant revision. I think that the software development process as it is currently is far slower than Ben's compositional process. In other words, he can write good music much more quickly than I can write good software. For the first movement (Spheres) the music and software seemed to co-evolve more easily than they have (so far at least) for this movement. If we weren't living in different cities perhaps this would not be such an issue?

Another note: it helps when I have the trombone with me when I'm coding. Using audio files to test the software just isn't the same as playing live.

Next, some updates. Firstly, Roger Malina is visiting CCS at the moment. His presentation outlined a case for 'deep' interaction between artists and scientists which provides a nice justification for the work Ben and I are doing- although I'm not sure I'm really in the role of 'scientist' in our case.

Also, last weekend was spent up in Brisbane - part ACID orientation, part Ben project work. Was a productive couple of days, largely trying out the prototype spinners (which we both agree are still pretty lame) as well as recording more versions of the spheres piece with (slightly) higher quality audio and generally talking over ideas. A big improvement in the spinners came when I changed the spinners so that they all rotated around the same central point. The picture below doesn't really show this too clearly, but the idea is that there will be a maelstrom of paper (or some other object) spinning around this central point responding to note articulations.

spinners prototype screenshot: now spinning around a single center point...

Ben has taken this a step further in the past few days, talking about the spinners working more like propellers. See this entry in his blog for details...

Later: Keith Martin's PhD thesis- Sound-Source Recognition: A Theory and Computational Model is referenced in the above and could also prove useful.

Follow up: AECT = Association for Educational Communications and Technology.

Oh, and the thesis made mention of the studio for electro-instrumental music in Amsterdam.

I noticed today that Tristan Jehan, who has written some nifty audio analysis externals for max/msp, has his PhD thesis online. Probably worth checking out.

The webpage has a nice overview of approaches to sound synthesis.

...and later: stumbled onto Visual Music: Synaesthesia in Art and Music Since 1900, a book to accompany an exhibition at the Museum of Contemporary Art in LA. Unfortunately, not in the library..

...still later: Very useful bit of extra documentation for pd, describing how to create objects, subpatches, etc dynamically can be found here.

As it's turned out, I've been having a greater influence on the artistic side than either of us expected. I think there are several reasons, but the main one is simply because I have a greater awareness of what the technology is capable of, what has been done before (in this limited sphere - pardon the pun - anyway) and what is probably worth trying. Thinking about this as I write, it occurs to me that our approach is kind of like a very slow-motion improvisation. It's like we're kind of trading fours or something, in that Ben will play what he's been working on, I'll show some of the computer stuff I've been doing and we'll see how it can work. So we start at the 'playing bad jazz' phase, where the music and visuals are rough and don't really mesh together and gradually tweak everything until we start getting somewhere. Who's in charge, for me anyway, is a non-issue - but interesting to think about.

I can understand Ben's concern (based on past experiences) about the visuals dominating the music. One thing that I like about the little we've done so far is that the computer parts are really about building extensions to the solo instrument. That is, the computer generated visuals/aurals are controlled by the trombone and the music that Ben writes. They are not accompaniments to his music and neither is his music a soundtrack for a computer generated multimedia work. Maybe you could say that Ben is composing music for 'augmented trombone'?

...and later that day...found/refound the fftease~ objects for pd, which may have some interesting possibilities for the Ben project.

A sidenote: A key characteristic of the computer stuff seems to be that relatively small changes to the output can have a huge impact on the character of the piece. For example, switching the shape of the spinners from spheres to triangles, to squares, etc changed the style quite radically. Perhaps I'm wrong, but I think this kind of effect is not so prominent in the composition of music. Perhaps it has parallels in orchestration??

On arrival late Saturday morning I set up the extension/revision of the Spheres software (with the Spheres arranged in a circle, forces pushing outwards) and Ben played around with it, trying out bits and pieces of the Spinner composition with it too. I was pleasantly surprised at how quick and accurate the pitch recognition was and how this was reflected in the visualisation. The code tidying I'd worked on for the past couple of weeks really paid off in that respect.

From the point of view of developing toys for musical practice which enhance players understanding of their playing, there was a real 'breakthrough moment' while Ben was experimenting with the circle spheres. When slurring between two notes he noticed that in-between notes were being detected - because the spheres associated with these notes were flickering and moving at the change of note. When he played a truly smooth slur the spheres reflected this. He pointed out how sensitive and 'critical' the spheres were and how he never realised how untidy his slurs could be. It was cool that by playing with the toy he figured out how to play a noticeably smoother (to both of us) slur. I think that with additional tweaking of the various sensitivity settings, etc in the software this could actually work!

Ben spent some time playing with the circle spheres, trying different phrases/styles to see what what worked. During this time, I didn't actually do much to alter the setup because everything was responding ok and it would have seemed disruptive to change things too much while Ben was experimenting.

Eventually we moved on to the Spinners prototype which was in a far less advanced state. I ran the software and showed the spinners as they were - spheres orbiting around (invisible) fixed points. I think there were 3 spinners and low, medium and high frequencies respectively would cause force to be exerted on them. Obviously, the more force the faster the orbit. At this stage, the frequency of the input sound would determine the amount of force. A very simple mapping was used: low frequency = little force, high = large force. Unsurprisingly, this mapping was not very effective. It was too hard to see the relationship between input and visuals.

I made some changes so that volume instead of frequency controlled the quantity of force exerted on the spheres, and this helped. Ben also requested that the spinner that was being acted upon should flash or light up somehow so that it was more obvious what the relationship between musical action and computer reaction was. I hooked them up to the same sort of system that we used for the Spheres piece, so that the spinners glowed blue when they were being acted upon and gradually faded from view when they weren't.

It was interesting how critical the aesthetics of the visualisation were. For me it seemed that once I changed the spinners from spheres to squares, that we both started to see how the piece could work. As Ben was improvising and composing in front of the big screen, I changed the spinners from spheres to triangles, circles, etc. The most effective shape seemed to be squares because, as Ben said, they gave the appearance of papers blowing in a breeze (see screenshot below). At around this time I added more spinners (to a total of 6) to give the screen a busier appearance. We also discussed having a much larger number of smaller spinners (maybe even hundreds?) on screen, each responding to more specific frequencies. This was left for later work as it would have taken too long to do it on the spot. I also experimented in texturing the squares with video/pictures/words/letters, etc. We agreed that they seemed superfluous.

Spinners prototype after the weekend...

Ben felt that the program should provide audio feedback also. He mentioned that when he was playing with the circle spheres that the sounds produced in response to his playing gave him more to work with- provided a kind of platform from which he could launch ideas. I hacked together a quick output patch to give us a starting point. Conceptually, each spinner outputs a frequency within the band that causes it to move. I chose the frequencies arbitrarily for ease of coding reasons. Lowest frequency spinner outputs 60Hz, next 120Hz, 180Hz, etc... The faster each spinner spins, the louder it outputs its particular pitch. The effect is like a drone which changes timbre based on what Ben plays. This worked somewhat, but the drone was too boring and didn't seem to really change enough.

We talked about making use of sampling somehow, so that the piece would record snippets of Ben's playing and play them back. I implemented a recorder which could store short (1-2 second) snippets of live audio into 6 banks (ie. one per spinner), but then stalled when I was trying to think about how recording and playback would be triggered. (eg. audio beyond a certain volume threshold triggers playback, or maybe specific frequencies trigger playback?) We were getting tired at this stage and spent time on the way back home talking about ideas for the audio. The most promising idea is to have the spinners act a little like ratchets- each with their own special timbre and perhaps physical characteristics (ie. hard to spin, easy to spin, etc.). These would act like percussion instruments activated by Ben's live sound. I like this idea! Next step is to try it out...

..and pmpd is working now. The final (I thought minor) dependency for the spheres piece was maxlib (which is only used for scaling, but is very handy). It took a while, but after downloading the latest cvs version of the pd 'externals' from sourceforge, I eventually got it to compile. The trick was to copy the 'include' directory from ../externals/build/ to ../externals/maxlib and then do the 'make pd_linux' command, which seemed to work ok.

One day I might get some time to have a look at these?

Also, Ernest has requested that publications for 2005 be blogged. Here they are:

• Johnston, A., Amitani, S. and Edmonds, E. 2005, "Amplifying Reflective Thinking in Musical Performance", Creativity and Cognition 2005, London, ed. Linda Candy, ACM Press, New York, pp. 166-175
• Weakley, A., Johnston, A., Edmonds, E., Turner, G. 2005, "Creative Collaboration: Communication, Translation and Generation in the Development of a Computer-Based Artwork", 11th International Conference on Human-Computer Interaction 2005, ed. G. Salvendy, Lawrence Erlbaum Assoc Inc., Las Vegas, Nevada USA. CD ROM.
• Johnston, A., Marks, B. and Edmonds, E. 2005, "'Spheres of Influence' - an Interactive Musical Work", Interactive Entertainment (IE2005), Sydney, Australia (in press)
• Johnston, A., Marks, B. and Edmonds, E. 2005, "An Artistic Approach to Designing Visualisations to Aid Instrumental Music Learning", Cognition and Exploratory Learning in Digital Age (CELDA 2005), Porto, Portugal (in press)