silvershard.technology
technology-enabled improvised dance music performance

silvershard.com   technomancy    carpeDM     articles    glossary


ADAPTIVE MUSIC
by Andrew Clark

   This article explores adaptive music in the context of language. This is because language is much easier to demonstrate and discuss in the written medium than music is! So, despite the title "Adaptive Music," this article doesn't talk about music at all - except by analogy.

an adaptive poem
   I used to write video game music for a living. It was a very exciting job, but it also had its share of frustrations. For instance, part way through one particular project, I discovered that the lead programmer hadn't bothered to install a sound card in his new computer. As one might expect, this was making it kind of hard for him to test the interactive score I was writing. ( True story! The next part is made up though. )
   I decided that it might be a good idea for me to write some interactive poetry for him to work with temporarily. ( We could even keep it as an option in the release version of the game, for the hearing impaired. ) Like the music, the poetry would change in real time in order to reflect the changing action of the game and enhance the atmosphere. It would be of extremely high quality, and interesting enough that the player wouldn't get sick of it after a mere eight hours of game-play.

an adaptive poem?!!
   Writing adaptive poetry sounds impossible, doesn't it? Well, it's essentially the same challenge that an adaptive music composer faces… but more on that later. For now, let's try to prototype an adaptive poem and see what happens. The first step will be nailing down the tone and style, and what we want the piece to communicate. The following example is a linear poem that will help us figure out what we want the interactive version to do. ( The game, by the way, is called Seal Hunter and features the heroic Captain Brave-O. )

example 1: linear poetry

The cold dawn breaks on the arctic ice
Gleaming hard on a cruel device:
Captain Brave-O's clubbing hook
Dripping blood from the lives it took

The evil seals are everywhere
Breathing our precious human air
Soon the air will all be gone
And no more human breaths be drawn

Then: BAM BOOM BANG, KER-CHUNK, KER-SPLAT!
The hook is WIELDED like a BAT
CAPTAIN BRAVE-O's in the FRAY,
HORDES of EVIL SEALS to SLAY!

His breathing slows, the battle done
This war, he knows, is far from won
And though there's not a seal in sight
It's far too quiet to be all right

adaptability at the word level
   The action and mood of Seal Hunter can change quite suddenly. Captain Brave-O could be walking around being brave and heroic one minute, and then diabolical seals could leap out at him the next! Ideally, our accompanying poem should be just as dynamic. Unfortunately, since the game player will be controlling Captain Brave-O's actions, we have no way of knowing in advance what is going to happen, when. One way to try to deal with this is to compose a poem that is flexible at the word level, so that it can change moods at any time.
   The next example randomly selects words from one of four word lists. Each list contains only words that are appropriate to a particular game mood. When the action of the game changes, the source word list is also changed, thereby changing the mood of the poem on the fly.

example 2a: adaptive word texture

( The level begins as Captain Brave-O enters heroically. Poem mood: HEROIC ) Captain Brave-O brave hero marching gleaming brave hero Captain brave adventure marching hook Brave-O brave brave hero ( The landscape changes; seals are nearby. Switch poem to CREEPY ) danger fear shadows lurking seal evil danger caution scary ( Seal surprise attack! Switch poem to COMBAT ) bam ker-chunk ker-splat hook bat fray slay bang ker-splat bat slay slay ker-chunk bam ker-splat slay slay slay ( The last seal is subdued but the territory is still scary. Also, Captain Brave-O's health is low. Switch to CREEPY, INJURED combo ) injured hurt danger fear injured lurking seal evil danger caution shadows

   That was surprisingly effective and very dynamic; however, aesthetically it might be considered somewhat lacking. It was more like word texture than traditional poetry -- there was no real attention to rhyme or meter. Let's see what happens when we try a less random ordering of words in an attempt to address this:

example 2b: adaptive word texture with meter and rhyme

( The level begins: HEROIC )
mighty gleaming courage gleaming
roaming courage hero marching
Brave-O mighty Captain marching
Gleaming courage courage roaming
Mighty courage Brave-O Brave-O
Gleaming mighty roaming hero
Courage Captain gleaming marching
Hero mighty --
( Surprise seal attack! COMBAT )
fray seal bam bat
slay slam spray splat
hook fray squeal slam
bat seal slay bam
fray slay fray seal
splat…
( The battle ends. )

   The second example was essentially the same as the first, except for two small changes. This time, the word list for a particular mood consisted only of words sharing the same foot. Additionally, I added special lists of rhyming pairs. Every fourth word was selected from this list, in order to form rhyming couplets. The result was a kind of pseudo phrase structure ( but without any real sentence-level meaning ).
   This more structured version came at a slight cost. Notice that when the sudden seal attack happened, it broke the rhyme and rhythm scheme of Brave-O's strut poem. If we'd waited for the line to finish, the attack poetry would have been late and much of the effect would have been lost. Having to make this kind of compromise makes this solution less dynamic than pure word texture. On the other hand, patterns of rhyme and meter do have an undeniable attraction.

patterns
   The human brain is fantastically adept at recognizing all kinds of patterns. The human brain is fantastically adept at recognizing all kinds of patterns. The human brain is fantastically adept at recognizing all kinds of patterns. ( I bet you skipped at least that last sentence without even reading it. ) Patterns in any medium involve some form of repetition, and have a certain aesthetic appeal. Unfortunately, with over-repetition comes boredom. Repetition provides unity and cohesiveness; variation provides interest. This is always a delicate balance!

example 3: patterns -- repetition and variation

3a
thbifaarakopthbifaarakofthbifaarakopibysaltls
( no pattern )

3b
t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t t
( a boring pattern )

3c
t t t t  t t t  t t t t  t t t  h h h h  h h h  t t t t  t t t
( less boring pattern )

3d
t t t t  t t t  t t t t  t t t  h h h h  h h h  t t t t  t t t
t t t t İt t t İt t t t İt t t İh h h h İh h h İt t t t İt t t
t t t t İt t t İt t t t İt t t İh h h h İh h h İt t t t İt t t
t t t t İt t t İt t t t İt t t İh h h h İh h h İt t t t İt t t
t t t t İt t t İt t t t İt t t İh h h h İh h h İt t t t İt t t
( boring again )

3e
t t t t  t t t  t t t t  t t t  h h h h  h h h  t t t t  t t t
t t t t İt t t İt t t t İt t t İh h h h İh h h İt t t t İt t t

t t t t İt t t İt t t t İt t t İh h h h İh h h İt t t t İt t t

t t t t İt t t İt t t t İt t t İh h h h İh h h İt t t t İt t t
t t t t İt t t İt t t t İt t t İh h h h İh h h İt t t t İt t t
( less boring )

   What is interesting about these examples is how important scale is when balancing cohesiveness and interest. For example, when your brain looked at boring pattern 3b, it saw repeating "t"s. Boring. When it looked at less boring pattern 3c, it saw the same repeating "t"s; however, its interest was held by the larger grouping patterns that began to emerge. Spend some time with each of the patterns and think about the meaning of "t" in each one. I'm serious! There is a lot to be learned from these simple examples.

boring adaptive poetry
   Think way back to the "adaptive word texture" examples ( 2a, 2b ). Which adaptive poem would get boring faster? Most definitely it's the "adaptive word texture with meter and rhyme" version. In that example, two patterns ( meter and rhyme ) repeat constantly without variation. By the second couplet, the pattern has become recognizable. By the third, it has become predictable. By the fourth, it begins to become tedious.
   One might argue that the "adaptive word texture" poem's structural organization is in fact more repetitious and therefore more tedious. After all, it's essentially just word, word, word, word, word… Why isn't this the case? In the first version the simplicity of the pattern works in its favor. It's unobtrusive -- it's basically a non-pattern. It doesn't distract from the interesting unpredictability of what the next word might be. By contrast, the higher level structural patterns in the second version engage the part of the brain that pays attention to higher level structural patterns -- and becomes bored with them. In this case, the ( repetitious ) structure dominates the ( non-repetitious ) content. This unbalance is due in part to the fact that our content only works at the word level, while the patterns ( and the brain ) are working at the phrase level.

adaptability at the phrase level
   The problem with word textures is that they are extremely limited in terms of actually expressing ideas. Compare "danger fear shadows lurking seal evil danger" ( from "adaptive word texture" 2a ) to "the enemy could attack at any moment." The first "sentence" is a pseudo-random collection of symbols. The second uses syntax to combine symbols into a coherent thought. It actually says something. ( And you thought syntax was only important for programmers! )
   What happens when we try to write an adaptive poem that actually communicates something at the sentence ( phrase ) level? A logical place to start might be with a sentence texture. This will work in much the same way as the word texture example, except that we will use four lists of complete sentences.

example 4: adaptive sentence texture

( SPOOKY cue ) This place is really scary. Is that shadow moving? It's really dark here. The enemy could attack at -- ( COMBAT cue ) Hot blood spurts like a fountain! The fighting is fast and furious! The violence is incredibly intense and visceral! Teeth are gnashing and flippers are slashing! The fighting is fast and furious! Seal bits are flying everywhere!

   An important thing to consider is the production cost of sentence texture compared to word texture. A word list that is twenty elements long requires only twenty words. A sentence list that is twenty elements long requires more than one hundred words! Plus, the production of a sentence involves a process of composition.
   Variety at the sentence level requires a bigger budget than the same amount of variety at the word level. On the other hand, a single sentence covers more "ground" in terms of words and time. The "adaptive sentence texture" example 4 used far fewer elements than the"adaptive word texture" example 2a, but was just as long. In terms of pure variation over time, perhaps fewer sentences are required.
   Another issue is that complete ideas demand more attention from the player. The danger is that instead of enhancing gameplay, we're going to start distracting the player from it. There's an old saying among film makers: "The best soundtrack is the one you never hear." ( You should be feeling it instead. ) The idea is that if you notice the music, it's because the composer's done something wrong. Well, the same goes for adaptive poetry. We want the player to respond on an emotional level, not an intellectual one. Complete sentences are dangerously high level.
   Lastly, sentence texture is a lot less flexible than word texture. It takes longer to state an idea than it does to state a word. Our "adaptive word texture" example 2a could switch moods literally in midstream. However, when dealing with adaptive structures at the phrase level, we're suddenly faced with logistical and creative concerns about what to do when the game mood changes in mid-sentence. How do we make a smooth and effective transition?

transitions
   That was a pretty good transition, wasn't it? It was even at a higher structural level than mere sentences, or even paragraphs. I changed sections almost seamlessly! OK, not so impressive. I already knew where I was going, and timing my arrival wasn't critical. So, I was able to compose my sections so that they fit together just so.
   In adaptive poetry, timing is critical. In order to enhance the mood, changes have to happen with the mood. If cues are delayed, even by a second, they aren't contributing to the emotional experience of a game event. They become their own distinct event, separate from the associated game event - which the player's brain will already have had time to process. Poetry mood changes have to happen on time!
   In the "adaptive sentence texture" example 4, this wasn't a big deal. When the seals sprung their surprise attack, it made sense to suddenly interrupt the previous mood. Ending a combat mood might be more complicated, though. Imagine that the sentence "When will the carnage end!?" is one of our combat list elements. Imagine that Captain Brave-O strikes down his last opponent midway through the playback of this particular sentence. We want to switch to victory poetry, and we don't want to be late. We also don't want to be awkward. One way to address this situation is to compose transition versions of the combat sentence. If the mood changes midway through the statement, we continue smoothly with the transition version.

example 5: adaptive sentence texture with transitions

When will the carnage end!?
( the original sentence )

When Captain Brave-O fights, he always wins!
( victory after one word )

When will they finally learn not to mess with the big C.B.?
( victory after two words )

When will the crazy critters learn?
( victory after three words )

When will the carnage be sweeter than this?
( victory after four words )

When will the carnage end? Right now, in victory!
( victory after five words )

   Immediately, the cost of this approach should be obvious. Each element of the sentence list will require similar treatment if the transition to victory is to be possible at any time. Additionally, transitions to other moods may be necessary. ( "When will Captain Brave-O be deader than this?" ) On the other hand, it does what it's supposed to de very well. What more could we ask for?

going for broke

example 6: adaptive sentence texture with meter, rhyme, and transitions

( HEROIC cue )
Captain Brave-O, he's our man
Bravest soul in all the land
Hard as steel and twice as smart
Tough as nails and…
( SPOOKY cue )
…knows these parts
Wow, it's really scary here:
Smells of fish and smells of fear
Is that a shadow moving there?
Damn seals could be…
( COMBAT cue )
…oh, beware!
Bam boom bang, ker-chunk, ker-splat!
Filthy seal, do you like that?!
Seal bits flying everywhere
In a frenzy, just don't care
When, oh when, will the…
( VICTORY cue )
…seals try again?
Victory is ours, my friends!

knowing your arsenal
   As these examples get progressively high-level in terms of structure, I am reminded a little bit of the Seal Hunter weapon selection options. The UltraNinja2000 ( a light, fast club ) is perfect for lightning strike strategies, but doesn't do very much damage. The more expensive Crusher is slower and difficult to wield effectively, but makes an impressive impact when it connects! Each weapon has its place, and is useful in different game situations. In fact, choosing the right tool for the job is a big part of winning the game.
   The same thing is true of the different adaptive strategies that we have examined. Each has advantages and disadvantages. There is no one perfect choice that is right for every game context. The best approach is to use all of them as necessary, applying the most effective solution on a case by case basis. "What do we want to hit the player over the head with for this part?" Choosing the right tool for the job is a big part of winning the game. It also helps to have a secret weapon handy…

adaptability at the letter level
   Those of you who have been paying attention to the structure if this paper will have been expecting either A) an example of adaptive poetry combining multiple approaches or B) "adaptability at the paragraph level." Instead, in a surprise move, I am unveiling a secret weapon that's been hiding under our noses all along!
   It's easy to overlook adaptability at the letter level because we don't normally think of packing much useful meaning into a single letter. If we could, it would offer incredible advantages in terms of flexibility. ( Even faster than the UltraNinja2000! )

example 7: adaptive letter texture

( The level begins as Captain Brave-O enters heroically. Poem mood: HEROIC ) thbifaarakopthbifaarako ( The landscape changes; seals are nearby. Switch poem to CREEPY ) pthbifaarakopibysal ( Seal surprise attack! Switch poem to COMBAT ) TLSWERIPIAMISFORAHACAA ( The last seal is subdued but the territory is still scary. Also, Captain Brave-O's health is low. Switch to CREEPY, INJURED combo ) uwocbrpuacvpitiaadb

   Shockingly, that little example of pure adaptive letter texture was packed full of meaning! How could this be? Well, don't forget that the game is its own narrative. The player already knows what's going on. Our job as adaptive poets is to merely to enhance the mood of the story -- telling it is somewhat redundant. Still, adaptive letter texture on its own is somewhat limited in terms of what it can express.
   ( HEROIC ) The real power of adaptive letter texture is its ability to com- ( CREEPY ) -bine with other techniques. It is fully com- ( COMBAT ) -PATIBLE WITH ALL OF THE OTHER APPROACHES THAT WE HAVE DISCUSSED. IT COULD even BE USED TO add AN ADAPTIVE element to PRE-COM- ( CREEPY, INJURED ) -posed material… and even com- ( NEUTRAL, INJURED )-pletely unrelated prose!
   Notice too that with this technique we've added an "analogue" element to our mood swings. Instead of limiting moods to either ON or OFF, BLACK or WHITE, we've added some shades of gray ( and red… ). For instance, the frequency of red characters ( or completely red words ) can indicate the status of Captain Brave-O's health meter, without interfering with the content or structure of other levels.

Seal Hunter: the final battle
   Now that I've introduced my secret weapon, I'll leave you with one final example. The final example demonstrates how one might combine some of these approaches into a complete adaptive game poetry solution. Don't forget that the purpose of this exercise has been to learn something about adaptability in structured art forms, so that we can apply it to making interactive video game audio soundtracks. Meanwhile, think about the example that follows. What techniques are being used? Why did the game's Poetry Director make the choices he did? How much would this cost to implement and produce?

example: Seal Hunter, Level 10

Captain Brave-O's back again,
He'll kick tail in Level 10!
Go Captain Brave-O, Go!


Tum te tum…


Tum te tum…

Tum te tum

Tum-tum

Tum-tum

Tum-tum tum-tum

Tum-tum tum-tum tum-tum tum-tum tum-tum tum-tum tum-tum tum-tum
SEAL ATTACK! SEAL ATTACK!
THIS ONE'S BIG BUT DON'T TURN BACK!
GO CAPTAIN BRAVE-O, GO!
BAM KER-CHUNK OW! KER-SPLAT HOOK bat FRAY SLAY BANG ker-splat OW! bat SLAY SLAY ker-chunk bam ker-splat SLAY slay slay
His breathing slows, the battle done
This war, he knows, is far from won
And though there's not a seal in sight
It's far too quiet to be all right


Tum te tum

Annotated Bibliography

Hofstader, Douglas R. Gödel, Escher, Bach - 20th Anniversary Edition: An Eternal Golden Braid. New York: Basic Books, 1999.
Hofstadter, Douglas R. Le Ton beau de Marot - In Praise of the Music of Language. New York: Basic Books, 1997.
   Both Hofstader works explore symbol and meaning, form and content. The former has cognition and self-awareness as its central theme, while the latter is about communication and relationships. Hofstader draws deep and engaging analogies between his themes and the systems and structures that he explores ( math, art, and music in the former; poetry, translation, and language in the latter ). Frequently ( particularly in GEB ), the very structure and/or form of his writing demonstrates the content - an impressive device that adds an extra dimension to his exposition. Both of these books contributed enormously to the form, content, and even tone of my little paper.

Jourdain, Robert. Music, the Brain, and Ecstasy - How Music Captures Our Imagination. New York: Avon Books, Inc., 1998.
   A very broad survey of the phenomenon of music, from the physical and biological to the psychological and the sublime. Not an academic reference work, it's very readable - yet obviously exhaustively researched across many disciplines. Especially interesting to me were the explorations of how different elements of music ( particularly patterns at different structural levels ) are processed in different parts of a listener's brain, and the incredibly complex relationships that are involved in even the perception of music. OK, put it like that and it sounds boring, but seriously - anyone who is even remotely interested in music or audio will thoroughly enjoy this book. I'm not kidding, buy it! The takeaway is universal, despite a fairly western-classical-tradition-centric focus. Room for a sequel I guess…

Schoenberg, Arnold. Fundamentals of Musical Composition. London: Faber and Faber, Ltd., 1970.
   You wouldn't think that one could learn how to compose from a book, but something special clicked into place for me with this one. Partly the timing was just right - years of intensive training probably had something to do with it… Whatever it was, I really did understand music differently after reading this book. Unexpectedly ( considering Schoenberg's prominent role in the deconstruction of tonality ), the book limits itself strictly to traditional classical composition. It doesn't discuss or challenge tonality at all; it merely explains how and why the structures work. Despite the specific focus, there are lessons to be learned here about cohesiveness of form, repetition, and variation that can be applied to structures in any genre ( …any medium? ). Kind of an enigma is that the structure of the book itself is confusing at some points almost to the point of loosing the point entirely. Also, unlike the other works cited in this bibliography, it's extremely dry - unless you're really interested in classical music theory. Nonetheless, for me it was a gem. Grab it… if you can find it!

 

http://www.silvershard.com