Last year, my friend Chris Novello closed the 2012 Open Hardware Summit with a demo similar to this performance — remixing SMB in ways nobody had ever thought of before — to wild and well-deserved applause, I might add. Here he is doing it again! He writes:

In this video, I directly manipulate the RAM state of Nintendo’s Super Mario Brothers to transform it from a game into a strange instrument.

First, I play the game as it is traditionally played.. but I have access to the game’s memory, so I change Mario’s Y position using the Madrona Labs Soundplane (a surface that sends data to the computer about where it is being touched). This is how I hover Mario during the playthrough.

Also, before I start playing, I flip a switch on illucia that I assigned to trigger recording — not video, but actually recording the entire memory state of the NES for each game frame.

Think about it – Mario’s universe is held in RAM, which the NES uses to draw his world for each frame of the game. By recording the entire state of the NES memory for every frame, I’m able to go back to any moment in Mario’s life.

So then I use the X-axis of the Soundplane to sweep through the timeline of Mario’s universe.

Not only that, but the Soundplane is multitouch, so I use a second finger to specify start and endpoints in a playback loop. Technically, this is similar to the way samplers and granular synths work in audio.. but with the entire memory state of the NES. Conceptually, it is like Super Mario meets Groundhog Day. Mario’s universe computer/time machine gets caught in hellish loops.

Then I start using illucia to send alien data into various other places in Mario’s universe, which makes for all sorts of audiovisual insanity amidst the spacetime loops. This is sort of like circuit bending, but in a protected sandbox – at any point I can revert back to the clean recording of RAM states (aka moments in Mario’s universe).

I then try to go back to “playing” the game, watching Mario navigate a melting world of glitched-out ephemera. I then push things into full on glitch insanity. I use a pair of rubberband mallets on the Soundplane to jump around in Mario’s universe while leaving illucia to send a heavy stream of alien data into Mario’s RAM state. I eventually (accidentally/luckily) land at a place that triggers the game over music, and end the take.