Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Experiential Audio

Experiential Audio

Audio Spatialization Effects with the Web Audio API

Janessa Det

October 22, 2012
Tweet

More Decks by Janessa Det

Other Decks in Programming

Transcript

  1. audio Audio Spatialization Effects in the Browser Janessa Det -

    WEBHACKER @TWITTERNYC experiential summary nodeS2noTE & PANNING START & resources THE WEB AUDIO API APPLICATIONS MOTIVATION sound&space CONVOLUTION DSP COMPONENTS SPATIALIZATION experiential audio primer SIMULATING ENVIRONMENTS
  2. whoami. experientialaudio janessa det (@jandet) @twitternyc Software engineer - web

    Discovery Front-End thefuture.fm lead front-end dev DJ Defined Radio columbia university M.S. Computer science Graphics & Vision > UX/HCI summary
  3. sound&space. sound in the digital age experientialaudio janessa det (@jandet)

    time vs. frequency digitalization browsers are 2d <audio></audio> sound has become flat summary
  4. sound&space. sound defines a space experientialaudio janessa det (@jandet) summary

    Sound interacts with surrounding space 3d space environment Environments perceived through sound human perception One of the key ways we perceive our surroundings Janet Cardiff The Forty Part Motet [1]
  5. primer. alas! the web audio api experientialaudio janessa det (@jandet)

    compatibility features • audio signals processing for your browser • easy to create, process, and control audio within web applications • complements webgl • chrome 10.0-24.0 • safari 6.0 • safari ios 6.0 • firefox NOOOOPE (BUT PLANNED) • AUDIO DATA API summary
  6. audio context primer. audio routing graph experientialaudio janessa det (@jandet)

    summary audio DESTINATION gain (volume) audio source convolver filter(s) audio source if  ('AudioContext'  in  window)  {    context  =  new  AudioContext(); }  else  if  ('webkitAudioContext'  in  window)  {    context  =  new  webkitAudioContext(); }  else  {    throw  new  Error('AudioContext  not  supported!'); }
  7. nodes2note. types of audio nodes experientialaudio janessa det (@jandet) •

    raw audio data into the system • final node to connect to • outputs to lineout • speakers/headphones AUDIO SOURCE AUDIO DESTINATION var  request  =  new  XMLHttpRequest();        request.open("GET",  options.path,  true);        request.responseType  =  "arraybuffer"; request.onload  =  function()  { createSoundSource(request.response); }; request.send(); soundSource.connect(context.destination); function  createSoundSource(audioData)  { soundSource  =  context.createBufferSource(); soundSource.buffer  =   context.createBuffer(audioData,  false); soundSource.loop  =  true; }; summary
  8. nodes2note. types of audio nodes experientialaudio janessa det (@jandet) •

    apply gain to change volume • 0: lowpass • 1: highpass • 2: bandpass • 3: lowshelf • 4: highshelf • 5: peaking • 6: notch • 7: allpass filterNode  =  context.createBiquadFilter(); filterNode.type  =  0; filterNode.frequency.value  =  220; volumeNode  =  context.createGainNode(); volumeNode.gain.value  =  0.5; summary gain (volume) filters
  9. types of audio nodes experientialaudio janessa det (@jandet) summary •

    delay audio for specified duration nodes2note. delay oscillator delayNode  =  context.createDelayNode(); delayNode.delayTime.value  =  5; • 0: sine • 1: square • 2: sawtooth • 3: triangle • 4: custom oscillatorNode  =  context.createOscillatorNode(); oscillatorNode.type  =  0;
  10. types of audio nodes experientialaudio janessa det (@jandet) summary •

    scriptprocessornode • synthesize and process audio directly with javascript • custom DSP effects • exposing internals of dsp to web applications nodes2note. fancypants js JS http://www.w3.org/TR/webaudio/
  11. audio context audio routing graph experientialaudio janessa det (@jandet) summary

    audio DESTINATION gain (volume) filter audio source nodes2note. //  initialize  nodes ... sourceNode.connect(filterNode); filterNode.connect(volumeNode); volumeNode.connect(context.destination);
  12. spatialization. INTRODUCTION TO panning experientialaudio janessa det (@jandet) panning algorithm

    • distance effects • sounds softer farther away • orientation • what direction are you facing • channels • adjust volume to l & R outputs • mimick human perception of space [1] summary
  13. USING PANNING nodes experientialaudio janessa det (@jandet) summary • connect

    to an audio source • makes sound source positional • sound cones • inner angle • outer angle • outer gain • note: velocity! • one listener per context • position (x, y, z) • orientation • front vector • up vector spatialization. panning node listener : ) pannerNode  =  context.createPanner(); pannerNode.setPosition(x,  y,  z); pannerNode.coneInnerAngle  =  30; pannerNode.coneOuterAngle  =  40; pannerNode.coneOuterGain  =  0.5; context.listener.setPosition(x,  y,  z); var  front  =  getFrontVector(angle); var  up  =  [0,  0,  1]; context.listener.setOrientation( front.x,  front.y,  front.z,   up.x,  up.y,  up.z);
  14. convolution. INTRODUCTION TO convolution experientialaudio janessa det (@jandet) what is

    convolution? h(t) x(t) y(t) = h(t) x(t) • “cross-multiplying” 2 signals • LTI system theory • impulse response summary
  15. USING convolver nodes experientialaudio janessa det (@jandet) summary • set

    buffer as impulse response • connect sound source to convolver node • convolves input through impulse response • emulate sound in any environment convolution. convolver node var  request  =  new  XMLHttpRequest();        request.open("GET",  "sounds/ir/church.m4a",  true);        request.responseType  =  "arraybuffer";       request.onload  =  function  ()  { createConvolverNode(request.response); } request.send(); function  createConvolverNode(audioData)  { convolverNode  =  context.createConvolver(); convolverNode.buffer   =  context.createBuffer(audioData,  false); }
  16. position orientation audio context audio routing graph experientialaudio janessa det

    (@jandet) summary audio DESTINATION convolution. //  initialize  nodes ... sourceNode.connect(convolverNode); convolverNode.connect(pannerNode); pannerNode.connect(context.destination); panner convolver audio source
  17. demotime. experientialaudio janessa det (@jandet) interaction controls START sound&space nodeS2noTE

    SPATIALIZATION CONVOLUTION APPLICATIONS primer MOTIVATION THE WEB AUDIO API DSP COMPONENTS & PANNING SIMULATING ENVIRONMENTS & resources experiential audio summary • player • listener • panning sound • sound • panning sound • panning sound • panner node • sound node classes • wasd - movement • panning sound • q/e - rotate ccw/cw • x - make tweeting sound • expansions to come panner-demo.jit.su
  18. experientialaudio janessa det (@jandet) summary future work areas to explore

    • VISUAL INTERFACE & TOOLS TO BUILD AUDIO ROUTING GRAPHS • INTERFACING WITH WEBGL explore full 3d orientation & positioning • INVESTIGATION OF AUDIO DATA API closer to the actual dsp implementations • WEB AUDIO API ON IOS6 SAFARI different ui, applications, and event handling resources. • rtfm http://www.w3.org/TR/webaudio/ • safari developer library http://bit.ly/RcVHxY • DAFX (ref [1]) • JOHN WILEY & SONS • MAR 8, 2011 • impulse responses http://fokkie.home.xs4all.nl/IR.htm
  19. SUMMARY PRESENTATION OVERVIEW GOBACK ORCHOOSEAsection applications & resources spatialization &

    Panning simulating environments convolution nodes2note dsp components START experiential audio sound&space motivation the web audio api primer