This is a research project I did that was intended to be an article about the comparison of developments in Data Sonification and Knowledge Representation.
Data Sonification is a field of processing data and rendering it as audio. Knowledge Representation is a sub-field of Artificial Intelligence that tries to de-abstract data into formats that are useful for computer systems. I found that both of these fields wrestle with similar notions of the abstract nature of data and the mapping of said data onto fixed, purpose-driven representations.
The article was not completed in it's original intention because I was not happy with the results it was producing. However, I do think the outlines I constructed and compiled from my research survey could be useful for future research. I gave a public presentation at the iARTA Hybrid Arts Lab Open-House at the University of North Texas in May 2013 with to elaborate on some of my findings. Below, you will find a PDF of the presentation slideshow, text of some of my notes and outlines, and a working bibliography (mostly of data sonification articles).
I am still interested in this project and may return to this work but for now, this page will be a repository for some of my notes.
Apoligies that nothing has citations. Please see the bibliography below and I can point out that my two primary sources at this stage were Davis and Hermann.
iARTA Hybrid Arts Lab Open-House Presentation
Preliminary Outline
I. Introduction
A. Identification of problems
1. Learning Curve
2. Data vs. Concept Mapping
3. Immediate Knowledge Acquaintance
4. Temporal Proximity Weight
5. Aesthetic Intentions
6. Possible Multiplicity of Views
B. Relevance of Electronic Instrument Models
C. Clarification of Terminology
D. Critique of Result Evaluation
1. Aesthetics vs. Practicality
2. Psychology of Taxonomy
II. Goal Orientation
A. Complexity of Goal Mapping
1. Learning
2. Intuition
3. Designer Mapping
4. User Feedback
a. Data Interaction
b. Mapping Function Interaction
c. Mapping Scale Interaction
d. Mapping Filter Interaction
e. Mapping Dimension Navigation Interaction
f. Pattern Recognition Temporality
B. Spectral Node Mapping
1. Point Events
a. Uni-Dimensional
b. Multi-Dimensional
2. Region Events
3. Weighted Region Events
4. Continuous Spectral Bands
C. Systemic Result/Acoustic Feedback
III. Techniques
A. Event vs. Continuous
1. Temporal Imperative of Goals
2. Temporal Nature of User/Data Feedback
B. Systemic Result/Acoustic Feedback
1. Open Loop
2. Closed Loop
3. Direct Interface
4. Peripheral Interface
C. Sonification Feedback Network
1. Intentional Action
a. Interface Physicality
b. Interface Mentality
c. Interface Emotion
2. Inverse Interface Mapping
3. Separation of Feedback Networks
a. Data Learning
b. Action Learning
IV. Abstract/Computed Physical Models
A. 3D Rigid Body Physics
1. Explanation of Terms
2. Critique of Demonstration
3. Critique of Unified Technique Model
B. Fluid Physics
C. Acoustic Physical Models
V. Philosophical Appraisal of Unified Theories
A. Map Philosophical Aesthetic Techniques
1. Music/Sound
2. Visual Art
3. Information Theory
B. Behavioral Science
VI. Conclusions
A. Anti-ad-hoc argument
1. Unified Theory vs. Unified Technique
B. Potential Amendments
C. Future Directions
Notes:
User Interaction - Computer System Learning
Relationship of Peripheral/Direct to Learning System
Temporality of Learning Recognition (Adaptive System and User)
Interface Physicality vs. Mentality
Evaluation of Information Spectra (Event vs. Continuous)
Interface Emotion?
Audio Games
Task Type Analysis
Sonification Task Types
1. Monitoring
2. Awareness of process / situation
3. Data exploration
4. Point Estimation / Comparison
5. Trend Identification
6. Data Structure Identification
7. Exploratory Inspection - (no a priori questions)
8. Multimodal tasking performance
Knowledge Representation Designations
1. Surrogacy within the viewer (stand in for the external)
a. Identity
b. Fidelity
2. Set of Ontological Commitments
a. Accumulation in Layers
b. Not a Data Structure (semantic network vs. graph)
3. Fragmentary Theory of Intelligent Reasoning
a. Fundamental conception of inference
b. Set of inferences that represent sanctions
c. Set of inferences it recommends
- deduction vs. human behavior vs. machine architecture vs. probability vs. utility
4. Medium for Efficient Computation
a. Event frames
b. Organization into taxonomic hierarchies
5. Medium of Human Expression
Signal flow for Sonification Model
- (sub-module toggles)
1. User-Designated Task Identification
2. Automated Task Identification
a. User Input Monitoring
b. Multi-layer task decision
-Intelligent Reasoning Model Decision?
3. Scaling
4. Filtering/Smoothing
5. Active Region Identification
6. Output Format Selection
a. User Selection
b. Automated Selection
c. Hybrid Selection
Preliminary Bibliography
Apoligies that this bibliography has some incomplete entries. I've tried to reconstruct it from my notes but I didn't go back and do the work to find all of the sources.
Katy Börner, Atlas of Science, All School Day, University of North Texas, Denton, TX. October 1, 2011.
Ballora, Mark, Bruce Pennycook, Plamen C. Ivanov, Leon Glass and Ary L.Goldberger. “Heart Rate Sonification: A New Approach to Medical Diagnosis,” Leonardo 37: No. 1 (2004): 41–46.
Barrass, Stephen, Mitchell Whitelaw and Freya Bailes. “Listening to the Mind Listening: An Analysis of Sonification Reviews, Designs and Correspondences,” Leonardo Music Journal 16 (2006): 13–19.
Barrass, Stephen and Gregory Kramer. “Using Sonification,” Multimedia Systems 7 (1999): 23–31.
Ben-Tal, Oded and Jonathan Berger, “Creative Aspects of Sonification,” Leonardo 37: No. 3 (2004): 229–232.
Chen, Kai, Janos Sztipanovits and Sherif Adbelwahed. “A Semantic Unit for Timed Automata Based Modeling Languages,” Proceedings of the Twelfth IEEE Real-Time and Embedded Technology and Applications Symposium (2006).
Davis, Randall, Howard Shrobe, and Peter Szolovits. “What is Knowledge Representation?,” AI Magazine 14: 1 (1993): 17–33.
Degris, Thomas and Joseph Modayil. “Scaling-up Knowledge for a Cognizant Robot,” Association for the Advancement of Artificial Intelligence (2012).
Delatour, Thierry. “Molecular Music: The Acoustic Conversion of Molecular Vibrational Spectra,” Computer Music Journal 24, No. 3 (2000): 48–68.
Drioli, Carlo and Davide Rocchesso. “Acoustic Rendering of Particle-Based Simulation of Liquids in Motion,” ___
Dunn, John and Mary Anne Clark. “’Life Music’: The Sonification fo Proteins,” Leonardo 32, No. 1 (1999): 25–32.
Eldride, A. C. “Issues in Auditory Display,” ___ (2005).
Harmann, Thomas. “Taxonomy and Definitions for Sonification and Auditory Display,” Proceedings of the 14th International Conference on Auditory Display (2008).
Hermann, Thomas, Andy Hunt, and John G. Neuhoff, ed. The Sonification Handbook Logos Verlag: Berlin, 2011.
Jones, Stuart. “Space-Dis_Place: How Sound and Interactivity Can Reconfigure Our Apprehension of Space,” Leonardo Music Journal 16 (2006): 20–27.
Kasavech, M. P. Boulanger, W. F. Bischof, and M. Garcia. “Augmentation of Visualisation of Using Sonification: A Case Study in Computational Fluid Dynamics, The Eurographics Association Short Papers (2007).
Klein, Eric and Oliver G. Staadt. “Sonification of Three-Dimensional Vector Fields,” Center for Image Processing and Integrated Computing ____
Kobbacy, K. A. K. “On the Evolution of an Intelligent Maintenance Optimization System,” The Journal of the Operational Research Society 55, No. 2 (2004): 139–146.
Labib, A. W., G. B. Williams, and R. F. O’Connor. “An Intelligent Maintenance Model (System): An Application of the Analytic Hierarchy Process and a Fuzzy Logic Rule-Based Controller,” The Journal of the Operational Research Society 49, No. 7 (1998): 745–757.
Le Groux, Sylvain, Jonatas Manzolli, and Paul F.M. J. Verschure. “VR-RoBoser: Real-Time Adaptive Sonification for Virtual Environments Based on Avatar Behaviour,” Proceedings on the 2007 Conference on New Interfaces for Musical Expression (2007): 371–374.
Lodha, Suresh K., John Beahan, Travis Heppe, Abigail Joseph, and Brett Zane-Ulman. “MUSE: A Musical Data Sonification Toolik,” University of Califorinia, Santa Cruz.
Martins, Antonio Cesar Germano and Rangaraj Mandayam Rangayyan. “Experimental Evaluation of Auditory Display and Sonifcation of Textured Images,” ____
McGee, Ryan. “Auditory Displays and Sonification: Introduction and Overview,” The University of California, Santa Barbar (2009).
Ouaknine, Joel and James Worrell. “Revisiting Digitalization, Robustness, and Decidability for Timed Automata,” ____
Polli, Andrea. “”Atmospherics/Weather Works”: A Spatialized Meteorological Data Sonification Project,” Leonardo 38, No. 1 (2005): 31–36.
____. Sonification: A Brief Study in Music ____
Sturm, Bob L. “Pulse of an Ocean: Sonification of Ocean Buoy Data,” Leonardy 38, No. 2 (2005): 143–149.
Thibault, Bill and Scot Gresham-Lancaster. “Experiences in Digital Terrain: Using Digital Elevation Models for Music and Interactive Multimedia,” Leonardo Music Journal 7 (1997): 11–15.
Walker, Bruce N. and Michael A. Nees. “Theory of Sonification,” Principles of Sonification: An Introduction to Auditory Display and Sonification ____
Weinberg, Gil and Travis Thatcher. “Interactive Sonification: Aesthetics, Functionality and Performance,” Leonardo Music Journal 16 (2006): 9–12.
Wu, Hsin-Fu. “Spectral Analysis and Sonification of Simulation Data Generated in a Frequency Domain Experiment,” Thesis at Naval Postgraduate School, Monte rey, California, 2002.