Course Syllabus

SLS 750 Seminar in Second Language Acquisition -- Multimedia Analysis

Fall 2013 | CRN: 75575

Dongping Zheng, Ph.D. 

zhengd@hawaii.edu | Twitter: @zhengdo | 635.0279 (in case of emergency)

Class meets at Moore Hall 155B | Monday and Wednesday 1:00-2:15

Office Hours: 2:30-4:30 Mondays | Moore Hall 555

Description:

In this course, we will explore new ways of looking at communication and interaction (e.g., student-teacher, human-computer, doctor-patient interactions, and other professorial and mundane interaction) by using an array of multimodal analytic toolkits. A common thread in these approaches is illuminated by Edward Hutchins’ seminal work on “Cognition in the Wild”, which considers material artifacts as part and parcel of human cognition and communication. Rather than treating them as decorations or backgrounds of communication, material artifacts or external representations augment our thinking and communication, extending cognition beyond the skull.  “They allow us to think the previously unthinkable” (Kirsh 2010). For example, a particular choice of color, in combination with other features, indexes a particular evaluative language stance; A particular gesture or body movement signals a pattern of meaning-making and sense-making along and/or in combination with language; A particular type of technology (such as Facebook social media, Youtube video, or Second Life virtual world) invites different trajectories of interaction and meaning-making practices.

Backgrounded on these perspectives, we will look at multimodality from different aspects of literature; for example, Baldry and Thibault’s multimodal transcription and text analysis, Charles Goodwin’s embodied interaction, Carey Jewitt’s multimodal approach to technology, literacy and learning. We will explore software packages that have been used for multimodal transcription and analysis, such as CHILDES (open source), Mutlimodal Web Analyzer (open source), ELAN (Open source,http://tla.mpi.nl/tools/tla-tools/elan/) and Transana (http://www.transana.org/, free to SLS 750 students), Discursis (Purchase, http://www.discursis.com/)

Who should take this course?

• Students from the College of Languages, Linguistics and Literature, the College of Education, the Department of Information and Computer Sciences, College of Business, School of Medicine, etc.
• Students who are interested in exploring interaction patterns from video and audio data and other textual data.
• Students who are interested in web technologies, and curious how social medias provide new learning opportunities that are extended by multimodal analysis.
• Students who are interested in material development and instructional design, such as designing courses within Laulima or using other course management and delivery systems.
• Professionals who are interested in investigating how understanding of interaction processes  can help with any learning and training situations.

Learning Outcomes:

• Can perform multimodal analysis of any given text genre (print, hypertext, audio, video).
• Can engage in intellectual discussion of how people communicate in multimodal ways with gestures, and body movements.
• Can skillfully use one of the transcription and analytic tools.
• Can design for meaningful interactions situated in different contexts: such as classrooms, computer assisted learning environments, online virtual environments.

Note: Students who would like to take this seminar for SLS730 credit can do a project that focus on teaching.

Textbook and materials:

Required TextBook:

Baldry, A., and P. J. Thibault. 2005. Multimodal transcription and text analysis: A

multimedia toolkit and coursebook. London: Equinox Publishing Ltd.  (MTTA hereafter). Available at Amazon

Other required journal articles and book chapters can be accessed from Laulima.

Recommended Books:

Coiro, J., M. Knobel, C. Lankshear, and D. J. Le, eds. 2008. Handbook of Research on New Literacies. Philadelphia, PA: Lawrence Erlbaum.

Bednarek, M., and Martin, J. R. (eds) (2010). New Discourse on Language: Functional Perspectives on Multimodality, Identity, and Affiliation. London and New York: Continuum. Available at UHM ebrary, http://uhmanoa.lib.hawaii.edu:7008/vwebv/holdingsInfo?searchId=1169&recCount=10&recPointer=2&bibId=3240808#

Jewitt, C. (2006). Technology, Literacy and Learning: A Multimodal Approach, London:Routledge.

Unsworth, L. (2008). Multimodal semiotics: functional analysis in contexts of education. London ; New York: Continuum.

Readings and Activities 

Week 1

*Jewitt, C. (2006). Technology, literacy and learning: A multimodal approach. London: Routledge.  Chapter 1, Introduction.

*Street, B. Pahl, K. and Rowsell. J. (2009) Multimodality and New Literacy Studies.In

Jewitt, C. ed. The Routledge Handbook of Multimodal Analysis.  London: Routledge. Pp191-200.

Jewitt, C. (2009). An Introduction to Multimodality. in C. Jewitt (ed), The Routledge Handbook of Multimdal Analysis. London and New York: Routledge, pp. 14-27.

*Unsworth, L. and Cleirigh, C. (2009). Multimodality and reading: The construction of meaning through image-text interaction. In Jewitt, C. ed. The Routledge Handbook of Multimodal Analysis. London: Routledge. Pp151-163.

Week 2

*Hodges, B (2009). Ecological Linguistics: Values, dialogical arrays, complexity, and caring. Pragmatics & Cognition 17:3 (2009), 628–652.

Steffensen, S. V. (2012). Care and conversing in dialogical systems. Language Sciences.

Recommended

Goodwin, C. (2007). Participation, Stance, and Affect in the Organization of Activities. Discourse and Society, 18(1). pp. 53-73.

Stuart, S. (2010). Enkinaesthesia, biosemiotics and the ethiosphere. In S.J. Cowley, J.C. Major, S.V. Steffensen & A. Dinis, (Eds), Signifying bodies: Biosemiosis,

interaction and health (305–330). Braga: Portuguese Catholic University Press.

Week 3

*Hutchins, E. (1995). Cognition in the wild. Cambridge, MA: MIT Press. Chap 7: Learning in Context.

*Magnani, L. (2006). Multimodal abduction: External semiotic anchors and hybrid representations Logic Journal of the IGPL14 (2):107-136doi:10.1093/jigpal/jzk009. 

Cowley, S. J. (2011). Distributed language. In S. J. Cowley (Ed.), Disitributed language (pp 1-14). Amsterdam: John Benjamins. Retrieved August 26, from, http://www.academia.edu/1977912/Distributed_Language

Recommended

*Cowley, S. (2007). The cognitive dynamics of distributed language. Language Sciences, 29, 575-583.

*Cowley, S. (2009). Distributed language and dynamics. Pragmatics & Cognition, 17(3),495-507.

&Kravchenko, A. (2007). Essential properties of language, or why language is not a code. Language Sciences, 29, 650-671.

Week 4

Linell, P. (2009). Rethinking language, mind, and world dialogically: Interactional and contextual theories of human sense-making. Information Age Publishing, INC. Charlotte, NC.

*Chapter 5 — Dialog and the Other

*Chapter 13 — Rethinking language in dynamic terms.

Linell, P. (2000) What is dialogism? Aspects and elements of a dialogical approach to language, communication and cognition. Lecture first presented at Växjö University, October 2000. This version: 2003-02-26

Recommended

*Cowley, S. & Zheng, D. (2011). The turning of the tide: Rethinking language, mind and world  [Review article of Linell, P. (2009), Rethinking language, mind, and world dialogically: Interactional and contextual theories of human sense-making]. Journal of Multicultural Discourses, 6(02), 197-210.

Week 5, 

MTTA Chap 1, Introduction: multimodal texts and genres.

*O‟Halloran, K. L. (2011). Multimodal Discourse Analysis. In K. Hyland and B. Paltridge (eds) Companion to Discourse. London and New York: Continuum.

Week 6

MTTA Chap 1, The Printed Page

Week 7

MTTA Chap 3, The Web page.

Week 8

MTTA Chap 4, Film/video texts and genres

Week 9

*Goodwin, C. (2013). The co-operative, transformative organization of human action and knowledge. Journal of Pragmatics. 46. pp. 8-23. http://dx.doi.org/10.1016/j.pragma.2012.09.003

*Streeck, J. (2013). Interaction and the living body. Journal of Pragmatics. 46(69-90)

*Piirainen-Marsh, A. & Tainio, L. (2009). Collaborative Game-play as a Site for Participation and Situated Learning of a Second Language, Scandinavian Journal of Educational Research, 53: 2, 167 — 183.

Stuart, S.A.J. 2010. Enkinaesthesia, biosemiotics and the ethiosphere, in Signifying bodies: Biosemiosis, Interaction and Health, ed. S.J. Cowley, J.C. Major, S.V. Steffensen, & A. Dinis, 305-30. Braga, Portugal: The Faculty of Philosophy of Braga, Portuguese Catholic University.

Week 10

*Steffensen, S. (2013). Human interactivity: Problem-finding, problem-solving, and verbal patterns in the wild.

*Zheng, D, Cowley, S. J., and Hu, Y. (in review). Learning from abduction: the power of open-ended learning environments (coming soon)

Week 11

*Norris, S. (2006) Multiparty interactions: A multimodal perspective on relevance. Discourse Studies, 8 (3), pp. 401-42.

*Atkinson, D. (2011). A Sociocognitive approach to second language acquisition: How mind, body, and world work together in learning additional languages. In Atkinson, D. (Ed), Alternative approaches to second language acquisition. London: Routledge.

Jones, R. (2010) Creativity and discourse. World Englishes. 29 (4): 467-480.

Week 12

Angus D, Watson B, Smith A, Gallois C, Wiles J (2012) Visualising Conversation Structure across Time: Insights into Effective Doctor-Patient Consultations. PLoS ONE 7(6): e38014. doi:10.1371/journal.pone.0038014

*Angus, D.; Smith, A.; Wiles, J.; , "Conceptual Recurrence Plots: Revealing Patterns in Human Discourse," Visualization and Computer Graphics, IEEE Transactions on , vol.18, no.6, pp.988-997, June 2012. doi: 10.1109/TVCG.2011.100

* readings are available in Laulima

** Any additions or changes will be colored in green