Connectionist symbol processing: any progress? (fwd)

Rejected message: sent to koglist(a)cogpsyphy.hu by ERDI(a)SUNSERV.KFKI.HU follows. Reason for rejection: sender not subscribed. ------------------------------------------------------------------------------- ---------- Forwarded message ---------- Date: Tue, 11 Aug 1998 03:34:27 -0400 From: Dave_Touretzky(a)cs.cmu.edu To: connectionists(a)cs.cmu.edu Subject: Connectionist symbol processing: any progress? I'd like to start a debate on the current state of connectionist symbol processing? Is it dead? Or does progress continue? A few years ago I contributed a short article on "Connectionist and symbolic representations" to Michael Arbib's Handbook of Brain Theory and Neural Networks (MIT Press, 1995). In that article I explained concepts such as coarse coding, distributed representations, and RAAMs, and how people had managed to do elementary kinds of symbol processing tasks in this framework. The problem, though, was that we did not have good techniques for dealing with structured information in distributed form, or for doing tasks that require variable binding. While it is possible to do these things with a connectionist network, the result is a complex kludge that, at best, sort of works for small problems, but offers no distinct advantages over a purely symbolic implementation. The cases where people had shown interesting generalization behavior in connectionist nets involved simple vector-based representations, without nested structures or variable binding. People had gotten some interesting effects with localist networks, by doing spreading activation and a simple form of constraint satisfaction. A good example is the spreading activation models of word disambiguation developed in the 1980s by Jordan Pollack and Dave Walts, and by Gary Cottrell. But the heuristic reasoning enabled by spreading activation models is extremely limited. This approach does not create new structure on the fly, or deal with structured representations or variable binding. Those localist networks that did attempt to implement variable binding did so in a discrete, symbolic way that did not advance the parallel constraint satisfaction/heuristic reasoning agenda of earlier spreading activation research. So I concluded that connectionist symbol processing had reached a plateau, and further progress would have to await some revolutionary new insight about representations. The last really significant work in the area was, in my opinion, Tony Plate's holographic reduced representations, which offered a glimpse of how structured information might be plausibly manipulated in distributed form. (Tony received an IJCAI-91 best paper award for this work. For some reason, the journal version did not appear until 1995.) But further incremental progress did not seem possible. People still do cognitive modeling using connectionist networks. And there is some good work out there. One of my favorite examples is David Plaut's use of attractor neural networks to model deep and surface dyslexia -- an area pioneered by Geoff Hinton and Tim Shallice. But like most connectionist cognitive models, it relies on a simple feature vector representation. The problems of structured representations and variable binding have remained unsolved. No one is trying to build distributed connectionist reasoning systems any more, like the connectionist production system I built with Geoff Hinton, or Mark Derthick's microKLONE. Today, Michael Arbib is working on the second edition of his handbook, and I've been asked to update my article on connectionist symbol processing. Is it time to write an obituary for a research path that expired because the problems were too hard for the tools available? Or are there important new developments to report? I'd love to hear some good news. -- Dave Touretzky References: Arbib, M. A. (ed) (1995) Handbook of Brain Theory and Neural Networks. Cambridge, MA: MIT Press. Plaut, D. C., McClelland, J. L., Seidenberg, M. S., and Patterson, K. (1996) Understandig normal and impaired word reading: computational principles in quasi-regular domains. Psychological Review, 103(1):56-115. Plate, T. A. (1995) Holographic reduced representations. IEEE Transactions on Neural Networks, 6(3):623. Touretzky, D. S. and Hinton, G. E. (1988) A distributed connectionist production system. Cognitive Science, vol. 12, number 3, pp. 423-466. Touretzky, D. S. (1995) Connectionist and symbolic representations. In M. A. Arbib (ed.), Handbook of Brain Theory and Neural Networks, pp. 243-247. MIT Press.