Course Information

Semester: 2024-2 (113-2)
Curriculum No.: GIBMS 7015 (454 M0390)
Credit: 3
Time: Fridays 9:10 - 12:10
Venue: College of Medicine RM508 (register here for NTU Eduroam access for Future Classroom Wifi)
Instructor(s): Joshua Goh
Language: English

Objectives

Learn the basic principles of how neural network models work.
Make one's own simple neural networks.
Learn how to evaluate neural network models.

Description

This course will introduce first principles of neural networks in relation to how it implements human cognition and it taught using the approach of applied practical programming of simple neural networks. Students will read three real modeling papers and apply the neural network models in these papers to create their own neural networks, in addition to regular class assignments. You will learn by doing. Four examples of network implementation will be covered:

Basic Perceptron
Hopfield Attractor Network (Hopfield, 1982)
Error Backpropagation (Multi-Layered Perceptron; Rumelhart et al., 1986)
Self Organization via Unsupervised Learning (Von Der Malsburg, 1973)

Note that the approach in this course emphasizes ground-up basic theory about neural network principles. This course will briefly consider but will not significantly engage with current artificial intelligence tools (e.g. Large Language Models) as in many contemporary Engineering applications. However, students are free to use these intelligence tools to append their understanding or in application in assignments as is creative.

Pre-Requisites

Students in Graduate Institute of Brain and Mind Sciences (others, case-by-case).
Your own computer with coding environment (below) ready to go.
Confidence in computer programming is not a requirement but is a plus.

Coding Environments

Either of the environments below are highly encouraged. The course will primarily use Python in class and some R.

Python in Jupyter Lab
R in R Studio
Matlab (see National Taiwan University campus-wide license, scroll to the bottom of the page)

Grading

Assignments
- Homework tutorial exercises.
- Three main assignments: Program the above three neural networks and apply them to real-life problems or simulations of human cognition.
Rubric
- Based on quality of assignments
  - Model success
  - Comprehensiveness and strategicness of model evaluation.
- Homeworks afford bonus credits.

Schedule

Week	Date	Topic	Resource
1	2025/02/21	Introduction: Biology, why model, and general approach. [video 1][video 2]	Biological neural networks Models and simulations
2	2025/02/28	Memorial Day (no class)	Do spend the time to brush up on linear algebra if not confident (see Jordan, 1986), and importantly, start reading Aggarwal, 2018, Ch. 1.
3	2025/03/07	Linear algebra: Vectors, matrices, and matrix operations. [video](video has bug, main screen not recorded, follow along with Jupyter Notebook)	Jordan, 1986 Linear Algebra Colab Jupyter Notebook Blackboard HW1 (due 2025/3/21)
4	2025/03/14	Perceptrons: Nomenclature, general neural network framework, application in logic problems [video]	Aggarwal, 2018, Ch. 1 Perceptron Colab Jupyter Notebook Blackboard HW2
5	2025/03/21	Attractor networks 1: Introduction to the principles of autoencoding and memory. [video]	Hopfield, 1982 Blackboard
6	2025/03/28	Attractor networks 2: Simple autoencoder architecture and learning rule to instantiate content-addressable memory, attractor properties. [video]	Binary Autoencoder Colab Jupyter Notebook
7	2025/04/04	Childrens Day, Tomb Sweeping Day (no class)
8	2025/04/11	Attractor networks 3: Evaluating and describing the autoencoder model. [video]	Assignment 1 Josh's Hopfield Network Gitlab
9	2025/04/18	Backpropagation 1: Introduction to the principles of multi-layered perceptrons and error-based learning. [video]	Rumelhart et al., 1986 Blackboard
10	2025/04/25	Backpropagation 2: Simple multi-layered perceptron to instantiate error-based learning, non-linear input-output mappings. [video]	Multi-Layer-Perceptron Colab Jupyter Notebook Blackboard
11	2025/05/02	Backpropagation 3: Evaluating and describing the multi-layered perceptron model. [video]	Assignment 2
12	2025/05/09	Unsupervised learning 1: Introduction to the principles of functional self-organization and convolution in V1 orientation selectivity. [video]	von der Malsburg, 1973 Hubel & Wiesel, 1963 Hubel & Wiesel, 1974 Bosking et al., 1997 Blackboard
13	2025/05/16	Unsupervised learning 2: Unpacking the VDM model. [video]	Self-Organizing VDM Network Colab Jupyter Notebook
14	2025/05/23	Unsupervised Learning 3: Evaluating the VDM model. [video]	Assignment 3
15	2025/05/30	Dragon Boat Festival (no class)
16	2025/06/06	Recurrent Neural Networks, Convolutional Neural Networks, Deep Neural Networks, Exploding and Diminishing Gradients, Overfitting, Regularization, Large Language Models. [video]	Aggarwal, 2018, Ch. 2 Aggarwal, 2018, Ch. 3

Bibliography

Jordan M. I. (1986). An introduction to linear algebra in parallel distributed processing. In Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations. Ed. David E. Rumelhart & James L. McLelland, p365–422, MIT Press, Cambridge: USA.
Neural Networks and Deep Learning: A Textbook. (2018). Charu C. Aggarwal, Springer, Cham, Switzerland. Ch. 1.
Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences USA, 79, 2554-2558.
Rumelhart, D., Hinton, G., & Williams, R. (1986). Learning internal representations by error propagation. MIT Press Cambridge, MA, USA.
Von Der Malsburg, C. (1973). Self-organization of orientation sensitive cells in the striate cortex. Kybernetik, 14, 85-100.
Goh, J. O. S. (2007). Hopfield Network. Unpublished. https://gitlab.com/joshuagoh/hopfield-network
Goh, J. O. S. (2007). Backpropagation in a non-linear layered network: learning from past mistakes. Unpublished. https://gitlab.com/joshuagoh/multi-layer-perceptron
Goh, J. O. S. (2007). Extensions of the von der Malsburg self-organized visual cortex model. Unpublished. https://gitlab.com/joshuagoh/unsupervised-learning.

Other Resources