Diarrhea of the Brain
I finished Anathem earlier. It is a good, if dense, book. I think the first 2/3 of the book probably could have been reduced to about 1/2 of its size, but it was still interesting and a good book. Entirely unrelated to the subject of the book, it got me thinking about neural networks.
I really, really like neural networks. Like, unreasonably so. One thing I've had bubbling in my brain is how to minimally characterize the information held in neural networks. Weights for NNs are represented with floating-point numbers, usually. I think that you could do it with integers ranging from 0 to (Maximum number of neurons connected to a neuron) on a local scale. (double edit: strikethru in wrong place) (Conversely, if this turns out to be true, that means that floating-point numbers could not correctly represent neurons wherein more than ~1000 neurons connect into a single, very important neuron.) This would be a vast improvement in terms of storage (especially since this could be done locally rather than globally), but it makes learning algorithms harder (Edit: Also, it changes how the iteration function of the NN would work, very slightly.). They'd have to operate on probabilistic weight-changes rather than on progressive weight-changes, which means we don't have guaranteed convergence. But that's okay, I'm pretty sure we lost than when we moved away from perceptrons (Even with backpropogation, I don't think anybody has proven convergence. People just really, really think that NNs will eventually converge.).
One cool thing would be the ability to estimate the amount of time that it would take to train a NN (via backpropogation?). I'm sure someone has done it. It seems like a problem that would take about O(c1^n), where n is the size of the NN and c1 is a constant relating to the training time. If it is anything like this, or even if it's just O(n^2), it would be really beneficial to break any NN system into multiple parts, with arbitrary inputs and outputs which would eventually be connected to a wider-but-not-necessarily-deeper neural network.
But anyhow, that leaves me with a proof I'm not quite sure how to do (How do I induct over the number of neurons given that there are about n^2 ways to arrange a new neuron, not even counting dealing with the weights?). I've not toyed with it using paper or for more than an hour, but it feels right. So I'll do a real proof before implementing it. (edit: came up with proof on the way back from last final of the day.)
I'm really just killing time here before the test. I can't really concentrate enough to study right now. I'm sure that my test-taking instincts will kick in when it's test time. 5 minutes left, and the tension is pretty strong here.
Closing thoughts: How many individual objects do you think the human brain can recognize? Again, I'm sure studies have been done about this. I'm not sure if "James's couch" is separate from "couch" with a strong-ish connection to "James" and other notable couches, but there you go.