Feb 13, 2016 [change history]

It has been a while since I've posted anything. Wow, has it really been five years... that's a very long while. When I last posted I was mostly focused on languages and compilers. I'm still really interested in these things. I have some ideas that I'd love to come back to some day, but for the moment my attention has shifted to machine learning and in particular deep learning which has seen some really impressive advances in the last five years or so. As a part of this I took the popular Coursera Machine Learning course. It's popular for good reason. Professor Ng is a leader in the field but more importantly he has a knack for explaining complicated concepts in a way that emphasizes building intuition. It's also clear that he is sincerely invested in the success of his students, so it's a course I'd definitely recommend it for anyone who wants to either learn or review the essentials of Machine Learning. While deep learning is not covered in this course it does prepare you to tackle deep learning material on your own (see bottom of post for some references). One warning though: if you're fairly advanced in mathematics and hoping for proofs and a detailed technical treatment of the underlying mathematical mechanisms you're not going to find that in this course.

I haven't gotten very far with TensorFlow yet and so I just want to write down some of my experience with the install in case it's useful to anyone out there. The system requirements for TensorFlow are really limited (it definitely shows that this was an internally deployed product with very specific hardware requirements). Only mac and a specific version of Ubuntu are supported platforms. If you want GPU acceleration Ubuntu is your only choice. On a positive note the documentation is really useful and fairly complete from what I've seen so far. For reasons that aren't important I was limited to installing on OpenSUSE and 13.2 is the most recent version that NVidia supports the required version of CUDA on. Checking out and building TensorFlow on OpenSUSE *almost* works as described in the TensorFlow docs even though it isn't a supported configuration. The one exception I ran into is that you have to update the bazel CROSSTOOL files as explained here: http://stackoverflow.com/q/35256110/160015

I built TensorFlow with, and without GPU support in order to get an idea for what kind of performance difference the GPU could make. I'm developing on a Lenovo IdeaCentre K450 which has an Intel Core i7 4770 3.4 GHz processor and an NVIDIA GeForce GTX 650 graphics card. This card has a CUDA Compute Capability of 3.0 which is not officially supported by TensorFlow, but fortunately they provide a non-standard build feature that allows you to use a 3.0 card. As documented on the TensorFlow site you need to build using:

In order to check the performance increase (I won't call this a benchmark since this check is in no way rigorous to the level a benchmark should be) that you can get from enabling GPU support I ran:

then with GPU acceleration enabled:

I am really happy with the GPU speed up. TensorFlow took 24m21.986s to fit the model just using the CPU (with all 8 virtual cores holding steady at about 75% utilization throughout) and 8m22.237s using the GPU for an almost 3x speedup. OK, so that's where I'm at for now. I plan on spending more time getting familiar with how to solve real problems using TensorFlow and expect to be posting on that experience pretty soon. Hopefully that will be more interesting than this post.

- Coursera Machine Learning course
- Deep Learning: this book is a great resource. I haven't worked my way through the whole thing but I've already gotten a lot out of the chapters that I have read. Draft versions of the book are freely accessible for the moment but I'll probably buy a final version when it's published, in part to support the authors' efforts.

The following links look promising but I haven't yet had a chance to dig into these:

- Stanford CS231n: Convolutional Neural Networks for Visual Recognition: the github page for the course seems worth looking into even if you don't plan on watching the lectures
- Stanford Deep Learning Tutorial
- Neural Networks and Deep Learning: free online book
- Understanding Neural Networks Through Deep Visualization
- Neural Network Data Normalization and Encoding