Torch 和 Theano  fastML

Recently we took a look at Torch 7 and found its data ingestion facilities less than impressive. Torch’s biggest competitor seems to beTheano, a popular deep-learning framework for Python.

It seems that these two have been having “who is faster” competition going for a few years now. It’s been documented in the following papers:

J. Bergstra, O. Breuleux, F. Bastien, P. Lamblin, R. Pascanu, G. Desjardins, J. Turian, Y. Bengio - Theano: a CPU and GPU Math Expression CompilerPDF

Ronan Collobert, Koray Kavukcuoglu, Clement Farabet - Torch7: A Matlab-like Environment for Machine LearningPDF

Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian Goodfellow, Arnaud Bergeron, Nicolas Bouchard, David Warde-Farley, Yoshua Bengio - Theano: new features and speed improvements arxiv

A figure from the Torch7 paper [2]. Torch - red, Theano - green. Higher is better.

And a quote from [3]:

Bergstra et al.(2010) showed that Theano was faster than many other tools available at the time, including Torch5. The following year, Collobert et al.(2011) showed that Torch7 was faster than Theano on the same benchmarks.

The results in the last paper are mixed, if you’re wondering.

The latest act in this friendly competition, which can be seen as one between Bengio’s and LeCun’s groups, appears to be about FFT convolutions, first available in Theanoand recently open-sourced by Facebookin Torch.

As a side note, the press really jumped at this second event with headlines about turbo-charging deep learningand the like. Probably the allure ofFacebookanddeep learningin the same sentence.

Let’s look at convnet benchmarks by Soumith Chintala. He is a Facebook/Torch guy and yet the Theano’s convolution layer is reported to be the fastest at the time of writing. Waiting for thosefbfftresults.

Anyway, speed isn’t everything and there’s more to life than FFT convolutions. From a developer’s perspective minor differences in speed are less important than other factors, like ease of use. Which leads us to what Soumith had to say about Torch, according to VentureBeat:

It’s like building some kind of electronic contraption or, like, a Lego set. You just can plug in and plug out all these blocks that have different dynamics and that have complex algorithms within them.

At the same time Torch is actually not extremely difficult to learn — unlike, say, the Theano library.

We’ve made it incredibly easy to use. We introduce someone to Torch, and they start churning out research really fast.

Well, you already know our opinion about the “incredibly easy” bit. Torch is not really a Matlab-like environment. Matlab, with all its shortcomings, is a very well polished piece of software with examplary documentation. Torch, on the other hand, is rather rough around the edges.

Besides the language gap, that’s one of the reasons that you don’t see that much Torch usage apart from Facebook and DeepMind. At the same time libraries using Theano have been springing up like mushrooms after a rain (you might want to take a look at Sander Dieleman’s Lasagne and at blocks). It is hard to beat the familiar and rich Python ecosystem.

Theano tutorials

The officialtutorial

Alec Radford’s talk and corresponding code

Colin Raffel’s tutorial notebook

The Portrait of a Machine Learning Priestess

Best framework for Deep Neural Netsthread at Reddit

P.S. What about Caffe?

Caffe is a fine and very popular piece of software. How does it compare with Torch and Theano? Here’ssieisteinmodel’s answer from Reddit:

Caffe has a pretty different target. More mass market, for people who want to use deep learning for applications. Torch and Theano are more tailored towards people who want to use it for research on DL itself.

Posted byZygmunt Z.2015-02-09 neural-networks,software