What are the most difficult software or programming related problems in finance?

The hardest problems I have come across are the following:

Concurrency -- when you are executing trades, getting into a deadlock or a runaway loop can be a massive problem, as we have seen in the example of Knight Trading.

Using your data center capacity fully to crunch as much data as possible.

Building a framework that can be used for research and production without having to rewrite code.

Building the right front-end tools for the operational team watching strategies to do so at scale.

How do High-Frequency Trading firms compete with each other?

The truly high frequency strategies are such that they look only at microstructure, and are agnostic to which market it is or any statistical prediction of that market.  If you see that you are edged out of individual trades when the microstructure shifts, you can usually conclude that there are other HFTs in play.

How is machine learning used in finance?

The finance industry is broad and different segments have different use cases for machine learning.  In general, there is less machine learning in finance than outsiders may imagine.  When I first joined the industry, while the term "machine learning" was not too common, I had anticipated that there would be a ton of very complex methods used to manipulate data and make predictions.  In fact, most business problems in finance have straightforward solutions that do not require those methods, which was surprising to me at the time.

With that said, here are some examples of machine learning in the trading space:

＃Signal generation and testing -- I prefer the term "signal" to "model" because ultimately the goal of trading is to generate trades based on some signal, not to "model the world" which is too lofty and far from what trading actually achieves.  Machine learning methods can be very useful in generating signals from past data, and in particular, approaches that have been refined the in machine learning domain around validation and testing for statistical significance are very critical.  Because financial prices have only one history, getting the significance part wrong is often the difference between success and failure.  The actual prediction can be relatively simple supervised learning using linear regression, logistic regression, or boosting.

＃Feature engineering -- unsupervised learning isn't useful for trading directly because you want / need to have a feedback look that ties profits to signals.  That said, individual features can be designed using whatever tools you want.  Bringing it external data sets like satellite data for example, or economist models for macro trading, etc., and applying clustering or more complex tools to then feed those results into a (relatively simple) trading strategy is often how  good strategies work.

＃Borrowing from speech recognition methods -- in both cases, you want to predict the future from the past.  Approaches from NLP and related areas of machine learning have been particularly helpful to the pioneers of the quantitative trading space like Renaissance Technologies.

Some areas where machine learning isn't used as much are the execution part of high-frequency trading, risk management, option pricing, and portfolio strategy.  To conclude, machine learning has its place in finance, but less than people think, and even the parts that use it rely more on the approach of modern machine learning than on particular models that are common in academia.

How did Vladimir Novakovski learn Machine Learning?

I was learning machine learning out of necessity, because I wanted to find better signals to predict markets and knew my competitors were using those methods, not because it was a buzz word or because I took a certain class.  In particular, the path I followed was starting with the foundations -- see my answer to How can I develop a deep/unified view of statistical mechanics, information theory and machine learning? -- and then reading the Elements of Statistical Learning book, all the while applying what I am learning to the actual data I was looking at.

How can I develop a deep/unified view of statistical mechanics, information theory and machine learning?

A couple of years ago, I spent some time researching the first principles that machine learning is derived from.  In addition to the literature mentioned in the other great answers to this, two seminal papers that (in conjunction) provide a link between physics and information theory are as follows:

A Mathematical Theory of Communication by C. E. Shannon (1948)

Stochastic Problems in Physics and Astronomy by S. Chandrasekhar (1943)