ML vs. probabilistic programming



  • This is more of a speculative, thought-experiment post, but I'd love to see us nerds talk about it.

    I've used a lot of machine learning over the last couple of years. What I've noticed is that most ML algorithms are black boxes - pop in training data, test it on some samples, and you get predictions. But you have no idea what's going on inside. More importantly, you also have no way of introducing a statistical bias, which can actually be quite important sometimes.

    Which is where probabilistic programming comes in - taking Bayes' theorem to its logical conclusions, and actually applying Bayesian methods to create better models. So for instance, let's say I'm building a model that is trying to analyze two trading strategies, and figure out which has a better chance of "beating" the market. That's where Bayesian methods shine; it implies that you have prior beliefs about your data, and you incorporate those into your model, and update them as and when the data indicates they should be updated. I tried this with wargaming, and it did wonders.

    I'm wondering if others have used Bayesian methods, and what your experiences have been?



  • This post is deleted!

    ga___jit

    0


  • Well, wargaming is just one area you can use it for. But yes, the idea is to run probabilistic models as an open inference engine.

    I'd like to try neural nets, but I do need more computing power for that. Like a GPU-powered cloud instance.



  • This post is deleted!

    ga___jit

    1


  • AWS now offers Linux-based GPU instances: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using_cluster_computing.html

    In my experience, training DNNs on GPUs is as much as 30x faster than on CPUs.

    Probabilistic models are used a lot in practice: medical diagnostics for example. For algorithmic trading though, I don't see any benefit in using PGMs over ML techniques. Have seen decent results with SVMs, Random Forest etc.



  • This post is deleted!

    ga___jit

    0


  • Neural Networks are fundamentally Bayesian in spirit. They adapt to training data.

    @rudimk You can try NNs online on your browser. ConvNet has a fantastic online NN implementations: http://cs.stanford.edu/people/karpathy/convnetjs/index.html

    BTW, GPUs do speed up tremendously. Not just a factor of 10 or 100x, sometimes they scale 1000x. For example, the following photo was generated by a NN running on my laptop overnight while on GPU it takes 2 minutes (yes, you read it right)

    NN generated image

    Did you know notCRUD has cool signatures? Use them by going to your User Profile > Edit.

    Personal blog: paraschopra.com/blog/

    0


  • This post is deleted!

    ga___jit

    0


  • I am really not sure what you mean by "ML algorithms are black boxes". I think that 'black box' is a fairly new terminology that people have started using for ML Algorithms that are way too complex to understand. What most of the people want is to give the input and get the result.
    Probabilistic programming is a programming paradigm specific to Bayesian Statistics. This is a fairly new area and I haven't done any work on it. But from what I know this areas is more about reducing the complexity of Bayesian programs.
    I don't think probabilistic programming can replace ML or even be as accurate as ML but it can help in the field of "Bayesian style Machine Learning". Just to give an example, introduce the concept of Time in the above example of "beating the market" and see which one out of ML & Probabilistic Programming shines.



  • This post is deleted!

    ga___jit

    0


  • This article is very pertinent to the thread. The author is putting forward a tantalising hypothesis that:

    deep learning studies a connection between optimization and functional programming.

    Did you know notCRUD has cool signatures? Use them by going to your User Profile > Edit.

    Personal blog: paraschopra.com/blog/

    0


  • That was pretty amazing read! Especially the correlation between Representations in NN and Types. Also simplicity in relating neuron to function was pretty cool!

    Another very good thing hidden in that article was this paper on "Show and Tell: A Neural Image Caption Generator" http://arxiv.org/pdf/1411.4555.pdf

    Lot of cool things happening in Parallel Universe !


Log in to reply