Taking too long? Close loading screen.
News

Python, Pine Scripts, And Pro Trading Bots

Good algorithmic trading bots are notoriously difficult to make functional and profitable. Pinebot seeks to alleviate this difficulty, making it easy for anyone to trade Bitcoin and other cryptocurrency online, algorithmically. Pinebot is still a beta project. It’s a premium beta project but still in a beta condition nonetheless. As we’ve had some stability issues lately, let’s take a walk down memory lane and then give an overview of the near-future roadmap.

Pinebot Beta Challenges

Before there was Pinebot, there were many Pine scripts being coded by our admin. At first, it wasn’t evident that Pine scripts could perform well enough to even be worth automating. Repaints were an issue and it wasn’t clear if they could be eliminated. Once we came up with an idea for how to mitigate that, it was then worth considering automation. How could this be done? At the time, the most popular bot for Pine scripts was Autoview. Autoview is a browser extension that monitors for TradingView alerts in order to get long/short signals from Pine scripts. This was the least desirable solution because it means that you can’t use the same code in your bot that you use in your backtests. Alert scripts are not the same as strategy scripts. If not an alert, how else?

There are two other ways to get a strategy’s trade positions from TradingView. One is to open a socket connection to TradingView, just as the browser does when it’s viewing the TradingView Strategy Tester. The other is to run a headless browser and just scrape the data from the Strategy Tester. The socket method was attempted but TradingView employs an encoded binary format. After several failed attempts, it was decided that sockets were out of the question. That left a headless browser. For this, we would use Python & Selenium. We programmed a proof-of-concept in Python, which worked. We were in business. Pinebot had a chance and thus was born. All of our problems weren’t yet solved but we had a way forward. The two big issues from then on were:

  • How to deal with strategy repaints.
  • Scaling the headless Python scraper for many strategies & users.

Repaints

Dealing with repaints was a huge annoyance. The first approach was to say “As long as the strategy gracefully repaints, we can program a repaint handler to deal with it.” By “Gracefully repaints” we mean that: If upon using the Replay tool or just loading the chart and letting the market play out the backtest still stays within a standard deviation of the original backtest, we will consider it gracefully repainting and employ a repaint protector.

For our Repaint Protector, we would give some options to the user. You could either honor a repainted position change or ignore it. This of course has some consequences. Namely, ignoring it can lead to the bot going out of sync with the strategy. Honoring it can lead to big exchange fees as a position flickers between existing and not. Our first default solution was to honor it but to wait for X amount of time to give the strategy a chance to settle down and stop repainting. This was imperfect and indeed did lead to exaggerated exchange fees. We then changed it to ignore repaints by default. At this point, we were inundated with support requests because the bot instances kept going out of sync with the strategy.

It took a while but we finally came to a better solution. We hit the drawing board to figure out what makes a Pine script repaint. We would then create a new class of strategies (At least in TensorTom’s Stratega shop) that simply did not repaint. This was difficult but eventually, there was a eureka moment and we solved it. The Compendium strategies were born. Not only that, we had a roadmap for converting all previous Stratega strategies to the no-repaint model. Not all previous Stratega repainted but some did gracefully repaint. Now we could eliminate that. As this was now a solved problem, we had one more to contend with.

Scaling The Bot

I’m a good programmer and I love engineering. I love good engineering. Bad engineering is the bane of my existence. What’s one of the easiest things to engineer badly? Enter scaling a headless web browser. First, let me explain something about Selenium on Python. Selenium is basically a Chrome (Or Firefox, if you choose) web browser. It’s used to automate any task that can be done in a web browser. What’s cool about that is that it can run headless. This means that it can run from the command prompt, invisibly but still as a fully-featured web browser. Using Python, one can automate any task. So logging into TradingView and extracting a strategy’s long/short calls is certainly something that can be done. This is what we did. However, when it comes to fetching data online, a headless browser is usually to be avoided at all costs, if possible.

One reason for that is that web site code changes all the time. If TradingView makes a change to their web site’s code, it can break the bot. So you have to be constantly on top of it. Another reason is that it doesn’t scale well. As it stands now, each time an instance is deployed, a new headless browser window has to be run to login to TradingView, load the strategy, and get the trades. It’s not that it can’t be done this way but it’s really undesirable and difficult to manage when it comes to multiple bot instances. Not only is the current approach prone to bugs and scaling issues, it’s expensive. Each instance launching a new browser window is very computationally expensive, requiring more CPU & RAM server-side. Scaling has been our biggest issue up to now.

The Final Solution

The Question

We once again have a backlog of support tickets due to Selenium scaling issues. The bot was recently taken offline in order to create a new patch. We still haven’t been able to employ a socket-based solution. We really need to get away from Selenium, somehow. I would much rather put a solid solution in place than make another patch that could wind up bugging out again. Is there another way forward we haven’t previously considered? Is it possible to have a super-stable Pine script-based platform without sockets? It turns out there is.

The Answer (Python To The Rescue)

As it turns out, sockets and headless web browsers weren’t the only way. After scouring documentation for hours-on-end, one more (The best of all of them) solution was discovered. If one could convert Pine Script to another programming language, one that easily runs on servers, all problems would be solved. Not only would they be solved, that solution would be beautiful, elegant, sexy. Epic even. If one could convert Pine Script to say, Python, not only would strategies be able to run on our servers with minimal resources. It would be reliable, instantly scalable, and have the added benefit of opening up a whole new world of tools and other possibilities.

Imagine it. You code and backtest a good TradingView strategy. You take your Pine Script and run it through the converter. Now, not only can Pinebot use the resulting Python code to run a super-efficient bot for you, it can do so much more. I love programming machine learning code (Tensorflow, PyTorch, etc.). All the best machine learning is done in Python. With a Pine to Python converter, we could combine the best parts of TradingView with raw power of machine learning algorithms. Anything that Python can do could be augmented with Pine scripts. But how? How feasible is such a converter?

Gods Bless The Engineers At TradingView

This is where we slightly depart from my area of expertise. I’m a full-stack developer, engineer, and even scientist in most regards. One thing I’ve never done is create my own programming language. I’ve just never had the need. So creating a Pine Script to Python transpiler isn’t exactly in my wheelhouse. I do understand many of the principals involved. I’m also somewhat familiar with some of the libraries needed to do it but from scratch? That would take a specialist. Thankfully, by the grace of the Matrix we live in, TradingView’s engineers published a starting point! They didn’t write any articles about it. There’s no blog post about it but they did happen to publish two source files aimed at aiding in the programmatic interpretation of Pine Script. This is a godsend.

Not only will this ensure the long-term success of Pinebot, it opens up a whole new world of possibility for us. It means that not only will Pinebot be stable, Pinebot will be propelled forward as quite possibly the most advantageous algorithmic trading platform on the Internet. Pine Script meets bot automation meets artificial intelligence and everything that every Python library has to offer. I can’t tell you how excited I am for this. Thanks to TradingView’s publications, not only is this going to come to fruition, it shouldn’t even take that much time. Onward we go.

Conclusions

It’s been a rocky road. As a platform that’s still in beta, Pinebot had a lot of potential, even without these new advents. Prior to them, we would have chugged along with our patches until our Selenium engine was at least stable enough to take us into the future. However, it would have come at some important costs, as we’ve outlined. Now, we’ve both eliminated that difficulty and discovered a stable, innovative, and futuristic (So-to-speak) way forward. There’s important work ahead but it won’t take much longer.

Our Pine Script to Python transpiler is already underway. For those that have stuck with us, we greatly appreciate your patronage. As soon as our transpiler is online, all Pinebot users who have ever held a subscription will be given a free month of Premium access. If you subscribed just prior to our recent downtime, you will also be issued extra credit equivalent to your subscription.

The transpiler not only enables stability and new features. It slashes the monetary cost of supporting bot instances. This will also translate into savings for new customers. As for our current, Selenium-based engine: If our transpiler looks like it’s going to take more than another week or so (We’ll know in the next few days), we’ll code another patch and bring it online in the interim. I can’t promise the long-term stability of such a patch (As we’re moving away from Selenium anyway) but I can promise that your subscription is worth it. Anytime we have maintenance, we more than compensate our subscribers for it. For updates on our transpiler and everything else, keep and eye on our blog or Twitter feed as we take Pinebot to the next level.

We will be happy to hear your thoughts

Leave a reply

Register New Account
We've been waiting for you.
Name (required)
TradingView Username
Reset Password