Predictoor: Introducing Two-Sided Prediction Bots
pdr-backend v0.3 is out! Bots do up *and* down predictions with confidence-weighted stakes. Bonus: upgrades to simulator
## Contents
0. Summary
1. Before: One-Sided Predictions
2. New: Two-Sided Predictions
3. Upgrades to Simulator
4. Other v0.3 Improvements
5. Conclusion
6. Appendix: Simulator Plots Details0. Summary
With Ocean Predictoor, you can run AI-powered prediction bots or trading bots on crypto price feeds to earn $. pdr-backend is a key part of Predictoor implementation: it’s the GitHub repo to run predictoor bots, run trading bots, do simulations & analytics, do payouts, submit truevals, and more.
Previously, a typical predictoor bot would only submit an “up” or “down” prediction in a given epoch. As of the new pdr-backend v0.3 release, bots submit “up” and “down” predictions with a stake-weighted confidence on each, i.e. “two-sided prediction”. This reduces epoch-to-epoch variance and increases profitability.
Also, the simulator got upgraded: predictions are two-sided, the trader bot uses model confidence, and there are new plots like predictoor profit vs. time and a visual of model probability response surface.
The v0.3 release has many other improvements: in trader bots, data pipeline, UX, and core bug fixes.
Try for yourself at pdr-backend’s Predictoor bot README.
The rest of this post is organized as follows. Sections 1 and 2 describe one- and two-sided predictions (the before and after). Section 3 describes simulator upgrades, and section 4 other upgrades for v0.3. Section 5 concludes. Section 6 shows more cool plots from simulation.
1. Before: One-Sided Predictions
In a given epoch, a predictoor bot would predict a direction (“up” or “down”), and stake on that prediction (e.g. 10 OCEAN if not confident, 1000 if highly confident). Call it “one-sided predictions”. The image below illustrates.
The problem is: accuracy in up/down predictions is typically 50.5% to 60% max. With such accuracies, a bot is never very confident in its up vs down prediction. It can nonetheless make $ doing these predictions, but is there a better way?
Furthermore, if a bot gets it wrong, they are slashed the full stake amount. So there is high variance in winnings from epoch to epoch: big win or big loss. High variance hurts profitability.
Towards a better way, here’s some brief theory.
- When a bot makes a prediction, its underlying model is a Bernoulli probability distribution [1]. That is, there are exactly two possible outcomes, a single trial, and the bot’s model has estimated the probability of each outcome.
- However, in the status quo code, the bot only expresses one side of the model— the most probable side — in the course of submitting a prediction. It’s leaving information on the table, and exposing itself to high variance.
2. New: Two-Sided Predictions
The idea of two-sided predictions is to let the bot express both sides of the distribution in the course of submitting a prediction.
For example, if the bot calculates a probability of up=30%, and therefore down=70%, and has 1000 OCEAN to stake, then the bot stakes 0.30*1000 = 300 OCEAN to “up”, and 700 OCEAN to down. The bot is always on both the winning side and the losing side. The image below illustrates.
The winning side will be there to catch the slashed stakes of the losing side, which in turn greatly reduces epoch-to-epoch variance.
Updated AI model. Before, the prediction bots a regression model to predict a next value of BTC etc. Now, it’s explicitly a classifier model, classifying up vs. down. pdr-backend currently has two linear classifiers: LinearLogistic and LinearSVC [2]. The classifiers give probability estimates.
How it’s implemented. In a given epoch, a predictoor bots does two transactions to the blockchain. It needs two accounts / private keys, because the Predictoor contracts allow only one tx per epoch per account.
How to run two-sided prediction bots. First, you need to set a second account, with envvar PRIVATE_KEY2.
export PRIVATE_KEY=<YOUR_PRIVATE_KEY 1>
export PRIVATE_KEY2=<YOUR_PRIVATE_KEY 2>Second, be sure to have ROSE and OCEAN, to pay for gas fees and stake respectively.
Related minor changes:
- The model-based predictoor bot used to be approach 3. Now it’s approach 2. Approach 3 is deprecated.
- Approach 1 used to randomly pick an up-vs-down prediction. Now it puts 50% stake into “up” and 50% stake into “down” (50–50). Therefore it needs two keys too.
3. Upgrades to Simulator
The prediction/trading simulator pdr sim got upgraded.
First, the simulator’s bots got upgraded.
- Sim prediction bots are now two-sided, behaving just like in the main predictoor bots. Therefore you can do rapid “what-if” experiments in simulation, before proceeding to run a predictoor bot for real $.
- Sim trading bots are now confidence-based. Before, the trading bot would always buy OCEAN according to a fixed amount of USD if there was an “up” prediction; and did nothing on a “down” prediction. Now, in an “up” prediction it buys, in a “down” prediction it sells, both proportional to the model’s confidence in up vs down.
Second, the simulator’s plots got a big upgrade. There are now realtime plots of: predictoor profit vs time, trader profit vs time, model accuracy vs time, predictoor profit distribution, trader profit distribution, and a contour plot to show the model’s probability response surface. The appendix shows these in detail. Or, run for yourself with pdr sim my_ppss.yaml. Expect simulator plots to continue to evolve 😎.
4. Other v0.3 Improvements
From v0.3.0 release notes, here’s a thorough list of changes.
- Predictoor bots: Two-sided prediction. Easy deployment of >1 agents via Kubernetes + CLI; pm2 deprecated.
- Trader bots: live-mocking ability, enabling “ghost” trades.
- Simulator: predictions are two-sided. Trades use model confidence. New plots like predictoor profit vs. time and a visual of model probability response surface. Live trading possible.
- Data pipeline: better structure, with DuckDB at the core. Analytics are starting to get built on top.
- UX: There is now proper Python-style logging. ppss.yaml tuned. Small CLI utilities added.
- Core: The codebase is more robust, including greatly-expanded type-checking and several bug fixes. Predictoor and trader agents are refactored to be simpler. Technical debt is low — no small feat.
5. Conclusion
As of pdr-backend v0.3, predictoor bots submit “up” and “down” predictions with a stake-weighted confidence on each, i.e. “two-sided prediction”. This reduces epoch-to-epoch variance and increases profitability.
Also, the simulator got upgraded: predictions are two-sided, the trader bot uses model confidence, and there are new plots like predictoor profit vs. time and a visual of model probability response surface.
The v0.3 release has many other improvements: in trader bots, data pipeline, UX, and core bug fixes.
Try for yourself at pdr-backend’s Predictoor bot README.
6. Appendix: Simulator Plots Details
This section gives plots from an example simulation run.
Settings. It was predicting BTC/USDT on Binance, on 5m epochs. The predictoor bot LinearLogistic model, using the most recent value of BTC and of ETH, staking 100 OCEAN per epoch. The trader used up to 1000 USD per epoch. Simulation ran for about 2000 epochs (1 week). The YAML file given below lists all parameters.
Below is predictoor profit vs. epoch. After 2000 epochs (1 week), it netted 915 OCEAN.
Below is trader profit vs. time. After 2000 epochs (1 week), it netted $2613.19 USD.
Below left is model accuracy vs time, where accuracy is the % time it was correct in predicting up vs. down. Green is the estimated value, and the grey shadow is the 95% [lower bound, upper bound] confidence intervals according to a Binomial distribution.
Below middle and right are predictoor and trader profit distributions as 1d scatterplots, respectively.
Below is a contour plot to visualize the model probability response surface. We can have a contour plot because this run has only two input variables: most recent value of ETH (y-axis), and most recent value of BTC (x-axis).
- The model predicts a direction of “down” for any (x,y)=(prev ETH, prev BTC) in the red region. The model predicts a direction of “up” for any (x,y) in the blue region.
- The decision boundary is white. The contours towards the white are lower confidence; contours moving farther from white are higher confidence.
- The red and blue points are training data for “went down” and “went up” respectively. Being encircled by yellow means that the model mis-classified on the training data; this is not surprising because we should not expect these two input variables to fully capture the response.
Below are the relevant settings in ppss.yaml for the run.
lake_ss:
parquet_dir: parquet_data
feeds:
- binance BTC/USDT ETH/USDT 5m
st_timestr: 2023-06-01_00:00 # starting date for data
fin_timestr: now # ending date for data
predictoor_ss:
predict_feed: binance BTC/USDT c 5m
approach: 2 # 1->50/50; 2->model-based
stake_amount: 100 # How much your bot stakes. In OCEAN per epoch, per feed
sim_only:
others_stake: 10000 # How much all others' bots stake. In OCEAN per epoch, per feed. Calcs: stake volume = 10e6 $/day/20_feeds = 34700 $/epoch/20_feeds (bc 288 5m epochs/day) = 1735 $/epoch/feed = 2313 OCEAN/epoch/feed (at 1 OCEAN=$0.75)
others_accuracy: 0.50001 # What percent of others' bots stake is correct. Value in range 0.0-1.0. 0.50001 means mostly 50-50 stake and some small fry predicting a bit accurately
revenue: 0.93006 # Sales revenue going towards predictoors. In OCEAN per epoch, per feed. Calcs: 37500 OCEAN/week/20_feeds (Predictoor DF rewards) = 18.6012 OCEAN/epoch/20_feeds (bc 2016 5m epochs/week) = 0.93006 OCEAN/epoch/feed
bot_only:
s_until_epoch_end: 60 # in s. Start predicting if there's > this time left
aimodel_ss:
input_feeds:
- binance BTC/USDT ETH/USDT c 5m
max_n_train: 5000 # no. epochs to train model on
autoregressive_n : 1 # no. epochs that model looks back, to predict next
approach: LinearLogistic
trader_ss:
feed: binance BTC/USDT c 5m
sim_only:
buy_amt: 1000 USD # buy this amount in each epoch
fee_percent: 0.0 # simulated % fee
init_holdings:
- 100000 USDT
- 0 BTC
bot_only:
min_buffer: 30 # in s. only trade if there's > this time left
max_tries: 10 # max no. attempts to process a feed
position_size: 3 # buy/sell this amount in each epoch
exchange_only:
timeout: 30000
options:
createMarketBuyOrderRequiresPrice": False
defaultType: spot
sim_ss: # sim only
do_plot: True
log_dir: logs
test_n : 2000 # number of epochs to simulate
exchange_only:
timeout: 30000
options:
createMarketBuyOrderRequiresPrice": False
defaultType: spotNotes
[1] “The Bernoulli distribution is a special case of the binomial distribution where a single trial is conducted (so n would be 1 for such a binomial distribution). It is also a special case of the two-point distribution, for which the possible outcomes need not be 0 and 1.” [Ref Wikipedia]
[2] Here are details on the default classifier models. LinearLogistic is “logistic regression”, via sklearn library. (Yes, it’s a classifier; the name is confusing). LinearSVC is support vector classification with a linear kernel, also via sklearn. The pdr-backend’s aimodel/ directory implements these.
Final Note
None of the content in this post should be taken as financial advice. Everything you do is your responsibility, at your discretion.
About Predictoor & Ocean Protocol
In Ocean Predictoor, people run AI-powered prediction bots or trading bots on crypto price feeds to earn $. Follow Predictoor on Twitter, and get support in discord. Track progress on GitHub at pdr-backend and more. Predictoor runs on Oasis Sapphire confidential EVM chain.
Predictoor is powered by Ocean Protocol, which provides tools to privately & securely publish, exchange, and consume data. Follow Ocean on Twitter or Telegram.
