Troubleshooting

Updated 31 Oct 2023

1. Current version of go-opera

The current supported version is go-opera 1.1.3-rc.5 for both mainnet and testnet.

For Go-opera older than 1.1.3-rc.5, you should download and use 1.1.3-rc.5.

2. Pruning node state

If your node is about to run out of space, you may consider to extend the machine's storage OR to prune the current node's datadir. To prune the datadir, please use the following steps:

2.1 Manual pruning

• Stop the node pkill opera

• Run at the terminal: ./opera snapshot --db.preset <preset> prune-state

The state pruning process may take a couple of hours for every hundreds of GBs of data and the amount of time required will depend on the machine's speed. Thus, you can run with nohup.

2.2 Automatic pruning

Automatic pruning is introduced since 1.1.1-rc.2. It will still be supported in the latest version 1.1.3-rc.5. You can run a node with --gcmode flag, either --gcmode full or --gcmode light. Note that

• Both --gcmode full and --gcmode light will prune data that is processed after gcmode is enabled. Old data (before gcmode is enabled) is untouched.

• Validator node can use --gcmode light but should not use --gcmode full.

• --gcmode full option will prune much more EVM nodes than --gcmode light at expense of worse performance.

3. Validator node

How to rerun a node if it is stopped

If your node is stopped (for some reason), please examine the server log to identify if there was any issue. After fixing the issue (if any), you can run the node in read mode to sync to the latest block. After it is synced up, you can stop the node and run in validator mode.

Note that, please make sure your node is synced in read mode first, before it is run in validator mode.

Migration to a new server

If you'd like to migrate your node to a new server, please follow the following steps: - Set up a 'read' node on a new server, and allow it to run to sync to the latest block. - Stop the old node in at least 40 mins before run the validator mode on the new server. - After the old node is stopped for 40 mins, then you can run in validator mode on the new server. Note that, you should not let the old node run again as it will result in a double-sign and slashing of your validator node.

How to stop a node

Find the running process of opera using ps, and then kill the process by id.

Note that, after your node is stopped, if you want to rerun it again, don't run directly in validator mode. Instead, please make sure your node is synced in read mode first, before it is run in validator mode.

Offline node

If your validator node is down for more than 5 days, then it will become offline (i.e., pruned from the network). For offline node, you can undelegate and wait for 7 days to withdraw (bonding time). After that, you can transfer fund to a new wallet and make a new validator if you wish. Note that, if undelegating a locked stake or locked delegation before the locked period is expired, it will incur a penalty.

How to permanently shut down a node

To shutdown a node permanently, you can simply stop running the node in validator mode for 5 days or more. After that, it will become Offline.

How to unstake

If your node stake is locked, you will first need to call unlockStake() to unlock it. Note that, penalty will apply for early unlocking before lockup is expired. https://github.com/Fantom-foundation/opera-sfc/wiki/Lockup-calls-reference Then you can call undelegate(), to unstake your stake.

https://github.com/Fantom-foundation/opera-sfc/wiki/Delegation-calls-reference

Then there is a waiting period of 7 days (so-called bonding time) after undelegation. This is required before you can call withdraw() to take out your stake. https://github.com/Fantom-foundation/opera-sfc/wiki/Delegation-calls-reference#withdraw

4. Troubleshooting

4.1 Syncing error

If your node is in dirty state (it may happen occasionally), please run: opera --db.preset legacy-ldb db heal --experimental lternatively, you may do a fresh resync as follows:

• Stop the node

• Remove the current (broken) datadir (the default datadir is located at ~/.opera)

• Download and build the latest version go-opera 1.1.3-rc5

• Run your node again in read mode

4.2 Slow syncing

Check your machine specs if it meets the minimum requirements.

• IOPS greater than 5000 (higher is better)

• connection speed > 1 Gbps (some ppl run with 10 or 20, if they can)

• cores: more than 4 cores (the number of cores is not important unless you will use it for serving API calls).

• CPU: > 3GHz.

You can also check the following flags, if you're using them to run your node. You can adjust to values suitable to your usage.

• maxpeers flag: default is 50, you can adjust it depending on your machine.

• cache flag: --cache 15792 (A larger value can give better performance).

• gcmode: gcmode is not enabled by default. If enabled, gcmode (light or full) it will take some extra CPU and time.

You can also increase the value of ulimit on your machine.

Increase open files limit

You can check your current limit value on Linux with the command ulimit -n. The default value of 1024, which may be not enough in some cases.

You can adjust the value to the recommended 500.000 open files limit by either: - ulimit -n 500000 - change it in /etc/security/limits.conf configuration file, limit type nofile.