Thread 'tokio-runtime-worker' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key #3575
Description
we run Moonbeam v0.48.0 and substrate-api-sidecar v20.2.2
Sometimes errors occur and the node shuts down. Any idea how to solve this problem?
Error logs,
Dec 09 14:24:48 glmr01 moonbeam[2045223]: Thread 'tokio-runtime-worker' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0x3ef9dd889a7c578a9ab9467e64878968f666c8334a882b5f9db69282280da8ba"', /root/.cargo/git/checkouts/polkadot-sdk-38b703c7469a7d1e/73cb19c/substrate/primitives/state-machine/src/ext.rs:175
Dec 09 14:24:48 glmr01 moonbeam[2045223]: This is a bug. Please report it at:
Dec 09 14:24:48 glmr01 moonbeam[2045223]: https://github.com/moonbeam-foundation/moonbeam/issues/new
Dec 09 14:24:48 glmr01 moonbeam[2045223]: ====================
Dec 09 14:24:48 glmr01 moonbeam[2045223]: Version: 0.48.0-570f126cd42
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 0: sp_panic_handler::set::{{closure}}
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 1: std::panicking::rust_panic_with_hook
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 2: std::panicking::begin_panic_handler::{{closure}}
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 3: std::sys::backtrace::__rust_end_short_backtrace
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 4: rust_begin_unwind
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 5: core::panicking::panic_fmt
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 6: core::result::unwrap_failed
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 7: <sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::storage
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 8: sp_io::storage::ExtStorageGetVersion1::call
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 9: <F as wasmtime::func::IntoFunc<T,(wasmtime::func::Caller,A1),R>>::into_func::wasm_to_host_shim
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 10:
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 11:
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 12:
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 13:
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 14: wasmtime_runtime::traphandlers::catch_traps::call_closure
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 15: wasmtime_setjmp
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 16: sc_executor_wasmtime::runtime::WasmtimeInstance::call_impl
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 17: sc_executor_common::wasm_runtime::WasmInstance::call_export
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 18: sc_executor::executor::WasmExecutor::with_instance::{{closure}}
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 19: <sc_executor::executor::WasmExecutor as sp_core::traits::CodeExecutor>::call
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 20: sp_state_machine::execution::StateMachine<B,H,Exec>::execute
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 21: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor>::contextual_call
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 22: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt>::call_api_at
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 23: <moonbeam_runtime::RuntimeApiImpl<SrApiBlock,RuntimeApiImplCall> as sp_api::Core<SrApiBlock>>::__runtime_api_internal_call_api_at::{{closure}}
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 24: <moonbeam_runtime::RuntimeApiImpl<SrApiBlock,RuntimeApiImplCall> as sp_api::Core<SrApiBlock>>::__runtime_api_internal_call_api_at
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 25: sc_transaction_pool::common::api::validate_transaction_blocking
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 26: <sc_transaction_pool::common::api::FullChainApi<Client,Block> as sc_transaction_pool::graph::pool::ChainApi>::validate_transaction::{{closure}}::{{closure}}
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 27: sc_transaction_pool::common::api::spawn_validation_pool_task::{{closure}}
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 28: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 29: <tracing_futures::Instrumented as core::future::future::Future>::poll
Dec 09 14:24:48 glmr01 moonbeam[2045223]: 30: tokio::runtime::task::raw::poll