Tezos indexer not robust to node timeouts
On timeout errors from a tezos node and indexing (index
management command), the indexer will fail and restart. This can happen quite often on long indexings leading to even longer indexing times.
Maybe the indexer should retry after a few moments? Maybe on a different node with a lower priority if possible?
Here is an example log:
2022-08-03 01:28:48,029 DEBUG Indexing level 2587817
2022-08-03 01:28:52,178 DEBUG Indexing level 2587816
2022-08-03 01:29:22,417 ERROR Mainnet.provider.index() failure!
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/djwebdapp/management/commands/index.py", line 13, in handle
blockchain.provider.index()
File "/usr/lib/python3.10/site-packages/djwebdapp/provider.py", line 151, in index
self.index_level(current_level)
File "/usr/lib/python3.10/site-packages/djwebdapp_tezos/provider.py", line 49, in index_level
for ops in block.operations():
File "/usr/lib/python3.10/site-packages/pytezos/rpc/query.py", line 99, in __call__
return self.node.get(
File "/usr/lib/python3.10/site-packages/pytezos/rpc/node.py", line 121, in get
return self.request('GET', path, params=params, timeout=timeout).json()
File "/usr/lib/python3.10/site-packages/pytezos/rpc/node.py", line 115, in request
raise RpcError.from_response(res)
pytezos.rpc.node.RpcError: ('<!DOCTYPE html>\n'
'<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->\n'
'<!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->\n'
'<!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->\n'
'<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->\n'
'<head>\n'
'\n'
'\n'
'<title>mainnet.api.tez.ie | 502: Bad gateway</title>\n'
'<meta charset="UTF-8" />\n'
'<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />\n'
'<meta http-equiv="X-UA-Compatible" content="IE=Edge" />\n'
'<meta name="robots" content="noindex, nofollow" />\n'
'<meta name="viewport" content="width=device-width,initial-scale=1" />\n'
'<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" '
'/>\n'
'\n'
'\n'
'</head>\n'
'<body>\n'
'<div id="cf-wrapper">\n'
'\n'
' \n'
'\n'
' <div id="cf-error-details" class="p-0">\n'
' <header class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-8">\n'
' <h1 class="inline-block sm:block sm:mb-2 font-light text-60 '
'lg:text-4xl text-black-dark leading-tight mr-2">\n'
' \n'
' <span class="cf-error-type">Error</span>\n'
' <span class="cf-error-code">502</span>\n'
' </h1>\n'
' <span class="inline-block sm:block font-mono text-15 lg:text-sm '
'lg:leading-relaxed">Ray ID: 734b476e8e0ab8f7 •</span>\n'
' <span class="inline-block sm:block font-mono text-15 lg:text-sm '
'lg:leading-relaxed">2022-08-03 01:29:22 UTC</span>\n'
' <h2 class="text-gray-600 leading-1.3 text-3xl font-light">Bad '
'gateway</h2>\n'
' </header>\n'
' \n'
' <div class="my-8 bg-gradient-gray">\n'
' <div class="w-240 lg:w-full mx-auto">\n'
' <div class="clearfix md:px-8">\n'
' \n'
'<div id="cf-browser-status" class=" relative w-1/3 md:w-full py-15 md:p-0 '
'md:py-8 md:text-left md:border-solid md:border-0 md:border-b '
'md:border-gray-400 overflow-hidden float-left md:float-none text-center">\n'
' <div class="relative mb-10 md:m-0">\n'
' <span class="cf-icon-browser block md:hidden h-20 bg-center '
'bg-no-repeat"></span>\n'
' <span class="cf-icon-ok w-12 h-12 absolute left-1/2 md:left-auto '
'md:right-0 md:top-0 -ml-6 -bottom-4"></span>\n'
' </div>\n'
' <span class="md:block w-full truncate">You</span>\n'
' <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light '
'leading-1.3">Browser</h3>\n'
' <span class="leading-1.3 text-2xl text-green-success">Working</span>\n'
'</div>\n'
'\n'
'<div id="cf-cloudflare-status" class=" relative w-1/3 md:w-full py-15 md:p-0 '
'md:py-8 md:text-left md:border-solid md:border-0 md:border-b '
'md:border-gray-400 overflow-hidden float-left md:float-none text-center">\n'
' <div class="relative mb-10 md:m-0">\n'
' <span class="cf-icon-cloud block md:hidden h-20 bg-center '
'bg-no-repeat"></span>\n'
' <span class="cf-icon-ok w-12 h-12 absolute left-1/2 md:left-auto '
'md:right-0 md:top-0 -ml-6 -bottom-4"></span>\n'
' </div>\n'
' <span class="md:block w-full truncate">Amsterdam</span>\n'
' <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light '
'leading-1.3">Cloudflare</h3>\n'
' <span class="leading-1.3 text-2xl text-green-success">Working</span>\n'
'</div>\n'
'\n'
'<div id="cf-host-status" class="cf-error-source relative w-1/3 md:w-full '
'py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b '
'md:border-gray-400 overflow-hidden float-left md:float-none text-center">\n'
' <div class="relative mb-10 md:m-0">\n'
' <span class="cf-icon-server block md:hidden h-20 bg-center '
'bg-no-repeat"></span>\n'
' <span class="cf-icon-error w-12 h-12 absolute left-1/2 md:left-auto '
'md:right-0 md:top-0 -ml-6 -bottom-4"></span>\n'
' </div>\n'
' <span class="md:block w-full truncate">mainnet.api.tez.ie</span>\n'
' <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light '
'leading-1.3">Host</h3>\n'
' <span class="leading-1.3 text-2xl text-red-error">Error</span>\n'
'</div>\n'
'\n'
' </div>\n'
' \n'
' </div>\n'
' </div>\n'
'\n'
' <div class="w-240 lg:w-full mx-auto mb-8 lg:px-8">\n'
' <div class="clearfix">\n'
' <div class="w-1/2 md:w-full float-left pr-6 md:pb-10 md:pr-0 '
'leading-relaxed">\n'
' <h2 class="text-3xl font-normal leading-1.3 mb-4">What '
'happened?</h2>\n'
' <p>The web server reported a bad gateway error.</p>\n'
' </div>\n'
' \n'
' <div class="w-1/2 md:w-full float-left leading-relaxed">\n'
' <h2 class="text-3xl font-normal leading-1.3 mb-4">What '
'can I do?</h2>\n'
' <p class="mb-6">Please try again in a few minutes.</p>\n'
' </div>\n'
' </div>\n'
' \n'
' </div>\n'
'\n'
' <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 '
'sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t '
'border-gray-300">\n'
' <p class="text-13">\n'
' <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong '
'class="font-semibold">734b476e8e0ab8f7</strong></span>\n'
' <span class="cf-footer-separator sm:hidden">•</span>\n'
' <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block '
'sm:mb-1">\n'
' Your IP:\n'
' <button type="button" id="cf-footer-ip-reveal" '
'class="cf-footer-ip-reveal-btn">Click to reveal</button>\n'
' <span class="hidden" id="cf-footer-ip">212.129.41.185</span>\n'
' <span class="cf-footer-separator sm:hidden">•</span>\n'
' </span>\n'
' <span class="cf-footer-item sm:block sm:mb-1"><span>Performance & '
'security by</span> <a rel="noopener noreferrer" '
'href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" '
'target="_blank">Cloudflare</a></span>\n'
' \n'
' </p>\n'
' <script>(function(){function d(){var '
'b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal");b&&"classList"in '
'b&&(b.classList.remove("hidden"),c.addEventListener("click",function(){c.classList.add("hidden");a.getElementById("cf-footer-ip").classList.remove("hidden")}))}var '
'a=document;document.addEventListener&&a.addEventListener("DOMContentLoaded",d)})();</script>\n'
'</div><!-- /.error-footer -->\n'
'\n'
'\n'
' </div>\n'
'</div>\n'
'</body>\n'
'</html>\n'
'\n',)
Found 0 blockchain to index! check is_active
Sentry is attempting to send 1 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit