Tezos indexer is not robust to invalid chunks
When indexing a Tezos contract, sometime the connection is broken leading to invalid chunk lengths. This will restart the indexer from the top and can lead to serious indexing times.
Maybe the indexing process should catch such exceptions and retry after a few moments and/or an another node with a lower priority?
Here is an example error log:
2022-08-03 01:43:18,689 DEBUG Indexing level 2588360
2022-08-03 01:43:21,045 DEBUG Indexing level 2588359
--
File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 575, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 755, in read_chunked
with self._error_catcher():
File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 458, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/djwebdapp/management/commands/index.py", line 13, in handle
blockchain.provider.index()
File "/usr/lib/python3.10/site-packages/djwebdapp/provider.py", line 151, in index
self.index_level(current_level)
File "/usr/lib/python3.10/site-packages/djwebdapp_tezos/provider.py", line 49, in index_level
for ops in block.operations():
File "/usr/lib/python3.10/site-packages/pytezos/rpc/query.py", line 99, in __call__
return self.node.get(
File "/usr/lib/python3.10/site-packages/pytezos/rpc/node.py", line 121, in get
return self.request('GET', path, params=params, timeout=timeout).json()
File "/usr/lib/python3.10/site-packages/pytezos/rpc/node.py", line 98, in request
res = requests.request(
File "/usr/lib/python3.10/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 687, in send
r.content
File "/usr/lib/python3.10/site-packages/requests/models.py", line 838, in content
self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
File "/usr/lib/python3.10/site-packages/requests/models.py", line 763, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
Found 0 blockchain to index! check is_active
Sentry is attempting to send 1 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit