Common Incidents and how debug and fix them

Error Code

Causes (Ordered by Frequency)

Block lags for a chain remain high for over 6 hours

  1. Bitquery Based

    1. if their block syncing is delayed, we will see elevated block lags

    2. airflow task which processes the micro-batching could stop executing

    3. writes into tigergraph can slow down significantly

  2. For chains built internally, common causes are

    1. hard fork on the blockchain and we have not upgraded our node

    2. third-party api we rely on is temporarily unavailable

    3. third-party api we rely on is rate-limiting us

    4. bigquery-public-datasets we rely on are broken

    5. streamer’s which query node / 3rd api are no longer publishing messages into pub-sub

    6. airflow task which publishes messages into pub-sub is not executing

    7. writes into tigergraph can slow down significantly