Failure handling

  • On failure, should increment count in DB + add failure date
  • Retry doesn't happen for X amount of time(configurable)
  • After Y number of retries(configurable), give up and mark row as unscrapeable
  • Hit callback with that information