So we store these "Bug" objects. How do they update?
Each Bug object doesn't know how to update itself. Instead, we have a nightly job per bug tracker. When that job wakes up, it refreshes all the bugs that came from that bug tracker.
So if a Bug object's last_polled stamp is more old than one day + one hour, then it means we're not properly updating it from its bug tracker. If it's older than two days, then something is terribly wrong!
These Bug diagnostics only apply to bugs we think are open. We happen to retain information on bugs that were once open (for example, if the bug was marked as resolved), but it is not important if those bugs become stale.
| Bugs last polled more than than two days + one hour ago: | 3767 |
| Bugs last polled more than three days ago: | 3715 |
| Bugs last polled more than three days ago (in percent): | 23.2056967956 |
These diagnostics explain how well we are handling "DIAs". A DataImportAttempt is created whenever a user submits a username or email address to the profile importer. When that happens, we create a DataImportAttempt object and save it to the database. We also enqueue a job to the background job daemon (celeryd).
If we're lucky, the background jobs get handled quickly. The imports from Ohloh can take minutes, whereas faster services like Bitbucket and Github respond within seconds.
We run some number (typically 4) of celeryd workers to execute these jobs. That means can run that many (typically 4) tasks at once, or so. If users are submitting jobs faster than we're processing them, then their experience will suck.
Really, at any given time, the goal is 0 uncompleted DataImportAttempts. As soon as someone hits submit to start the importer, there will be a few uncompleted DIAs. Hopefully we'll finish them within one minute. If we take more than 5 minutes, then something is definitely wrong!
| Uncompleted DIAs older than 5 minutes: | 0 |
| Uncompleted DIAs: | 0 |
| Uncompleted DIAs older than 1 minute: | 0 |