Notification System
**FINAL EDIT Thu Dec 10 02:15:47 UTC 2009**
So there is the final update... Over the past day we have processed around 11 million jobs out of the 12 million that were in queue at that time. Please bear in mind that over this past day, more jobs for notifications are also created. So while the queue has been dropping, we are still not fully caught up at this point, due to backlog and new jobs. We have roughly 3 million jobs still pending that involve the notification system in some manner. We had hoped we could have fully cleared the queue in a day, but unfortunately we can't clear it too quickly, since we need the rest of the site to operate normally. From our current perspective on the amount of jobs that are left in queue, and how many it has processed thus far, we believe it will take around another 8 - 12 hours to process everything.
And finally some answers to some questions:
1. The free snowflake gift that some people found in the gift shop over the weekend wasn't meant to be there; it was a side-effect of code related to the new payment processing and LiveJournal shop backend we've been working on most of this year (and a good portion of which went live in release #59). No user-facing differences were supposed to be noticeable, but this one v-gift was inadvertently added temporarily and marked as free. As part of the upgrading of our payment system, we'll be able to offer free v-gifts in the future, so if you enjoyed having this to send around over the weekend hang tight for a few weeks and we'll have the capability to offer you some more.
2. The snowflake v-gift wasn't the direct cause of the notification issue, but was indirectly the source of these problems. To make a long story short, the notifications caused error messages (since this wasn't supposed to be where it was) that overloaded our queues (for those of you familiar with LiveJournal's backend, it was error-ing out like woah in the schwartz queue). Thanks to some really fast and good work by our engineering department, those errors are no longer happening. Unfortunately, if you change your notification settings to remove v-gift notifications it won't affect the notifications that are already in the queue, only those going forward.
3. Notifications are back again, and all of the ones that weren't sent on time over Monday and Tuesday are in the process of going out. Over the next 8 - 12 hours or so, we anticipate the backlogged notifications will all be sent out and things will return to normal. We know it's frustrating, we feel your pain (and ours too) - thank you very much for all the patience and understanding you've had in the past day or so.
Thanks,
LJ Operations
**EDIT Wed Dec 9 00:28:16 UTC 2009 **
The notification system has been fixed in the new release... we are currently processing the queue, which is upwards of 12m jobs... please bear with us while our workers chew through this large queue and get your notification / emails out. Some may come more quickly than others due to weights on the notifications themselves, but we are hoping in the next 24 hours to have all the queues cleared and all notifications delivered that had been queued up over the past few days.
Again I apologize for this inconvenience, but we are almost out of the woods as soon as we are done clear cutting some of the forest ;)
Thanks,
** END EDIT**
Hey Guys,
Unfortunately with our last release, and its instability, we were forced to roll back releases. Unfortunately in doing so, it would seem that our notification system has been broken somehow. Our engineers are working on this issue as quickly as possible. We hope to have a patch within the next day, so we can deploy our code and fix the notification system at the same time. Please *bear* with us ;)
Currently all notifications are being queued up so they can be processed as soon as the fix is pushed and verified to be working correctly.
Thank you,
So there is the final update... Over the past day we have processed around 11 million jobs out of the 12 million that were in queue at that time. Please bear in mind that over this past day, more jobs for notifications are also created. So while the queue has been dropping, we are still not fully caught up at this point, due to backlog and new jobs. We have roughly 3 million jobs still pending that involve the notification system in some manner. We had hoped we could have fully cleared the queue in a day, but unfortunately we can't clear it too quickly, since we need the rest of the site to operate normally. From our current perspective on the amount of jobs that are left in queue, and how many it has processed thus far, we believe it will take around another 8 - 12 hours to process everything.
And finally some answers to some questions:
1. The free snowflake gift that some people found in the gift shop over the weekend wasn't meant to be there; it was a side-effect of code related to the new payment processing and LiveJournal shop backend we've been working on most of this year (and a good portion of which went live in release #59). No user-facing differences were supposed to be noticeable, but this one v-gift was inadvertently added temporarily and marked as free. As part of the upgrading of our payment system, we'll be able to offer free v-gifts in the future, so if you enjoyed having this to send around over the weekend hang tight for a few weeks and we'll have the capability to offer you some more.
2. The snowflake v-gift wasn't the direct cause of the notification issue, but was indirectly the source of these problems. To make a long story short, the notifications caused error messages (since this wasn't supposed to be where it was) that overloaded our queues (for those of you familiar with LiveJournal's backend, it was error-ing out like woah in the schwartz queue). Thanks to some really fast and good work by our engineering department, those errors are no longer happening. Unfortunately, if you change your notification settings to remove v-gift notifications it won't affect the notifications that are already in the queue, only those going forward.
3. Notifications are back again, and all of the ones that weren't sent on time over Monday and Tuesday are in the process of going out. Over the next 8 - 12 hours or so, we anticipate the backlogged notifications will all be sent out and things will return to normal. We know it's frustrating, we feel your pain (and ours too) - thank you very much for all the patience and understanding you've had in the past day or so.
Thanks,
LJ Operations
**EDIT Wed Dec 9 00:28:16 UTC 2009 **
The notification system has been fixed in the new release... we are currently processing the queue, which is upwards of 12m jobs... please bear with us while our workers chew through this large queue and get your notification / emails out. Some may come more quickly than others due to weights on the notifications themselves, but we are hoping in the next 24 hours to have all the queues cleared and all notifications delivered that had been queued up over the past few days.
Again I apologize for this inconvenience, but we are almost out of the woods as soon as we are done clear cutting some of the forest ;)
Thanks,
** END EDIT**
Hey Guys,
Unfortunately with our last release, and its instability, we were forced to roll back releases. Unfortunately in doing so, it would seem that our notification system has been broken somehow. Our engineers are working on this issue as quickly as possible. We hope to have a patch within the next day, so we can deploy our code and fix the notification system at the same time. Please *bear* with us ;)
Currently all notifications are being queued up so they can be processed as soon as the fix is pushed and verified to be working correctly.
Thank you,