I forgot to post this earlier, but the timing was not clear until about 2 hours ago:
We’ll be shutting down leng (which hosts EVERYTHING) for ~1h to replace a faulty memory module.
Aprox. 2-3pm PDT.
Mail will queue and be delivered when the machine comes back.
Apologies for the short notice.
The virtual machines on which we’re running most of the services that make up slackers.net are losing time. Some of them very badly. This would be only a minor annoyance, except that Kerberos (our authentication service) is finicky about times matching across servers.
We’ve identified the problem, and have a solution in place, but it requires changing kernel parameters (which is to say, a reboot). We’re planning on taking things down quickly around 7pm PST. We’ll do it in a cascade so that there is no interruption for mail delivery, and minimal disruption for other services.
Thanks for your patience.
We've just received word that the fiber link to our upstream provider's facility was cut (save vs. backhoe: failed!) We have no ETA on restoration of service, but we will keep you posted as we know more.
[I wrote an update on the train yesterday, and forgot to post it. My machine did its "I'm awake, but the screen is dark." thing this morning, and I can't find it anywhere. I'm afraid I may have saved it in /tmp, which gets cleared on boot.]
Everything is live and working.
Unfortunately, an office nyetwerk hiccup just as I was testing the mail queue drain caused me to be unable to prevent the entire queue from draining before the configuration was correct. (It didn't take long, the new machine is fast!)
I hadn't yet made a backup and forgot to set the configuration to soft-bounce, so I once again managed to lose everything in the queue. I feel wretched about this, but there's nothing I can do about it.
Worse, all of the senders received a bounce message with an obnoxious(ly cryptic) error message: "You are not me. Nice Try. FOAD." (This is because the configuration the messages tripped is normally reserved for spammers who claim to be on the internal network when they're clearly not)
I'm terribly sorry about the whole thing, this is hardly the auspicious beginning I'd hoped for. On the plus side, from here on out things should be MUCH more reliable (and fast!) from here on out.