More Facebook c***ery

[Snake Plissken]

So, is this just a coincidental outage or is Facebook being massively fucked with here? This is a long, long delay.

Self inflicted by the looks of it.

[wingco]

I'd bet on that too; same day as the whistleblower thing makes it a bit rum.

[Snake Plissken]

OK, explanation by crap analogy.

Imagine you have someone coming to your house and there is no such thing as sat nav. (Kids, ask your parents.) You would give them directions. Something like "Leave the M1 at junction 9, travel half a mile, turn left, turn right, second left and we're 100 yards on the right." The important bit is that you have left it up to the other person to get themselves to M1 junct 9, and then been very specific about the last bit of the directions.

But what if you got those directions wrong? Everyone makes their way to M1 junct 9 and then follows the detailed instructions and... gets lost.

That's basically what Facebook have done. They've push a wrong routing update out to the internet. So any computers trying to get to them can't do so, because that last bit of routing info is wrong.

What makes this sort of amusing is that because they've fucked up the directions to the computer, they can't get back into the computer remotely to fix it. They have to do that bit in the physical data centres, which is precisely where the techies with the knowledge aren't. And how do the techies talk to each other to coordinate. On Facebook of course! And WhatsApp. And all the other communication systems that have just gone down.

[wingco]

Right. But this sort of bungle is unprecedented, right? Why doesn't it happen more often?

[Snake Plissken]

It does, just on a lower scale. And occasionally a larger scale. For instance, a Pakistani ISP a few years back tried to block YouTube for its customers and accidentally pushed the routing update too far, knocking YouTube off the internet completely.

Internet infrastructure runs largely on trust. It isn't one big network, it's millions of small networks all talking to each other. These small networks pass traffic between each other. That traffic can be email, web, video, whatever. Part of the traffic is updates on where domain names and websites live. Because the Internet is all these small networks and to avoid a single controlling point (or point of failure) then the networks necessarily have to trust each other. So if Facebook does what it did and pushes a message out that says "we no longer exist" then all the small networks pass that message along and it propagates worldwide.

Normally when the problem is spotted Facebook would push another message saying "Ignore that, we're here", but because they locked themselves out of access to their own system by saying "we no longer exists", they couldn't send the follow-up message. Meanwhile, the original message is happily making its way through millions of networks, slowly but surely knocking Facebook, WhatsApp and Instagram off the Internet.

[Sheep]

https://twitter.com/cameron_kasky/status/1445091373703643137

[wingco]

Very interesting. How long might it be out? And what are the consequences for a site that is generally perma-present and generating a vast amount of activity?

[ursus arctos]

Because those updates are normally subject to numerous checks to catch errors

Maybecq bunch of people were asleep at the switch, or maybe they rushed something, or maybe a few people in the chain are really pissed off.

Having the doors be app dependent is hilarious. If only the weather in Palo Alto was worse.

[Discordant Resonance]

OK, explanation by crap analogy.

Imagine you have someone coming to your house and there is no such thing as sat nav. (Kids, ask your parents.) You would give them directions. Something like "Leave the M1 at junction 9, travel half a mile, turn left, turn right, second left and we're 100 yards on the right." The important bit is that you have left it up to the other person to get themselves to M1 junct 9, and then been very specific about the last bit of the directions.

But what if you got those directions wrong? Everyone makes their way to M1 junct 9 and then follows the detailed instructions and... gets lost.

That's basically what Facebook have done. They've push a wrong routing update out to the internet. So any computers trying to get to them can't do so, because that last bit of routing info is wrong.

What makes this sort of amusing is that because they've fucked up the directions to the computer, they can't get back into the computer remotely to fix it. They have to do that bit in the physical data centres, which is precisely where the techies with the knowledge aren't. And how do the techies talk to each other to coordinate. On Facebook of course! And WhatsApp. And all the other communication systems that have just gone down.

Ironically, or perhaps not, the big Irish political debate of the last week has concerned data centres, and whether their huge electrical and environmental footprint outweighs the commercial benefits of building them.

[ursus arctos]

In personal news, I happened to be scheduled for a periodic What's App backup (in background) today.

it has been spinning its tyres for eight hours now

[Snake Plissken]

They've got a holding page back up so it will probably be an hour or so, I reckon.

It will come down to user typing the wrong value in. It usually does. Some poor bastard just experienced the infamous unit of time known as the "ohnosecond".

[ursus arctos]

I'd bet on that too; same day as the whistleblower thing makes it a bit rum.

It was only the 60 Minutes interview that was last night in the US

The Wall Street Journsl have been publishing stories and putting out podcasts based on the material for almost two weeks

[Snake Plissken]

And here's a more techie explanation.

https://blog.cloudflare.com/october-...cebook-outage/

[ursus arctos]

They've got a holding page back up so it will probably be an hour or so, I reckon.

It will come down to user typing the wrong value in. It usually does. Some poor bastard just experienced the infamous unit of time known as the "ohnosecond".

The finance version of this is the "fat finger" error, which only makes the news when it involves a loss of more than USD 100 million.

[wingco]

I bow to the techies here. I have no alternative wisdom or perception here, just curious.

[Sits]

Such a shame. Like many I use these platforms, but it’s still absolutely hilarious.

[Amor de Cosmos]

In personal news, I happened to be scheduled for a periodic What's App backup (in background) today.

it has been spinning its tyres for eight hours now

Out of interest, how many other people on here didn't notice anything wrong at all? Except that some of the ads on sites I normally visit had disappeared.

[caja-dglh]

Very interesting. How long might it be out? And what are the consequences for a site that is generally perma-present and generating a vast amount of activity?

Till Wyatt Earp and Spearmint Rhino find their way back here.

[Snake Plissken]

I bow to the techies here. I have no alternative wisdom or perception here, just curious.

No worries. I don’t pretend to know the ins and outs of networking but I find the sheer level of cooperation and trust required to keep the whole thing running fascinating. The entire Internet is a massively Heath Robinson thing requiring millions of things to work together just to avoid collapse.

[ursus arctos]

Out of interest, how many other people on here didn't notice anything wrong at all? Except that some of the ads on sites I normally visit had disappeared.

I would have had no idea if it hadn't been for the backup issue

I never use FB or Insta and rarely use What's App (especially recently)

[Nocturnal Submission]

Out of interest, how many other people on here didn't notice anything wrong at all? Except that some of the ads on sites I normally visit had disappeared.

I use WhatsApp quite a lot and had sent a message to my nephew this afternoon that oddly didn't dispatch. I then made a few inquiries and found a BBC story about the issue.

Rarely use FB and Insta not at all.

[Discordant Resonance]

Tis back now.

[wingco]

Back up. I posted thus. So, how did you spend your time away from Facebook? Productively, in my instance. Mindfulness, yoga and so forth. I suppose you people were pressing the “refresh” button every five minutes, so sappily attached to the matrix as you are. Not me. I re-read Italo Calvino’s Invisible Cities. I baked bread. I bred free range chickens. I practised good breathing techniques. I learned to play the oboe, an instrument I never previously considered but in which I am now semi-proficient, tootling a tune of untrammelled spirituality. I commenced work on a longform poem, Armando Ianucci style, The Ballad Of The Outage. In Spanish, a language I have also acquired. I am so much better, better than you, frankly. But then, that is to your good.

[Gangster Octopus]

I have no alternative wisdom or perception

Ha!

[S. aureus]

I noticed the news in passing, but it wouldn't have effected me at all if my daughter hadn't tried to use the Quest after doing her homework (then thrown a hissy fit when she couldn't).