GitHub was down

by bithavoc on 7/31/2017, 4:31 PM with 241 comments
  • by kaikai on 7/31/2017, 4:50 PM

    I would love to see a chart of traffic to other sites when GitHub goes down. My bet is that HackerNews and Twitter both get significant spikes from all those bored developers.

  • by alexchamberlain on 7/31/2017, 6:07 PM

    When I break our GitHub webhooks, I joke it's time for people to practice our Disaster Recovery (DR) procedures. In all seriousness, this is a good opportunity to practice work without GitHub. Any service can go down; can you deploy a critical bug fix without it? If not, why not and what can you do to fix it?

  • by kmfrk on 7/31/2017, 4:49 PM

    I had to change a username from capitalized to uncapitalized and use my updated remote afterwards, apologies if I broke it for everyone.

  • by _Marak_ on 7/31/2017, 7:40 PM

    If anyone is interested, I've been working with a git host that is actually distributed across a p2p network using SSB.

    see:

    https://github.com/clehner/git-ssb

    https://github.com/noffle/git-ssb-intro

    It's been working fairly well so far. We are using git-ssb to manage a few projects instead of putting them into Github.

  • by slap_shot on 7/31/2017, 4:34 PM

    Status now shows Major Service Outage:

    12:32 EDTMajor service outage.

    https://status.github.com/

  • by luhn on 7/31/2017, 4:39 PM

    Pages Builds Failure Rate spiked to over 2000%. I don't know how that's possible, but it seems pretty bad.

  • by tambourine_man on 7/31/2017, 5:32 PM

    Insert remark on why we use a centralized service for a distributed source control system, etc. No one seems to care, unfortunately

  • by rcthompson on 7/31/2017, 5:28 PM

    Looking at the status graphs, it seems like there was some clearly anomalous data starting around midnight, about 9 hours before the actual outage "began". Maybe a gradual botnet ramp-up, and 9:27 AM is when it got bad enough to overload some critical service? (Or really any other threshold-based failure scenario.)

  • by sashk on 7/31/2017, 4:57 PM

    What was happening to Github for a week or so in late June - early July? I see "The status is still red at the beginning of the day" for a whole week.

    https://status.github.com/messages/2017-07-03

  • by relaxitup on 7/31/2017, 7:24 PM

    Do these general Github outages affect GH Pages as well, or is that service portion segmented to some degree?

  • by leesalminen on 7/31/2017, 4:38 PM

    I think it started as minor as I was receiving a unicorn once per 10 pages. It's currently happening on almost all.

    Of course, I'm trying to dig into a WebKit issue and need the issues to load!

  • by pmoriarty on 7/31/2017, 6:18 PM

    Where is github hosted?

    Do they use AWS or another commercial cloud provider, or do they have their own servers in data centers (hopefully scattered around the globe)?

    If AWS, are their services spread among multiple availability groups? I'm just wondering how this could happen.

  • by pramodzion on 7/31/2017, 5:59 PM

    Github is back online.

  • by Cafey on 7/31/2017, 4:40 PM

    It has leveled up to a major outage!

  • by peterwwillis on 7/31/2017, 4:45 PM

    Dang. It's too bad their customers' source control files aren't distributed and decentralized, or they could keep working and ignore this.

  • by loomer on 7/31/2017, 5:53 PM

    Anyone have any knowledge of what specifically happened?

  • by AlphaWeaver on 7/31/2017, 8:21 PM

    I saw a comment earlier mentioning that GitHub allegedly doesn't release post mortems publicly? If this is true, that's upsetting.

  • by yellowapple on 7/31/2017, 5:46 PM

    My apologies. I knew my Perl 6 wrapper for GLFW was bad, but never realized it'd be so bad that GitHub would choke to death on it.

  • by ibgib on 7/31/2017, 4:48 PM

    Are there any other major sites that are down?

  • by vishesh92 on 7/31/2017, 4:33 PM

    It just became a major service outage.

  • by DevKoala on 7/31/2017, 7:16 PM

    This is happening too frequently now.

  • by Osiris on 7/31/2017, 4:51 PM

    It's starting to work again for me. I was able to approve a PR and merge it.

  • by xxkylexx on 7/31/2017, 4:33 PM

    Looks like I am still able to push to/pull from my repos without issue.

  • by on 7/31/2017, 4:33 PM

    undefined

  • by blurbleblurble on 7/31/2017, 5:32 PM

    Whatever happened to gittorrent?

  • by adam_seeliger on 7/31/2017, 5:24 PM

    Everyone calm down. GitHub is working just fine.

  • by revmoo on 7/31/2017, 4:51 PM

    Well you all know to commit your vendor libs so you can still deploy when Github is down, right? RIGHT?!

  • by nicodjimenez on 7/31/2017, 5:33 PM

    Pretty shocking how accustomed we've all become to Github outages. Mission critical developer tools should have gratuitously high uptime. Github is a 100M company whose most important operational objective is just keeping the site running. I call shenanigans.

  • by on 7/31/2017, 6:07 PM

    undefined

  • by tevonsb on 7/31/2017, 4:36 PM

    Thoughts on the cause?

  • by GrumpyNl on 7/31/2017, 5:04 PM

    How does this affects all your dependencies?

  • by adtac on 7/31/2017, 4:33 PM

    >GitHub is having a minor service outage

    It's definitely not minor.

  • by moomin on 7/31/2017, 5:03 PM

    I knew I shouldn't have released the new version of my project yesterday. :p

    Sorry everyone

  • by ProAm on 7/31/2017, 4:44 PM

    Githubs uptime is pretty bad. Isn't it under 95% for the year now?

  • by __s on 7/31/2017, 4:52 PM

    In the face of a lack of information, HN comments begin to throw around unfounded speculation & tongue-in-cheek jokes run rampant. I suppose that in the absence of information, many stay silent, & the remaining see a thread lacking comments

    & now we've got this meta one in the mix

  • by on 7/31/2017, 4:44 PM

    undefined

  • by 0xbear on 7/31/2017, 4:44 PM

    How many more can we expect before they develop appreciation for testing _before_ they push to prod?