'Completely unprecedented' outage causes havoc with IT systems across globe — as it happened
www.abc.net.au
A massive IT systems outage caused by issues with CrowdStrike software affects banks, airports, supermarkets and media companies across Australia and around the world.

All our servers and company laptops went down at pretty much the same time. Laptops have been bootlooping to blue screen of death. It’s all very exciting, personally, as someone not responsible for fixing it.

Apparently caused by a bad CrowdStrike update.

Edit: now being told we (who almost all generally work from home) need to come into the office Monday as they can only apply the fix in-person. We’ll see if that changes over the weekend…

    • rozodru@lemmy.caEnglish
      782·
      4 months ago

      yeah someone fucked up here. I mean I know you’re joking but I’ve been in tech for like 20+ years at this point and it was always, always, ALWAYS, drilled into me to never do updates on Friday, never roll anything out to production on Friday. Fridays were generally meant for code reviews, refactoring in test, work on personal projects, raid the company fridge for beer, play CS at the office, whatever just don’t push anything live or update anything.

      And especially now the work week has slimmed down where no one works on Friday anymore so you 100% don’t roll anything out, hell it’s getting to the point now where you just don’t roll anything out on a Thursday afternoon.

          • jedibob5@lemmy.worldEnglish
            2·
            4 months ago

            Is the 4x10 really worth the extra day off? Tbh I’m not sure it would work very well for me… I find just one 10-hour day to be kinda draining, so doing that 4 times a week every week feels like it might just cancel out any benefits of the extra day off.

            • meanmon13@lemmy.zipEnglish
              2·
              4 months ago

              I am very used to it so I don’t find it draining. I tried 5x8 once and it felt more like working an extra day than getting more time in the afternoon. If that makes sense. I also start early around 7am, so I am only staying a little later than other people

        • rozodru@lemmy.caEnglish
          6·
          4 months ago

          sorry :( yeah I, at most, do 3 days in the office now. Fridays are a day off and Mondays mostly everyone just works from home if at all. downtown Toronto on Mondays and Fridays is pretty much dead.

          • corsicanguppy@lemmy.caEnglish
            2·
            4 months ago

            I changed jobs because the new management was all “if I can’t look at your ass you don’t work here” and I agreed.

            I now work remotely 100% and it’s in the union contract with the 21vacation days and 9x9 compressed time and regular raises. The view out my home office window is partially obscured by a floofy cat and we both like it that way.

            I’d work here until I die.

      • Blackmist@feddit.ukEnglish
        15·
        4 months ago

        Yep, anything done on Friday can enter the world on a Monday.

        I don’t really have any plans most weekends, but I sure as shit don’t plan on spending it fixing Friday’s fuckups.

        • sugar_in_your_tea@sh.itjust.worksEnglish
          3·
          4 months ago

          And honestly, anything that can be done Monday is probably better done on Tuesday. Why start off your week by screwing stuff up?

          We have a team policy to never do externally facing updates on Fridays, and we generally avoid Mondays as well unless it’s urgent. Here’s roughly what each day is for:

          • Monday - urgent patches that were ready on Friday; everyone WFH
          • Tuesday - most releases; work in-office
          • Wed - fixing stuff we broke on Tuesday/planning the next release; work in-office
          • Thu - fixing stuff we broke on Tuesday, closing things out for the week; WFH
          • Fri - documentation, reviews, etc; WFH

          If things go sideways, we come in on Thu to straighten it out, but that almost never happens.

      • sasquash@sopuli.xyzEnglish
        2·
        4 months ago

        Actually I was not even joking. I also work in IT and have exactly the same opinion. Friday is for easy stuff!

    • merc@sh.itjust.worksEnglish
      3·
      4 months ago

      You posted this 14 hours ago, which would have made it 4:30 am in Austin, Texas where Cloudstrike is based. You may have felt the effect on Friday, but it’s extremely likely that the person who made the change did it late on a Thursday.

      • Toribor@corndog.socialEnglish
        58·
        4 months ago

        This is fine as long as you politely ask everyone on the Internet to slow down and stop exploiting new vulnerabilities.

        • Ookami38@sh.itjust.worksEnglish
          22·
          4 months ago

          I think vulnerabilities found count as “something broken” and chap you replied to simply did not think that far ahead hahah

          • huginn@feddit.itEnglish
            14·
            4 months ago

            For real - A cyber security company should basically always be pushing out updates.

            • sugar_in_your_tea@sh.itjust.worksEnglish
              4·
              4 months ago

              Exactly. You don’t know what the vulnerabilities are, but the vendors pushing out updates typically do. So stay on top of updates to limit the attack surface.

              Major releases can wait, security updates should be pushed as soon as they can be proven to not break prod.

            • wreckedcarzz@lemmy.worldEnglish
              1·
              4 months ago

              always pushing out updates

              Notes: Version bump: Eric is a twat so I removed his name from the listed coder team members on the about window.

              git push --force

              leans back in chair productive day, productive day indeed

      • Hotzilla@sopuli.xyzEnglish
        2·
        4 months ago

        This is AV, and even possible that it is part of definitions (for example some windows file deleted as false positive). You update those daily.