DNSSEC chain validation issue for .nz

Incident Report for InternetNZ

Postmortem

Following the DNSSEC chain validation issue for .nz domains on 29-31 May 2023, we have now published the independent review on the InternetNZ website..

‌

The InternetNZ Council commissioned this independent review to examine the events leading up to the incident and our response and to make recommendations to prevent similar failures in the future. The report affirms our internal technical analysis of why the incident happened, and it will be discussed at the InternetNZ Council meeting this Friday, 13th September 2023.

‌

Please reach out to office@internetnz.net.nz if you have any questions regarding the report.

Posted Oct 09, 2023 - 12:44 NZDT

Resolved

No further issues have been encountered during the monitoring period.

We have produced a technical incident report which can be found on our website

Posted Jun 28, 2023 - 09:15 NZST

Update

We're pleased to announce that all manual validations were completed successfully, and regular zone file transfers resumed from 18:30, 1 June NZST. This completes the current key rollover process, and we have returned to normal operational mode.

We continue to monitor for any unexpected behaviour and are available via standard escalation channels.

Posted Jun 01, 2023 - 19:59 NZST

Update

We're manually validating our final DNSSEC updates for .nz zone today, necessitating a temporary pause of zone pushes from 3:30 pm to 5:30 pm NZST.

Following the recent incident, we temporarily halted the DNSSEC key rollover process. This increased the size of our DNS responses for a sustained period. We're now completing the key rollover process and removing outdated keys to minimise potential technical issues.

We'll continue monitoring and provide updates. Your patience is appreciated.

Posted Jun 01, 2023 - 14:54 NZST

Update

As of 10:45 pm NZST on 31 May 2023, we have high confidence that the last mismatched records from the KSK rollover event have expired. All internet users should be able to access all .nz domains, including .nz, .co.nz, .ac.nz, .geek.nz, .gen.nz, .kiwi.nz, .maori.nz, .net.nz, .org.nz, .school.nz, .cri.nz, govt.nz, .health.nz, .iwi.nz, .mil.nz, and .parliament.nz without any disruption.

However, we want to remind all operators that if a validating recursive server is still not resolving any of these domains and is returning SERVFAIL, please flush the validating recursive server’s cache for the relevant zone to restore normal functionality.

We will continue to monitor the situation for any further issues closely. Thank you for your patience during this time.

Posted Jun 01, 2023 - 11:13 NZST

Update

We are continuing to monitor for any further issues.

We have issued a news article on our website

Posted May 30, 2023 - 10:57 NZST

Update

We have identified a further issue with the same KSK rollover event that affected .ac.nz, we have expanded the scope to all .nz domains including .nz, .co.nz, .geek.nz, .gen.nz, .kiwi.nz, .maori.nz, .net.nz, .org.nz, .school.nz, .cri.nz, govt.nz, .health.nz, .iwi.nz, .mil.nz and .parliament.nz, starting at 2023-05-29 10:45 PM NZST

This issue will resolve over time, however if a recursive server is still not resolving any these domains, and returning SERVFAIL, the validating recursive server cache can be flushed for the relevant zone to pick up the new DNSSEC records and return it to a functioning state.

We recommend preemptively flushing all .nz zones before the issue occurs.

Posted May 30, 2023 - 09:21 NZST

Update

We are continuing to monitor for any further issues.

Posted May 29, 2023 - 17:20 NZST

Monitoring

We have identified an issue affecting the signing of ac.nz during a KSK rollover at 13:00 NZST, if you encounter issues with validation of ac.nz please flush any recursive server cache for ac.nz.

Posted May 29, 2023 - 16:59 NZST

This incident affected: .nz DNS Network (.nz DNSSEC).