Телеграм канал «/var/log/DMIT-NOC.log»

https://telegra.ph/DMIT-August-26-Network-Security-Bulletin-08-29

DMIT Internal Temproray Self-Testing Tool
https://dmit-abuse-team-temp-security-response.dmit.com/

All impacted services have been stopped.

DMIT proactive security detected that 2% of tenants were deploying potentially risky applications in the DMIT LAX data center. These tenants' applications sent significant large traffic to China several hours ago.

- DMIT detected abnormal traffic within the network 12 hours ago but took no action due to complexity.
- However, this potential risk factor was officially exploited 8 hours ago.
- DMIT will officially notify affected customers within 12 hours and require them to resolve the issue within 24 hours to avoid service interruption for a long time and abuse penalties.
- The relevant users will receive an email soon, and the VM will be immediately shut down to avoid the transfer being used up. (DMIT will not recover this transfer quota since it was due to the client's unsafe application.)

This vulnerable application cannot directly generate such a large amount of traffic, even though it is open to the internet. Exploiting this type of vulnerability requires real data and requests; it does not possess amplification characteristics.

DMIT initially speculates that attackers in restricted areas are exploiting this TCP 443 application design flaw on a large scale to make super-large-scale real and accepted requests to OpenAI.

Further details will be officially provided after the issue is resolved.

DMIT has observed that some customers' applications running pose serious security risks and are being exploited on a large scale. This has already impacted DMIT's network security.
Investigation and mitigation in progress.

AS9929 LAX is utilizing some capacity at SJC.

Weeks ago, there was a failure of a DWDM line card on the CUA network, which resulted in a loss of capacity for LAX-SJC.

CUG informed us that this issue has been resolved and is expected to be fully operational at this time.

CN2 core CoreSite LA1: Datacenter cooling system water leak, which lead to capacity issue on CN2 backbone on LA1 site.

CN2 core Digital Realty LAX10: Backbone cable fault.

Both faults happening together made CN2 backbone has significant capacity issue.

DMIT keeps monitoring about this issue.

LAX Pro:

DMIT observed CN2 backbone issue.
We’ve minimized the packet loss by switching all traffic to one of the CN2 core.
The packet loss has dropped from 30% to 4%.

DMIT NOC also reached the CT NOC about this matter.