Problem
We just recently added an additional node to our existing Availability
Group cluster.
Our team uses a PowerShell script to easily format multiple drives in a specific allocation unit size.
Our team uses a PowerShell script to easily format multiple drives in a specific allocation unit size.
The issue happened when we formatted the drive on the new node. It is supposed to only format the drive on the server where it was ran. But for some reason, it also formatted the drive on the existing nodes.
This wiped out the drives on the existing nodes. The existing system went down and we have to restore everything from our backup.
We have to do a RCA (Root Cause Analysis) because it affected our
production system.
Root Cause Analysis
We have to build a similar environment to simulate the issue.
To cut the story short, we were able to pin point the issue on Format-Volume command in PowerShell.
This particular issue only affects servers that are part of Windows cluster and have same drive letters on each node.
For SQL Server Always On:
- This does NOT affect Failover Cluster Instance (FCI) since it is using shared storage.
- This only affects the Availability Group configuration because each node have it's own storage and they can have same drive letters.
As a test we ran the Format-Volume to format the Z drive on node1 (of a 3
node cluster).
It formatted the Z drive on of all the nodes instead of just formatting the
drive on node1.
Solution
Open a case with Microsoft to check if this is a bug or if there is a
different method that we can do to run the Format-Volume only on the node
where it is executed. (At the time of this writing, we have not heard back
from Microsoft.)
As a precautionary measure for our team, we only use this feature on the following situation:
- Standalone servers
- Always On Failover Cluster Instance servers
- New server builds of Always On Availability Group
