How to recover a destroyed dataset on a ZFS pool

If you’re reading this, you are probably dying from anxiety after realizing your mistake so let’s begin immediately with a procedure to recover the dataset you destroyed on your ZFS pool (using Ubuntu 20.04 and assuming it is still possible):

Step 1: get the transaction group number from your pool log with the command:

zpool history -il zvol5f  

zvol5f is the ZFS pool name for this example. You should get something like this:

2022-02-05.16:33:43 [txg:4934] create zvol5f/media (424) [on home-server02]
2022-02-05.16:33:48 ioctl create
input:
type: 2
props:
[user 0 (root) on home-server02:linux]
2022-02-05.16:33:48 zfs create zvol5f/media [user 0 (root) on home-server02:linux]
2022-02-05.16:42:59 [txg:5171] destroy zvol5f/media (424) [on home-server02]
2022-02-05.16:43:06 zfs destroy zvol5f/media [user 0 (root) on home-server02:linux]

From the log, notice the txg number 5171.

Step 2: unmount your ZFS pool with the command:

zpool export zvol5f

Step 3: import the pool with with the rollback to the last transaction before the destroy command. The last number on the log was 5171 so we’ll use 5171 – 1 = 5170. In this case with the command:

zpool import -T 5170 zvol5f

After step 3 is the time to get religious and pray to get you data back, or maybe some benzodiazipine if you have some… because it will take some time (actually, a long time) to do the import of the pool.

Now that you are more relaxed and waiting for that magic import you should read the rest of this post about what you’ve just done and adjust your expectations. It is critical for the success of this procedure that you stop any activity on that pool as soon as you realize your mistake. Even it the import succeeds, any activity on that pool after the destroy command can easily corrupt your data, preventing you from recovering the destroyed dataset. Because of this I already had the practice of separating data pools from pools where I’m running software, where writing on the file system happens without your control, like logs or databases. Specially the pools where the family photos are stored… it’s when you realize that you just destroyed you family memoirs that triggers that “failure is not an option” mode that eventually leads you to this article. By the way, I did get all my data back… and thanks to all the people that contributed on the internet with the information to make this possible.

In a more technical perspective, although ZFS already provides a way to recover a destroyed pool the same cannot be done for a dataset. When you destroy the ZFS pool, any activity on the pool stops and the data will remain intact in that pool devices after the destroy command. Because of this you can recover a destroyed ZFS pool with the command:

    zpool import -D zvol5f

The devices on the pool are just marked as available to be used again in a new pool. However with a dataset, after the destroy command, the space used by the dataset is recycled and therefore there’s no way to ensure that the previous state can be restored since the activity on the pool continues. The procedure described here only works because of the ZFS transaction log. The ZFS transaction log models state changes in the pool and it’s used to ensure is consistency. This allows you go back to a previous state of the file system using the log, although that doesn’t ensure that you can get your data back. Going back to a previous state will allow you the get your old references to the data, directories, the file names, etc, but that doesn’t mean that the data is still there. It might have been already overwritten after the dataset destroy command. That’s why you have to stop immediately all activity on the pool to have any chance of recovering your data. Check this post to learn more on transaction groups and ZFS file system recovery: https://www.klennet.com/zfs-recovery/output.aspx

There was a reason for ZFS developers to use such a aggressive word as destroy. Think three or four times before using it… don’t go scrolling back on your command history and hit Enter on the first zfs destroy command that pops up…

Leave a Reply