2

I'm seeing what seems to me to be contradictory results when looking at disk performance with dd versus iostat on two hosts (EC2 instances with an EBS drive). The hosts are identical except that one uses EXT4-formatted ebs and the other XFS-formatted ebs.

If I look at iostat, the EXT4 host seems to outperform the XFS host. TBoth are doing roughly the same write throughput (about 25MB/s) at about 100% utilization, but the EXT4 host on average has less await (lower disk latency). It's this smaller await that makes me say EXT4 is outperforming XFS:

EXT4:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
xvdf              0.00    11.00    0.00 6331.00     0.00    26.96     8.72    71.00   11.32   0.16  99.60

XFS:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
xvdf              0.00     2.00    0.00 6211.00     0.00    27.38     9.03   144.95   23.24   0.16 100.40

But, if using dd to measure performance, XFS is the clear winner, as it takes much less time to complete a fully synchronous write. The command is dd bs=1M count=256 if=/dev/zero of=./test conv=fdatasync:

EXT4:

2.8 MB/s

XFS:

24.0 MB/s

What would be the reason for EXT4 looking much better with iostat but looking much worse with dd?

UPDATE 5/25/2018: After running the hosts for a couple days, dd and sync now show equivalent response times for both ext4 and xfs. I suspect this has to do with a difference (if any?) in the way they handle something like sparse files. The first day the hosts were up they were both busy laying down a bunch of new files on the filesystem (this is a graphite carbon-cache application). This has settled down to where small updates are being written to these files, but new files are no longer being created and the total amount of used disk space is no longer increasing.

So, there must be something fundamentally different in the way XFS allocates new disk blocks versus EXT4. Any insight into what this could be, would be welcome.

Improve this question
2
  • XFS-formatted elb -> XFS-formatted ebs
    – c4f4t0r
    May 24, 2018 at 9:41
  • yes, sorry that should be ebs not elb
    – Michael Martinez
    May 24, 2018 at 13:49

1 Answer 1

Reset to default
2

Correct me if I'm wrong, but I believe you use EBS volumes.

Try to format the ext4 partition without lazy initialization, as it can affect testing performance.

mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/xvdf

EBS Volumes

Several factors will determine EBS volume performance. They are not entirely intuitive.

  1. Type of the volume. Provisioned IOPS io1, General Purpose gp2, Throughput Optimized HDD st1, Cold HDD sc1. characteristics.

  2. Size of the volume. Generally, bigger the volume, better the performance. Except for the EBS Provisioned IOPS (io1), volumes use a burst model 1 and I/O can fluctuate or drop significantly if all I/O credits are used. In short, each volume gets base minimum 100 IOPS, and for every +1 GB added (after ~33GB) performance will increase for 3 IOPS. Also the volume can burst up to 3000 IOPS if there are enough I/O credits available.

Size vs IOPS
(source: amazon.com)

  1. EC2 Instance type. Larger the instance, faster the network performance. It's also important whether it's an EBS-Optimized instance or not.

  2. If you restored your volume from a snapshot, you'd have to pre-warm the volume to get the maximum performance.
    See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html

More details:

Update 1

It's difficult to tell why you see those results as there's just not enough information in your post for me to deduce it.

This CloudFormation template is my attempt to recreate your results.
https://gist.github.com/an2io/c68b2119f18192d83a685651905623e9

EXT4

wrqm/s  w/s     wMB/s   avgrq-sz    avgqu-sz    await   svctm    %util  
0.00    528.62  66.08   255.84      127.07      217.16  1.61     85.39    
2.69    287.21  35.86   255.71      142.20      504.73  3.52    101.01    
0.00    285.28  35.59   255.49      134.93      462.90  3.52    100.33    
3.02    277.52  34.54   254.89      115.54      460.62  3.51     97.32    
0.00      0.00   0.00     0.00        0.00        0.00  0.00      0.00    
512+0 records in  
512+0 records out  
536870912 bytes (537 MB) copied, 11.9003 s, 45.1 MB/s

XFS

wrqm/s  w/s     wMB/s   avgrq-sz    avgqu-sz    await   svctm    %util
1.68    542.28  67.44   254.07      176.84      301.59  1.66     90.47
0.00    284.95  35.62   256.00      143.74      508.38  3.52    100.33
0.00    285.28  35.58   255.46      184.36      609.81  3.52    100.33
0.00    265.10  32.95   253.27      162.34      692.59  3.48     92.75
0.00      0.00   0.00     0.00        0.00        0.00  0.00      0.00

512+0 records in  
512+0 records out  
536870912 bytes (537 MB) copied, 11.7641 s, 45.6 MB/s

df

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.9G   72K  1.9G   1% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
/dev/xvda1      9.8G  1.2G  8.5G  12% /
/dev/xvdb       3.9G  8.1M  3.7G   1% /media/ephemeral0
/dev/xvdf1      3.7G  520M  3.0G  15% /mnt/ext4
/dev/xvdf2      3.9G  545M  3.3G  14% /mnt/xfs

mount

/dev/xvdf1 on /mnt/ext4 type ext4 (rw,relatime,data=ordered)
/dev/xvdf2 on /mnt/xfs type xfs (rw,relatime,attr2,inode64,noquota)
Improve this answer
3
  • Thanks for responding. I like what you have written, but it doesn't address why there is a difference between iostat and dd. Perhaps you could rewrite your answer to address this.
    – Michael Martinez
    May 24, 2018 at 13:52
  • Andi: try running your test but this time while generating a workload, where the workload is something that creates a bunch of small files around 50K size each.
    – Michael Martinez
    May 29, 2018 at 18:32
  • for the workload, creating a bunch of 50K files with zeroes should suffice, as this is what carbon-cache does.
    – Michael Martinez
    May 29, 2018 at 22:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .