I think you may be running into the max flash read speed. The flash can only read at 40 Mhz. We have a pre-fetch buffer mechanism to fetch 2 words instead of one when running faster than 40 mhz. In a tight loop like the SysCtlDelay you may find that the the pre-fetch cannot do it's job well due to the jumping in the loop.
Some things to try that might verify this is true.
Change SYSCTL_SYSDIV_2_5 to SYSCTL_SYSDIV_5. See if the delay is exactly twice as long or if you are hitting the read time limit then the delay will be the same.
A second test would be to try using ROM_SysCtlDelay() with the rest of your software the same as shown. The ROM can run at the full 80Mhz and will not be slowed down.