Cannot get IIS ipsecurity working when behind a AWS Elastic load balancer

Question

I have set up IP restrictions (installed on IIS8, set feature delegation to read/write) and IIS appears to be reading from my web.config OK, blocking all IP addresses but mine when I hit the server directly, and not through the load balancer

When I do go through the load balancer it doesn't seem to respect the client IP.

So even though my IP is in the allowed IP list and I have selected 'enable proxy mode', I am still being blocked.

I have debugged the headers and the x-forwarded-for IP is coming through as the specified allowed IP. If I reverse it all, set everything to allow and then deny my specific IP with the proxy checkbox set, it works, I just can't get it to deny all and allow my IP.

Here is my config:

 <security>
      <ipSecurity allowUnlisted="false" enableProxyMode="true">
        <add ipAddress="80.200.100.40" allowed="true"/>
      </ipSecurity>
    </security>