IPoE, but static IPv4

Continuing from previous post, at the end I mentioned about using Vultr to avoid paying extra for static IPv4 address through my ISP.

Well, there has been a different problem with IPv4 connection crapping out every now and then so I ended up getting that ISP static IP option hoping it will lessen the problem. No comment on that yet because it’s only been less than 12 hours since I got it set up.

So, the setup itself, because I’m not using one of the supported routers, I had to figure it out myself.

The ISP provides a few needed information for the setup:

  • Static IP Tunnel Endpoint: an IPv6 address to connect for IPv4 connectivity
  • Interface ID: IPv6 address suffix (last 4 group)
    • FreeBSD doesn’t support it (it’s ip-token in Linux) but it really is just for address suffix. Mine’s ::feed so my expected address is 2409:11:1c0:2300::feed. I have it set as external IP address
  • Static IPv4 Address: this is to be set at tunnel interface as source address.
    • There’s no IPv4 target address provided which is required for FreeBSD’s gif interface but apparently any address works. I put in 10.0.0.0
    • This blog says to use source as target as well but apparently it results in packet being forwarded back and forth indicated by 14ms ping to the source IP
  • “Update Server Details”: I have no clue what this actually does
    • It’s a set of URL, username, and password where you’re supposed to make a request to to update… something. The form is simple, just $URL?username=$USERNAME&password=$PASSWORD. The URL uses internal domain so the DNS server from IPv6 autoconfiguration is required to resolve it
    • I just hit it with curl and the move on
    • I suspect it’s to tell the tunnel provider the expected source IPv6 address?

Geared with information above, there are a few changes needed since last post for setup on FreeBSD:

  • IP address on internet port should be suffixed with provided interface ID
  • Tunnel source and target address need to be adjusted
  • Tunnel interface need IPv4 address
  • Default routing for IPv4 is no longer on interface level (-iface gif) but instead the random IPv4 address used as tunnel target address (10.0.0.0 in my example above)
  • NAT is not automatically available anymore so PF is required
  • Also on NAT, MSS will need to be fixed as well
    • I still don’t really understand how this works

Most of the changes should be obvious. And here’s the config for PF:

# This is pf.conf for FreeBSD and won't work on OpenBSD

# variable to not hardcode interface names and stuff
ext_if = gif0
net_local = "192.168.0.0/24"

# I still don't know if this is needed. Or even what the correct value is.
scrub on $ext_if max-mss 1420

# basic nat
nat on $ext_if from $net_local -> ($ext_if)

IPoE, FreeBSD, and Realtek

Update: made it work again with Realtek (see update at the end).

Also: DS-Lite, Japanese ISP, ND proxy, and static IPv6.

After upgrading my server to FreeBSD 13, my ethernet failed to obtain autoconfigured IPv6 address which was weird. It’s been kinda weird before occasionally not receiving address manually after reboot but at least it works if I let it autoconfigure during boot.

Thanks to the fact the IPv6 works pretty much plug and play and usable on multiple systems just with switch, I booted up another FreeBSD 13 system hoping to find out if it’s some broken configuration on FreeBSD 13 on the server or something else.

The result was test system got its IPv6 autoconfigured, even manually with rtsol. Also weird was the main server got its address autoconfigured as well.

While at it, I wondered if I can just use static IP so the overall configuration can be simplified. And the answer was yes: it just works as long I enter the detail manually. I’ve been entering them mostly manually anyway so this was good news.

Good news it was, until I tried it on the main server itself: it worked when the modem and server are bridged by another switch but not when connected directly. It just didn’t work. Swapping back to the switch made it work again.

Back to testing, I tried direct connection to test server, and interestingly enough it worked right away. It also survived reboot, disconnect/reconnect, reconfiguration, etc.

At that point I pointed down it to the possibility of Fast Ethernet mode (100Base-TX) of Realtek just being weird and whipped out my old trusty USB ethernet dongle. And it just worked. Good job, Realtek.

So, yeah, something is broken with Realtek but I don’t care enough to dig deeper so dongle life it is.

As an extra, here’s my configuration, complete with ND proxy so the main server can distribute IPv6 address to other clients at home without having to bridge the modem directly (which gives horrible result of unwanted DNS suffix especially on Windows).

ISP is Interlink and using DS-Lite tunnel (“Multifeed” for this ISP) for IPv4 access.

### BEGIN /etc/rc.conf
# ue0 = internet port (connected to modem)
# em0 = internal port (connected to home switch)

# Basic static IPv6 configuration
ifconfig_ue0=up
# promisc option is probably set by ndproxy and not needed to be explicitly set here but I haven't tested it
# prefixlen 128 so no routing added for this port while keeping the requirement for internet port
# the address prefix and default route can be obtained when using autoconfiguration
ifconfig_ue0_ipv6="inet6 2409:11:1c0:2300:: prefixlen 128 promisc"
ipv6_defaultrouter="fe80::21e:13ff:fec2:e9c5%ue0"

# DS-Lite tunnel
cloned_interfaces=gif0
# target address can be obtained by searching the internet (multifeed) or just ISP documentation
# MTU is from experiment: raise MTU and ping around until it times out (and then add 28 bytes header)
# example: ping -s 1432 -D answers.microsoft.com
# and then try 1434 (with MTU 1500)
ifconfig_gif0="inet6 tunnel 2409:11:1c0:2300:: 2404:8e00::feed:100 prefixlen 128 mtu 1460"
defaultrouter="-iface gif0"

# nd proxy. Don't forget to install the package first: pkg install ndproxy
ndproxy_enable=yes
# interface that connects to the uplink (internet)
ndproxy_uplink_interface=ue0
# mac address of the interface above. Or maybe random address could also work. Not sure
ndproxy_downlink_mac_address="00:22:cf:xx:xx:xx"
# same as defaultrouter above but without interface name
ndproxy_uplink_ipv6_addresses="fe80::21e:13ff:fec2:e9c5"

# internal connection (with local IPv4 for NAT)
ifconfig_em0="10.0.0.1/24"
# same prefix as external interface but prefix 64
ifconfig_em0_ipv6="inet6 2409:11:1c0:2300::1 prefixlen 64 -accept_rtadv"

# for distributing ipv6 addresses. No configuration needed
rtadvd_enable=yes
rtadvd_interfaces=em0

# not sure which of the following are actually needed
ipv6_activate_all_interfaces=yes
# pretty sure at least corresponding forwarding sysctl are needed to be set if those two lines are not enabled
ipv6_gateway_enable=yes
gateway_enable=yes

### END /etc/rc.conf

Interestingly NAT doesn’t need to be manually configured: the DS-Lite tunnel magically handles it. I also keep forgetting about this and confused by the lack of NAT setting in my pf.conf.

Note that the outgoing address 2409:11:1c0:2300:: isn’t reachable from internal network with this configuration. Use 2409:11:1c0:2300::1 instead, including for external access (like this blog).

I should also write up my Wireguard-based external IPv4 one of these days… (because I’m too cheap to pay for Interlink’s static IPv4 – Vultr additional IP for 220yen vs Interlink IPoE static IPv4 for 1100yen).

Update 2021-05-18: I installed Realtek driver (realtek-kmod package) and it works. I previously had to use it as the driver was missing in FreeBSD 12 but switched to the updated built-in driver when upgrading to 13. Tried again with the driver and it works in 13.