bhyve, zvol, zfs, and importing zpool from host

I’ve been having problem with importing zpool of a vm which was using a zvol as its disk image.

I don’t know the exact detail but the only way I can import it so far is by dd-ing the image to a file and then attaching it as md device as such:

# mdconfig -a -S 512 -f /mnt/vm/somevm/disk0.img
md0

And then the pool can be normally imported from the md0 device.

Of course, had the vm been created as image file on zfs filesystem, the dd step can be skipped and just md the file right away.

Conclusion?

Just use plain image file for vm disk. Either that or use ufs for the vm filesystem instead (ew).

Other benefit of using image file includes less noisy gpart output.

IPoE, but static IPv4

Continuing from previous post, at the end I mentioned about using Vultr to avoid paying extra for static IPv4 address through my ISP.

Well, there has been a different problem with IPv4 connection crapping out every now and then so I ended up getting that ISP static IP option hoping it will lessen the problem. No comment on that yet because it’s only been less than 12 hours since I got it set up.

So, the setup itself, because I’m not using one of the supported routers, I had to figure it out myself.

The ISP provides a few needed information for the setup:

  • Static IP Tunnel Endpoint: an IPv6 address to connect for IPv4 connectivity
  • Interface ID: IPv6 address suffix (last 4 group)
    • FreeBSD doesn’t support it (it’s ip-token in Linux) but it really is just for address suffix. Mine’s ::feed so my expected address is 2409:11:1c0:2300::feed. I have it set as external IP address
  • Static IPv4 Address: this is to be set at tunnel interface as source address.
    • There’s no IPv4 target address provided which is required for FreeBSD’s gif interface but apparently any address works. I put in 10.0.0.0
    • This blog says to use source as target as well but apparently it results in packet being forwarded back and forth indicated by 14ms ping to the source IP
  • “Update Server Details”: I have no clue what this actually does
    • It’s a set of URL, username, and password where you’re supposed to make a request to to update… something. The form is simple, just $URL?username=$USERNAME&password=$PASSWORD. The URL uses internal domain so the DNS server from IPv6 autoconfiguration is required to resolve it
    • I just hit it with curl and the move on
    • I suspect it’s to tell the tunnel provider the expected source IPv6 address?

Geared with information above, there are a few changes needed since last post for setup on FreeBSD:

  • IP address on internet port should be suffixed with provided interface ID
  • Tunnel source and target address need to be adjusted
  • Tunnel interface need IPv4 address
  • Default routing for IPv4 is no longer on interface level (-iface gif) but instead the random IPv4 address used as tunnel target address (10.0.0.0 in my example above)
  • NAT is not automatically available anymore so PF is required
  • Also on NAT, MSS will need to be fixed as well
    • I still don’t really understand how this works

Most of the changes should be obvious. And here’s the config for PF:

# This is pf.conf for FreeBSD and won't work on OpenBSD

# variable to not hardcode interface names and stuff
ext_if = gif0
net_local = "192.168.0.0/24"

# I still don't know if this is needed. Or even what the correct value is.
scrub on $ext_if max-mss 1420

# basic nat
nat on $ext_if from $net_local -> ($ext_if)

IPoE, FreeBSD, and Realtek

Update: made it work again with Realtek (see update at the end).

Also: DS-Lite, Japanese ISP, ND proxy, and static IPv6.

After upgrading my server to FreeBSD 13, my ethernet failed to obtain autoconfigured IPv6 address which was weird. It’s been kinda weird before occasionally not receiving address manually after reboot but at least it works if I let it autoconfigure during boot.

Thanks to the fact the IPv6 works pretty much plug and play and usable on multiple systems just with switch, I booted up another FreeBSD 13 system hoping to find out if it’s some broken configuration on FreeBSD 13 on the server or something else.

The result was test system got its IPv6 autoconfigured, even manually with rtsol. Also weird was the main server got its address autoconfigured as well.

While at it, I wondered if I can just use static IP so the overall configuration can be simplified. And the answer was yes: it just works as long I enter the detail manually. I’ve been entering them mostly manually anyway so this was good news.

Good news it was, until I tried it on the main server itself: it worked when the modem and server are bridged by another switch but not when connected directly. It just didn’t work. Swapping back to the switch made it work again.

Back to testing, I tried direct connection to test server, and interestingly enough it worked right away. It also survived reboot, disconnect/reconnect, reconfiguration, etc.

At that point I pointed down it to the possibility of Fast Ethernet mode (100Base-TX) of Realtek just being weird and whipped out my old trusty USB ethernet dongle. And it just worked. Good job, Realtek.

So, yeah, something is broken with Realtek but I don’t care enough to dig deeper so dongle life it is.

As an extra, here’s my configuration, complete with ND proxy so the main server can distribute IPv6 address to other clients at home without having to bridge the modem directly (which gives horrible result of unwanted DNS suffix especially on Windows).

ISP is Interlink and using DS-Lite tunnel (“Multifeed” for this ISP) for IPv4 access.

### BEGIN /etc/rc.conf
# ue0 = internet port (connected to modem)
# em0 = internal port (connected to home switch)

# Basic static IPv6 configuration
ifconfig_ue0=up
# promisc option is probably set by ndproxy and not needed to be explicitly set here but I haven't tested it
# prefixlen 128 so no routing added for this port while keeping the requirement for internet port
# the address prefix and default route can be obtained when using autoconfiguration
ifconfig_ue0_ipv6="inet6 2409:11:1c0:2300:: prefixlen 128 promisc"
ipv6_defaultrouter="fe80::21e:13ff:fec2:e9c5%ue0"

# DS-Lite tunnel
cloned_interfaces=gif0
# target address can be obtained by searching the internet (multifeed) or just ISP documentation
# MTU is from experiment: raise MTU and ping around until it times out (and then add 28 bytes header)
# example: ping -s 1432 -D answers.microsoft.com
# and then try 1434 (with MTU 1500)
ifconfig_gif0="inet6 tunnel 2409:11:1c0:2300:: 2404:8e00::feed:100 prefixlen 128 mtu 1460"
defaultrouter="-iface gif0"

# nd proxy. Don't forget to install the package first: pkg install ndproxy
ndproxy_enable=yes
# interface that connects to the uplink (internet)
ndproxy_uplink_interface=ue0
# mac address of the interface above. Or maybe random address could also work. Not sure
ndproxy_downlink_mac_address="00:22:cf:xx:xx:xx"
# same as defaultrouter above but without interface name
ndproxy_uplink_ipv6_addresses="fe80::21e:13ff:fec2:e9c5"

# internal connection (with local IPv4 for NAT)
ifconfig_em0="10.0.0.1/24"
# same prefix as external interface but prefix 64
ifconfig_em0_ipv6="inet6 2409:11:1c0:2300::1 prefixlen 64 -accept_rtadv"

# for distributing ipv6 addresses. No configuration needed
rtadvd_enable=yes
rtadvd_interfaces=em0

# not sure which of the following are actually needed
ipv6_activate_all_interfaces=yes
# pretty sure at least corresponding forwarding sysctl are needed to be set if those two lines are not enabled
ipv6_gateway_enable=yes
gateway_enable=yes

### END /etc/rc.conf

Interestingly NAT doesn’t need to be manually configured: the DS-Lite tunnel magically handles it. I also keep forgetting about this and confused by the lack of NAT setting in my pf.conf.

Note that the outgoing address 2409:11:1c0:2300:: isn’t reachable from internal network with this configuration. Use 2409:11:1c0:2300::1 instead, including for external access (like this blog).

I should also write up my Wireguard-based external IPv4 one of these days… (because I’m too cheap to pay for Interlink’s static IPv4 – Vultr additional IP for 220yen vs Interlink IPoE static IPv4 for 1100yen).

Update 2021-05-18: I installed Realtek driver (realtek-kmod package) and it works. I previously had to use it as the driver was missing in FreeBSD 12 but switched to the updated built-in driver when upgrading to 13. Tried again with the driver and it works in 13.

FreeBSD pkg (manual) upgrade thingy

For working with locked packages (the ones which must be upgraded through compilation because of using custom options)

#!/bin/sh
# filename: pkg-lock-outdated

pkg query -e '%k = 1' %o | while read pkgorig; do
  pkg version -ovL => -O "$pkgorig"
done

The script above is to list locked packages which need upgrading. And to upgrade everything at once (and sit in front of PC waiting for whole process)

#!/bin/sh

listfile="/tmp/pkgforupgrade.$(date '+%Y%m%d%H%M%S')"
pkg-lock-outdated | cut -f 1 -d '<' > "$listfile"

while read <&3 outdated; do
  pkg unlock "$outdated"
  portmaster "$outdated"
  pkg lock "$outdated"
done 3< "$listfile"

rm -f "$listfile"

There’s another alternative of unlocking all packages at once, run batched portmaster, and lock them all again.

PostgreSQL authentication quick start

Connecting to PostgreSQL from command line can be a bit confusing.

For starter, just like MySQL, psql command defaults to connecting to socket instead of tcp. To make matter confusing, most PostgreSQL installation defaults to ident (also called peer)authentication for socket connection: it basically matches current user’s username (ssh login, etc) with PostgreSQL equivalent.

So, instead of using this to login from root to PostgreSQL superuser (usually named postgres or pgsql):

# psql -U postgres

you do this (assuming sudo installed):

# sudo -u postgres psql

The configuration for this is located in pg_hba.conf of PostgreSQL data (or config in Debian) directory (/etc/postgresql/$version/main in Debian, /usr/local/pgsql/data in FreeBSD, /opt/PostgreSQL/$version/data in EnterpriseDB PostgreSQL).

To switch to password based authentication for all methods just replace ident (or peer) with md5 in respective lines and reload/restart the service. Don’t forget to set password for postgres user first before changing this otherwise you won’t be able to connect. You can then connect using psql to any user using password.

FreeBSD is Rolling Release (the ports)

Don’t get tricked by the “release” system. Apart of the base system, FreeBSD perfectly qualifies as rolling release. I guess it’s also why the binary package management sucked so badly. You won’t find how to upgrade certain packages using binary method in their Ports’ UPDATING page.

Here’s the example:

20120225:
  AFFECTS: users of archivers/libarchive
  AUTHOR: glewis@FreeBSD.org

  libarchive has been updated to version 3.0.3, with a shared library bump.
  This requires dependent ports to be rebuilt.

  # portmaster -r libarchive
  or
  # portupgrade -r archivers/libarchive

You would think the dependent packages got version bump to ensure their proper dependency – but they didn’t. Instead you had to recompile everything depending on it.

And then there’s another case:

20120220:
  AFFECTS: users of graphics/libungif
  AUTHOR: dinoex@FreeBSD.org

  libungif is obsolete, please deinstall it and rebuild all ports using
  it with graphics/giflib.

  # portmaster -o graphics/giflib graphics/libungif
  # portmaster -r giflib
  or
  # portupgrade -o graphics/giflib graphics/libungif
  # portupgrade -rf giflib

Of course, ArchLinux kind of managed to do it but that’s a purely binary rolling release Linux distro. The maintainer worked hard to ensure such kind of thing get handled properly by all their users which mostly use binary packages. FreeBSD on other hand tried to claim capable of both but it really isn’t (unless I missed something).

I’m intending to contact pkgng creator to ask his opinion about this but have yet to do it…

OpenBSD 5.0

This blog is now running on OpenBSD 5.0. Too bad php-fpm didn’t get to 5.0.

OpenBSD tomoka.myconan.net 5.0 GENERIC.MP#59 i386

The upgrade process went without any problems. Upgrading packages also went relatively well apart of php being a failure because of change to infrastructure (which allowed multiple versions to be installed). Otherwise everything upgraded without hitch and finished quickly. Sure is nice depart from FreeBSD’s ports which takes hours to update a package (upgrading system is relatively quick though using freebsd-update).

Unless there’s critical security vulnerability or something happened to the datacenter, I expect there will be no reboot until next upgrade (6 month uptime). We will see.

ZFS on Desktop – Which OS?

I’ve been thinking about this lately, especially after seeing this:

[root@einhart ~]# pkg_version -vIL=>
binutils-2.21 < needs updating (index has 2.21.1)
bitstream-vera-1.10_4 < needs updating (index has 1.10_5)
chromium-12.0.742.112 < needs updating (index has 12.0.742.124)
droid-fonts-ttf-20100214_1 < needs updating (index has 20110324)
exiv2-0.21,1 < needs updating (index has 0.21.1,1)
gawk-3.1.8 < needs updating (index has 4.0.0)
gcc-4.6.2.20110701 < needs updating (index has 4.6.2.20110708)
gnutls-2.12.7_1 < needs updating (index has 2.12.7_2)
libidn-1.19 < needs updating (index has 1.22)
openjdk6-b22_6 < needs updating (index has b23)
p5-Net-DBus-0.33.6 < needs updating (index has 1.0.0)
p5-libwww-5.837 Fetching ‘/usr/ports/www/chromium’

To build Chromium, you should have around 1 GB of memory
and a fair amount of free diskspace (~ 1.5GB).

===> Vulnerability check disabled, database not found
===> License BSD LGPL21 MPL accepted by the user
===> Found saved configuration for chromium-12.0.742.112
=> chromium-courgette-redacted-12.0.742.124.tar.xz doesn’t seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch http://download.goodking.org/downloads/chromium-courgette-redacted-12.0.742.124.tar.xz
chromium-courgette-redacted-12.0.742.124.tar.x 0% of 114 MB 35 kBps^C

And did you notice that “you should have around 1 GB of memory and a fair amount of free diskspace”? Yeah, it does use that much memory to update a web browser. Something that takes few seconds on Windows and Fedora and few minutes on Ubuntu takes almost hours to do on FreeBSD. Downloading source notwithstanding, compiling it also takes long, long time.

If anything, this is why I hate FreeBSD. And you will mention use package except that there’s no H.264 video support and possible chance of conflicting package difference between my system and the buildbot. As much as I want to use binary packages, they come in unwanted forms:

* GNOME 2 pulls Samba4 for God knows why reason
* Some pacakges don’t have optimization enabled
* Mplayer doesn’t have VDPAU
* Some packages must be installed manually (Java, Opera)
* Portupgrade takes minutes to compute what to do (something which is done in seconds in Ubuntu)

As much as I can tolerate this shit, the very fact that VirtualBox under FreeBSD takes much more CPU certainly doesn’t help.

So, I guess I’ll try yet another OS. There are several options so here are overview of current choices.

FreeBSD
——-
Pros:

* Extremely simple
* Up to date packages
* Highly customizable
* Stable ZFS

Cons:

* Customization comes at cost: compile time
* No good virtualization option
* A package management that is comparable with slowpoke
* No flash (yes there’s linux flashplugin ports but it’s a joke)
* Awesomely slow video playback even optimized on Firefox

PC-BSD
——
Pros:

* Based on FreeBSD, has some of familiarity
* Binary packages that’s not too slow
* Stable ZFS

Cons:

* Also no good virtualization
* Epic size of packages (eg. [a hundred megabytes for Firefox](http://www.pbidir.com/bt/pbi/49/firefox))

OpenIndiana
———–
Pros:

* Stable, greatest ZFS
* Zones
* Good VirtualBox support
* Flash support

Cons:

* Complex
* Doesn’t survive trivial hardware change without Live CD/USB
* ACL
* Outdated packages
* Non-existent packages
* Manual compiling of `mplayer` is a must since the defaults on various repositories are crap
* Still beta
* Doesn’t recognize FreeBSD’s GPT (at least on `oi_148`)
* Ugly freetype2 (no support for subpixel hinting apart of manual compile)

Ubuntu Linux
————
Pros:

* Latest, greatest software
* Good software support
* Good virtualization

Cons:

* Unstable ZFS
* Linux

Note that I only include Ubuntu in Linux category since it’s the only OS having both working nvidia driver and precompiled ZFS module, saving me lots of headaches.

FreeBSD: 0, NetBSD: 1, OpenBSD: 1

Sil3112 (SiliconImage SATALink RAID PCI Card) is not properly supported by FreeBSD. The installer halted successfully on all my three attempt at installing it. (First one even stopped when formatting – said that it can’t find the hard disk drive LOL).

OTOH, NetBSD and OpenBSD can use that piece of hardware just fine albeit the claim of the card as “crappy” on certain mailing lists.