…one does not simply run both PHP and Rails applications (and MySQL) in single 128 MB box.
This is what happens if you try doing it. Swapping all the time. Hopefully Debian will fare better but it means I’ll need to reinstall the box again. I’ll do a test run in my local machine first though as not to waste my time doing backup/restore again like today.
(Yeah, Zeropaste is up at p.myconan.net)
Tonight, when trying to compile Rubinius with Rubinius, I got some errors and wanted to report to relevant party. As usual, the log of what happened is required so I used my usual pastebin – pastie.org to send the logs. But then I noticed that the “Raw” link in it doesn’t provide an actual raw file anymore. What the fuck. It is now a html disguised as txt.
So I decided to whip up a new pastebin (because the world need one more pastebin) which doesn’t have any actual features (like tag highlighting, etc). I also learned the way to create shortest RESTful path possible (read: ‘/’).
There may or may not be more features coming. Developed in Rubinius because I can. Using mysql because of where it’ll be deployed at (see below).
I’ll get around deploying this soon after reinstalling VPS running this blog to Ubuntu or Debian. Running yum in a 128 MB box is suffering.
[ Source Code ]
(main purpose of this post is to link this “server error” page of rubygems.org)
The relevant issue in Rails Issue Tracker (3789). AFAICT, there are few ways to “fix” (read: workaround) this:
Well, they all sucks. Hopefully someone comes up with actual working solution for this.
Oh, there’re another solutions:
Um, yeah.
Update: I figured out how to “fix” it. Check it out in Moebooru (requires this).
Be careful when using read_multi with dalli: it may return nil-valued key instead of the correct key.
The issue is tracked here and thanks to this I dropped the read_multi usage in moebooru and used the much simpler (and most likely slower) single fetch (per entry) instead. There’s alternative way to use it – do a read_multi and refetch whatever missing/nil-keyed but apparently I’m too lazy to do it.
Due to NIH syndrome and Drepper being Drepper, the only remotely secure password hashing algorithm in RHEL5/6 is multi-rounds SHA512. The default is just salted SHA512 which sucks.
Also applies to CentOS, ScientificLinux, and other RHEL clones.
Anyway, to update the default setting, these files need updating:
/etc/login.defs: add new line SHA_CRYPT_MIN_ROUNDS 5000/etc/pam.d/system-auth-ac: find line with `password sufficient pam_unix.so sha512` and append rounds=5000.Note that the change to last file may or may not be persistent. I have no idea how to properly set it up.
Finally, run this command: authconfig --updateall.
If you’re using RHEL5, run authconfig --passalgo=sha512 --update first.
Finally reached the goal: Rails 3.2.6. The isn’t much change between 3.1 and 3.2 which is why I skipped 3.1. As there isn’t much front-end change, it means there isn’t any user-visible changes. Or at least there shouldn’t be.
Plans:
3.2.0 has been branched and currently running on some servers. The next version, 3.2.1 is on default branch and being tested on moe.myconan.net. As (finally) there are database changes, it will not be tested directly on live servers until it is stable enough.
Changes include better support for JRuby, use mini_magick instead of custom plug-in, less monkey-patches (but more bugs), and various clean-ups.
Steps to be into Rails in subdirectory:
Update config.ru to understand the subdirectory mapping:
run Moebooru::Application
change to
map (ENV['RAILS_RELATIVE_URL_ROOT'] || '/') do run Moebooru::Application end
And then start Rails with correct environment variable (example if you use Unicorn):
RAILS_RELATIVE_URL_ROOT='/img' bin/unicorn
And that’s it. No need to mess with routes.rb as I previously thought after searching and experimenting for few hours. Links etc are properly generated with correct prefix. Or at least based on my quick testing.
May or may not work with earlier version(s) as I haven’t bothered to test it anywhere else.
Configuring mail system is annoying. There are quite a bit different components which must be configured to work together.
My main choice for mail system is dovecot/postfix. As I don’t really understand how all this thing goes, I may have missed or misunderstood some parts. Or most of them. Feel free to correct this post.
For starter, most of basic configurations for Postfix and Dovecot has already been done by Ubuntu (or Debian) default configuration which includes enabling IMAP and TLS.
LDA (or MDA) delivers received mails to correct user and location. I let dovecot handle this thing because it’s easier this way. In /etc/postfix/main.cf:
mailbox_command = /usr/lib/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT"
[ Source ]
And that’s about it.
The Mailbox format. The alternatives are mbox (ancient, shouldn’t be used anymore, I believe), or dbox (Dovecot only), or some other formats (which I don’t really care about). So basically I go with Maildir.
/etc/dovecot/conf.d/10-mail.conf:
mail_location = maildir:~/Maildir
[ Source ]
The last one, Postfix authentication. I use Dovecot SASL because it’s easier.
/etc/postfix/main.cf:
smtpd_sasl_type = dovecot smtpd_sasl_path = private/auth smtpd_sasl_auth_enable = yes smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
/etc/dovecot/conf.d/10-master.conf:
unix_listener /var/spool/postfix/private/auth {
mode = 0666
}
[ Source ]
For Dovecot config, it’s usually already in there, one just uncomment it to enable.
Restart Dovecot and Postfix, and that’s it. As I mentioned before, Ubuntu has preconfigured many things which leaves me only few additional tasks to be done to enable simple mail system (with TLS, IMAP, and whatnot).
Protip: use Google Apps or Live Domain instead of managing your own mail server.
I’ll put it here for my own reference:
def self.slow_has_all_tags(tags)
p = Post.scoped
pt = PostsTag.arel_table
pt_arels = []
tags.each do |t|
t_id = Tag.where(:name => t).first[:id]
pt_arels << pt.where(pt[:tag_id].eq(t_id)).project(pt[:post_id])
end
pt_arels.each do |q|
p = p.where(:id => q)
end
p
end
And in fact, relatively fast.
Took me few weeks to find out that this one-liner does wonder:
echo manual >> /etc/init/mysql.override
(the line above is to disable mysql, obviously. And must be done as root)
The answer is on first hit (as of this post’s writing) of googling “ubuntu disable service” but you need to scroll down a bit and ignore shitload of crappy, outdated explanations to find that small gem.
Unfortunately doesn’t apply to previous LTS. Or does it?
Apparently I like using VM. Lots of them.
Only BSD and Linux though. And maybe SunOS and Minix sometime later this month if I feel like it.
After working on it for several weeks, finally it got to the point where it’s relatively usable. No more Bundler-on-Rails2 evilry and the fact that 3.0 branch is still supported.
The best part is the one above. Yes, Moebooru now runs on Rubinius/Puma. The only foreseeable problem is it uses Process.pid on file uploads which, when several people uploading (or working with) files at same time, the temporary filename will collide.
Though due to me aiming more for workable implementation instead of correct implementation, there are quite a lot legacy stuff still in there:
On the brighter side, now I can start cleaning up the code. Or upgrade all the way to Rails 3.2 ( ¬‿¬)
Um, yeah. Running Unicorn on DFBSD (in VMware) caused kernel panic. I have no idea why.
There’s a problem with nested query I previously mentioned: query time is unstable. The worst case is at least as slow as multiple self-join method – when the first few search key result is too big.
While digging more into the code called “Moebooru” which was forked from “Danbooru”, I noticed this:
def self.included(m)
m.extend(ClassMethods)
m.after_create :increment_count
m.after_destroy :decrement_count
end
def increment_count
connection.execute("update table_data set row_count = row_count + 1 where name = 'users'")
end
def decrement_count
connection.execute("update table_data set row_count = row_count - 1 where name = 'users'")
end
Counting takes ages, right. Except it is not. I’ve done this, yes, but on a table with 10+ millions of data (this one has ~400k in mainline danbooru), with multiple data inserted (this one got, uh, one every other week?) and queried every second (see below), and with the required count method not a simple select count(1) on some_table (which is what the example above used for).
The best part? It’s only used once, when user registers:
def set_role
if User.fast_count == 0
self.level = CONFIG["user_levels"]["Admin"]
elsif CONFIG["enable_account_email_activation"]
self.level = CONFIG["user_levels"]["Unactivated"]
else
self.level = CONFIG["starting_level"]
end
self.last_logged_in_at = Time.now
end
I haven’t been able to work on Moebooru for most of this month but I got to work on it this weekend. At ~30 changes, it went from -alpha to -beta.2 after receiving good amount of tests.
Summary of changes (more or less):
This afternoon I experimented with tag searching. Current implementation doesn’t scale beyond 6 tags due to parsing requirement. You can argue about binding variable all day but in the end it can be abused easily if the limit is lifted.
Continue readingSince I forgot to branch the original source, the branching looked awesomely crappy. Therefore I decided to rebase entire thing to ease up keeping track with Moebooru “mainline”. All my commits are now in branch “default”. If you didn’t do any change, backout up to revision 9174b6b5b02d and then pull again. And then, don’t ever touch moe branch again anymore.
It’s still quite slow even on my system. I should switch to Chrome again one of these days. I don’t even use Firefox anymore apart of on this specific system. Oh and Flash also crashes every other minutes in this browser.
Last week I posted about my random project which involves modernizing moebooru without doing complete rewrite (see this for yet another complete rewrite attempt).
Let’s revisit the plan:
Sure looks good. Need more testing though. There’s also one part which I totally had no idea why should be changed when upgrading to 1.9. Just grep for FIXME to see which it is and hopefully fix it up for me (or explain what it does).
As usual, having completed the work for today, live demo is up and open for everyone to break (…if there’s anyone, that is).
[ Live Demo | Repository ]
WARNING: using method below will lock yourself out when using emergency console since whatever crypt it’s using surely doesn’t understand bcrypt (as I experienced myself). Additionally, this solution won’t add bcrypt support to other applications using crypt interface like proftpd unless it’s started by preloading libxcrypt.so first (also from my own experience).
As much as Drepper want to pretend bcrypt is wrong solution, it actually gives one benefit: ease of switch to Linux. Some systems use bcrypt by default or configurable to use it. On other case, there might be time where you need system’s (or applications using system’s) crypt to handle bcrypt passwords from external system (usually web applications).
It’s quite difficult to enable bcrypt support in RHEL based distro as there is no libxcrypt and pam_unix2 packages available. Thankfully it’s available in Debian (and derivatives) in package libpam-unix2.
The README.Debian says to modify files in /etc/pam.d but if I remember it correctly, it confused apt PAM handling system or whatever. Fast forward few weeks, I discovered a better way to use it by creating PAM configuration in /usr/share/pam-configs. Since it’s mostly equivalent to normal pam_unix, I just copy and modify the file using this (long-ass) oneliner sed:
sed -e 's/pam_unix.so/pam_unix2.so/g;s/^Name: Unix authentication$/Name: Unix2 authentication/;s/pam_unix2.so obscure sha512/pam_unix2.so obscure blowfish rounds=8/;s/ nullok_secure//' /usr/share/pam-configs/unix > /usr/share/pam-configs/unix2
Then execute pam-auth-update, select Unix2 authentication and deselect Unix authentication. Don’t forget to update passwords for all other users as well or they won’t be able to login since pam_unix2 doesn’t recognize sha based hashes.
Actually, change all other users password to use md5 first before replacing the PAM with pam_unix2.
Update 2012-04-01: Removed nullok_secure since it isn’t supported.
Update 2012-06-09: Added warning.
If you didn’t know, the current moebooru running on oreno.imouto is using ancient version of many things. It also uses a custom lighty module (mod_zipfile) which doesn’t seem to be available anywhere.
I’ve updated it with latest Rails 2.x and made it compatible with nginx. Mostly. You can see it running here.
The plans:
We’ll see if I can actually finish this one. Grab the source here. Yeah, I’m using Mercurial for a Rails project.
I’ve enabled Danbooru repository sync to Bitbucket again as I’m planning to work on it again – mainly upgrading to Rails 3 and reducing insanity required for installation. And perhaps import the danbooru-imouto changes back.
As usual, the WordPress mirror is still going.
Because of one awesome bug inflicts eye-cancer when using Consolas font and deactivated “Bold text is a different colour”, I had to recompile PuTTY by hand (more like, by gcc). I initially tried to compile the PuTTYTray one but apparently they successfully mixed C and C++ code and completely broke the build procedure using mingw. Or I missed something obvious.
Anyway, I went back to vanilla PuTTY. As it turns out, compiling using latest mingw’s gcc isn’t a good idea since it removed -mno-cygwin option and therefore broken unless you do some magic edit. Thanks to that, I stopped bothering trying to compile it under Windows and used mingw-gcc for Linux (which is able to produce Windows binary). Here be the steps from beginning. Tested on Debian 6.
apt-get install mingw32 subversion perl svn co svn://svn.tartarus.org/sgt/putty putty cd putty perl mkfiles.pl cd windows make VER="-DSNAPSHOT=$(date '+%Y-%m-%d') -DSVN_REV='$(svnversion)' -DMODIFIED" TOOLPATH=i586-mingw32msvc- -f Makefile.cyg putty.exe
Patch is done before make (duh) and the diff can be found here. If you’re lazy (like me) you can just download the build at my server (link at bottom). Should be virus-free but I guess you can notify me if you encounter one. Built everyday until it breaks.
I kept forgetting them whenever I need one so I’ll put mine here and be happy:
###### Debian Main Repos deb http://http.debian.net/debian squeeze main contrib non-free #deb-src http://http.debian.net/debian squeeze main contrib non-free ###### Debian Security Update Repos deb http://security.debian.org squeeze/updates main contrib non-free #deb-src http://security.debian.org squeeze/updates main contrib non-free ###### Debian General Update Repos deb http://http.debian.net/debian/ squeeze-updates main contrib non-free #deb-src http://http.debian.net/debian/ squeeze-updates main contrib non-free ###### Debian Backports Repos deb http://http.debian.net/debian-backports squeeze-backports main contrib non-free #deb-src http://http.debian.net/debian-backports squeeze-backports main contrib non-free ###### Dotdeb Repo #deb http://packages.dotdeb.org squeeze all #deb-src http://packages.dotdeb.org squeeze all
It should cover mostly used packages and will keep me sane. Also debian-volatile has been replaced with debian-updates (god knows why it’s called like that) since squeeze (6.0) but in case I need to take care a lenny or earlier (derp) machines, this should also be added:
deb http://volatile.debian.org/debian-volatile lenny/volatile main contrib non-free
Oh and Debian/kFreeBSD within a FreeBSD system is quite funny.
Update 2012-12-27: Use improved CDN.
Many thanks to the creator himself (Igor Sysoev) for the tips.
Wow. With original release date as far as 8 years ago, they still releases patches for it!
AFAIK, this is one of the longest supported commercial game ever… 😀
[ Download here ]
So it apparently includes “what-if” stories for each classes.
Too bad it’s arcade only for now 🙁
