An OpenVPN configuration menu based generator

May 18, 2012

Setting up an openvpn server isn't a diffcult thing, but you have to deal with root certificates, rsa keys, confs, subnet addresses, iptables ... such trifles are annoying.

So I wrote this script to make my life easier, only to provide basic infomation like the server IP, and those configuraion will be done at the background, then tared packages are ready there for both server and clients.

ovpn_menu.png

Source is available at Google Code.

Script is recommanded running at your work station, and then upload only the server part tared file to the server, for secure concideration.

Features

  • Provide tared config which ready for any server distribution.
  • Random VPN subnet will be generated to avoid conflicting.
  • Random digital subffixed server/client CommonName will be assigned (if you don't provide one) for clearer management.
  • Configuration file is copied from the distributed OpenVPN in your system, which include full explnations of different options when you have to edit them. Default values is set by sed.
  • tls-auth enabled by default.

Usage

For new setup:

1
./ovpn_menu.sh

Just do as promoted. When select 5 to exit, all the files will be packed into a single NAME-all.tar.gz, you should save it to somewhere safe. And if you want to sign some more certificate from this root ca, put this tar file as the argument.

1
./ovpn_menu.sh /path/to/YOUR-VPN-all.tar.gz

At last the script also provide you iptables commands that should be useful to setup the server.

Download

Direct download via wget/curl should be ok:

http://ptcoding.googlecode.com/git/ovpn_menu.sh

"Leave password blank if dont want to change" in a django admin field

April 27, 2012

Django is a complicated but useful python web-framework, comparing to light weight frameworks like Bottle, web.py. I recently switch from Bottle, making use of it's powerful admin site to build a Email Account Management system.

An AdminSite offer interfaces to manage databases. It includes traditional authentications, permissions, data display and POST saving mechanisms, etc. , which are nasty trifles if you try implement them from scratch using bottle or other things.

Now I can build a decent system in about 100 lines code, after many many document reading and code digging, I want to share the story how I solve the problem encountered while implementing the "Leave password blank if don't want to change it" requirement. As commonly seen in other applications when user try to update their profile.

Firstly the background. My data Model:

1
2
3
4
5
class Users(Model):
    account = models.CharField(max_length=128, unique = True)
    password = models.CharField(db_column = 'crypt',max_length=384, blank=True)
    name = models.CharField(max_length=768)
    #...

The password is stored hashed in the DB crypt column. If I just let the AdminSite pickup things like that, it just show the password value in the plain input text.

Firstly I tell django to show it as a password input widget.

1
2
3
4
5
6
7
8
class MailUsersForm(forms.ModelForm):
    class Meta:
        model = Users
        widgets = {
                'password': forms.PasswordInput(render_value = False),
        }
class MailUsersAdmin(admin.ModelAdmin):
    form = MailUsersForm

Ah, overriding the form in the ModelAdmin does the trick.

Now in the edit page, the Password field shows an empty input widget.

django-password-field.png

Now I need to hash the password before commit to the DB. I found the save_model method.

1
2
3
4
5
6
7
class MailUsersAdmin(admin.ModelAdmin):
    # ...
    def save_model(self, request, obj, form, change):
        new_psw = request.POST["password"]
        if len(new_psw):
            obj.password = obj._hash_password()
            obj.save()

This works, but buggy. The password field is defaultly blank. It will clean the field in the DB without processing. The problem is, by the time in the save_model method is triggered, the obj is already updated by the values from the POST request. There's no way to get the old password value here. All I can do without changing the old password, is to skip the obj.save(). But what if I only want to update other things?

Through out the documents, I found some other places to approch the Model instance.

Firstly the django.forms.ModelForm.save method. The module instance is returned from this method (then passed to the save_model), so I inspect the object with ipdb to see if the data is updated here:

1
2
3
4
5
6
class MailUsersForm(forms.ModelForm):
 
    def save(self, *args, **kw):
        import ipdb; ipdb.set_trace()
        obj = super(MailUsersForm, self).save(*args, **kw)
        return obj

Unfortunately Yes. The instanced obj from the derived save is already updated to the POST value.

And django signals, pre_save, all the same way. (The signal thing is even emmited after the forms.ModelForm.save)

Googled around and nothing suitble, I decided to find the right way myself.

There must be somewhere in django codes that sets the attributes with values from the POST requsts. So I add a hook in the Model.

1
2
3
4
5
6
7
8
class Users(Model):
    #....
    def __setattr__(self, name, value):
        if name == "password":
            if self.id == 370:
                import ipdb; ipdb.set_trace()
 
        return Model.__setattr__(self, name, value)

The if self.id thing is for filtering the specific page I submit the save in the browser. Now ipdb let me in debug mode.

ipdb> w
 
#................. Many Many unrelated things
 
  /usr/lib/python2.7/site-packages/django/forms/forms.py(272)full_clean()
    270         self._clean_fields()
    271         self._clean_form()
--> 272         self._post_clean()
    273         if self._errors:
    274             del self.cleaned_data
 
  /usr/lib/python2.7/site-packages/django/forms/models.py(309)_post_clean()
    307         opts = self._meta
    308         # Update the model instance with self.cleaned_data.
 
--> 309         self.instance = construct_instance(self, self.instance, opts.fields, opts.exclude)
    310
    311         exclude = self._get_validation_exclusions()
 
  /usr/lib/python2.7/site-packages/django/forms/models.py(51)construct_instance()
     49             file_field_list.append(f)
     50         else:
---> 51             f.save_form_data(instance, cleaned_data[f.name])
     52
     53     for f in file_field_list:
 
  /usr/lib/python2.7/site-packages/django/db/models/fields/__init__.py(454)save_form_data()
    452
    453     def save_form_data(self, instance, data):
--> 454         setattr(instance, self.name, data)
#---------------^ this matters!
    455
    456     def formfield(self, form_class=forms.CharField, **kwargs):
 
> /home/boypt/Projects/maildbadmin/maildbadmin/models.py(55)__setattr__()
     53             if self.id == 370:
     54                 import ipdb; ipdb.set_trace()
---> 55                 print "set :", value
     56 #
     57 #

From the call stack, the setattr call to the Model changes the data. I check django/db/models/fields/__init__.py source, find that this is a Field base class, so the obvious solution is to override this method.

1
2
3
4
5
6
7
8
9
10
11
class PasswordCharField(models.CharField):
    def save_form_data(self, instance, data):
        if data != u'':
            data = instance._hashed_pwd(data)
            setattr(instance, self.name, data)
 
class Users(Model):
    #...
    password = PasswordCharField(db_column = 'crypt', max_length=384, blank=True)
 
    #.....

OK, now everything's done here. The save_model overriding is no more needed, for the Model instance's password will not be set to empty at all when the user left the field blank when they submit.

Mysql Replication 最简单配置

April 24, 2012

Google 一下Mysql Replication可以找到相关配置说明满地都是,作为双机热备方案很多时候需要用到,但是稍微看下这些资料发现都是乱七八糟的,所以果断去读MySQL 5.1 Reference Manual: 16.1.1. How to Set Up Replication。资料很长,不过最后总结的配置其实非常简单。

配置

1. In Master (例子IP 10.6.7.7)

my.cnf 添加这两行:

[mysqld]
log-bin=mysql-bin
server-id=1

终端中运行:

1
2
3
mysqldump -uroot -p --all-databases --master-data | gzip -9 -c > dbdump.db.gz
scp dbdump.db.gz user@10.6.7.8:~
echo "CREATE USER 'repl'@'%' IDENTIFIED BY 'slavepass'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%';" | mysql -uroot -p

2. In Slave (例子IP 10.6.7.8)

my.cnf 添加和Master不同的ID:

[mysqld]
server-id=1001

终端中运行:

1
2
3
MASTER_IP=10.6.7.7
 
(echo "SLAVE STOP; CHANGE MASTER TO MASTER_HOST='$MASTER_IP', MASTER_USER='repl', MASTER_PASSWORD='slavepass';"; zcat dbdump.db.gz;echo "SLAVE START;") | mysql -uroot -p

OK, 收工。

验证

要验证同步,在Master执行:CREATE DATABASE test_repl;, 在Slave执行 SHOW DATABASES;,可以看到test_repl同步完成,在Master执行:DROP DATABASE test_repl;,Slave的也相应消失。

完整的启动LOG /var/log/mysql/error.log大致如下,可看到replication线程启动正常。

120424 16:34:51 [Note] Plugin 'FEDERATED' is disabled.
120424 16:34:52  InnoDB: Initializing buffer pool, size = 8.0M
120424 16:34:52  InnoDB: Completed initialization of buffer pool
120424 16:34:52  InnoDB: Started; log sequence number 0 1174665
120424 16:34:52 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.000001' at position 1060, relay log './ub1110-relay-bin.000024' position: 251
120424 16:34:52 [Note] Event Scheduler: Loaded 0 events
120424 16:34:52 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.61-0ubuntu0.11.10.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
120424 16:34:52 [Note] Slave I/O thread: connected to master 'repl@172.28.16.82:3306',replication started in log 'mysql-bin.000001' at position 1060

收尾

Slave的/var/log/mysql/error.log可能会看到一个warnning

120423 18:01:41 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his host name changed!! Please use '--relay-log=XXXXX-relay-bin' to avoid this problem.

如它所说,把这句加入到my.cnf[mysqld]即可。

日常维护

如果数据库操作频繁,binlog消耗的磁盘空间挺大的,设置Master的expire_logs_days可以控制存储binlog的文件个数。

如果留下了大堆binlog需要清理,可以执行这句清理7天前的binlog:

1
mysql -uroot -p -e "PURGE MASTER LOGS BEFORE DATE_SUB( NOW(), INTERVAL 7 DAY);"

Midware beaker using Google App Engine Memcache API as backend

April 11, 2012

What

Beaker is a library for caching and sessions for use with web applications and stand-alone Python scripts and applications. -- Beaker Offcial Document

Why

I recently writting a small app running on Google App Engine, Beaker is the ideal midware working together with light weight python web frameworks, eg bottle, web.py, to handle session things.

The GAE provides a bunch of external services, which are extremely convinent for building apps. Beaker supports using google's Datastore api as backend, but operations over the Datastore is expensive (actually still free to me, hah), it would be more delightful to use the (currently) unlimited memcache service.

Beaker supports memcached backend, too, but only to those ordinary APIs, like pylibmc, cmemcache which connect to specified addresses as a client. The GAE memcache however, with exactly same interfaces, but no need to connect to somewhere before operating data. So, it needs some work over Beaker.

Patch

  1. Download this googlememcache.py ext module and save as beaker/ext/googlememcache.py
  2. Apply the following patch in the beaker directory. (Minor modification over cache.py)
diff -r 0bf25d5b4287 beaker/cache.py
--- a/beaker/cache.py   Mon Mar 12 14:19:57 2012 -0700
+++ b/beaker/cache.py   Wed Apr 11 17:29:30 2012 +0800
@@ -18,6 +18,7 @@
 import beaker.ext.database as database
 import beaker.ext.sqla as sqla
 import beaker.ext.google as google
+import beaker.ext.googlememcache as googlememcache
 
 # Initialize the cache region dict
 cache_regions = {}
@@ -115,6 +116,7 @@
           'ext:database':database.DatabaseNamespaceManager,
           'ext:sqla': sqla.SqlaNamespaceManager,
           'ext:google': google.GoogleNamespaceManager,
+          'ext:googlememcache': googlememcache.GoogleMemcacheNamespaceManager,
           })

Usage

Nothing much different from using the original ext:google

1
2
3
4
5
6
session_opts = {
    'session.cookie_expires': True,
    'session.type': 'ext:googlememcache',
}
 
app = SessionMiddleware(orig_app, session_opts)

Upstream

I submitted this patch to the Beaker offcial, not sure if they will adopt it as a new feature in the later releases.

挫挫的工商银行网银密码器

March 31, 2012

3月27号手机收到工行短信说网银密码卡可以免费换成密码器,期限是3月31日(还真有心,sigh)。刚好今天最后一天,有空就去银行换了回来。

密码器早就见过,很多年前就见过汇丰银行的密码器,只有一个按钮,用来显示随机密码,当时工行才在推U盾。

hsbc-code.jpeg

今天拿到工行的密码器,卧了个去:

icbc-passcode-1.jpg

感情这个密码器的设计人还学人当物理键盘控啊。

体积够宽大就算了,还!够!厚!

icbc-passcode-2.jpg

第一次开机后,要输入业务单上的激活码,然后设置一个6位开机密码:

icbc-passcode-4.jpg

icbc-passcode-5.jpg

以后每次开机都要输入这个密码,输入错了有失败计数器。没看到有修改密码的功能,所以,第一次弄要小心点了。 更正:说明书上有修改密码的操作说明。

开机后,密码器处于输入状态。可以随便输入数字标点,按确定后会算出一个6位数字。

icbc-passcode-7.jpg

但是这不是网银要使用的功能,一般使用还是和其他密码器那样,使用基于时间参数的算法。 更正:网银有些地方,如转帐操作需要输入页面显示的号码来获取密钥,而其他地方只需输入以下时间随机码。

icbc-passcode-8.jpg

在密码器刚开机的输入状态直接按“确认”,就进入这个每60秒变化一次的动态密码界面:

icbc-passcode-6.jpg

总结

应该说ICBC终于意识到一直推广的U盾的硬伤,除了开发硬件,还得开发驱动程序处理各个操作系统(版本)的兼容,更重要的是去年出了几个案件是有用户忘了拔U盾被木马盗取网银的事情;采用和PC系统之间独立的密码器就避免了兼容问题和这个安全漏洞。

虽然说工行有安全意识,却一直使用ActivX的过时的不安全的技术来处理网银的输入框,这点就像U盾一样,既不安全,又影响用户体验,起码在MacOS和Linux等非windows桌面系统里面就无法使用;采用动态密码器其实完全可以去掉这些所谓的安全输入控件,因为密码的一次有效性,即使内容被第三方劫持,也没有再次使用的价值。

一次本博客的性能故障排查

March 31, 2012

无关背景

Ubuntu 10.04 这个版本已经服役两年,虽说是LTS,但最近起发现已经有点力不从心,主要是ppa上一些比较重要的库,如PHP 5.3,ningx的团队已经停止维护,uwsgi则总是落后半年的样子。很大一个原因是这些包在新版的Ubuntu里面已经有官方维护,ppa的第三方维护会缺乏跟进。所以在一定意义上,可以宣布Ubuntu 10.04死亡了。

但Ubuntu的包维护策略就是那样,要么自己维护所有用到的包,要么每隔一段时间就跟着官方一次大升级。觉得不爽就干脆把VPS的系统也改成Archlinux。

现象

一番大迁移后所有的东西都正常上线,但直到一个多星期后的昨天晚上才注意到博客的性能问题。 因为博客那里有nginx直接读取静态的缓冲机制,所以动态执行非常慢之前都没留意到。

把缓冲关闭就非常明显了,任何一次点击的页面都要10秒才能打开。

排查

引起应用缓慢的因素是非常多的,概括来说有两种:IO慢运算慢

运算慢是不大常见的,虽说PHP性能一直受到诟病。但如果是这个问题是很明显的,top里面php的子进程会占满CPU,高居不下。此前排查过一个drupal的站点,是因为前端模板的组件方式存在循环引用,用profile过程看,正则替换的regrex函数占了绝大多的CPU时间。

在SSH看到,打开页面的10秒内php-fpm的子进程基本没占CPU,排除这个可能。

IO慢就复杂了,每个组件都有IO,得先确定IO的范围。

首先想到是数据库Mysql,arch的包太新,有bug?linode主机的磁盘快挂了,磁盘很慢?这些都不好判断,确定这些得借助工具。

Profiling是追踪一个应用的运行流程,记录所有的函数调用栈、记录调用时间的过程,是追查性能问题的最佳帮手。Python里面是有个repoze.profile的wsgi中间件很方便进行排查,但我没做过完整PHP开发,就暂时不大清楚有什么方便的profile方案 (以前弄过早忘了),但google一下还是有很多方案的 ,不过我首先想起newrelic的应用监控系统就提供了这个功能。

Newrelic针对WEB应用和服务器监控服务,其中的服务器监控是免费的,但对应用的监控只有14天试用;所以我赶紧重新申请个帐号,用来监控一下wordpress。安装好监控模块后,超过2s的响应会在他们的Transaction traces记录下来。

newrelic-profile-apt-blog.png

图表数据可以排除数据库问题,几十个数据库的操作都是在ms级别完成,而在apt-blog.net的耗时花了10秒。其实我一开始对这个报告也没看懂,newrelic的直观性还有待提高,其实在这里出现域名的意思是有网络请求,比如askimet的评论、插件的更新等都要和外部请求,这里就会出现域名。

而现在出现了自己博客的域名,那问题就是,程序里面某个地方需要请求自己的域名,可能是检查状态的操作,被卡住了,直到超时才返回。

本机程序访问不到本机,基本确定是iptables规则出问题了,在filter表的最后插入这样一句:

-A INPUT -j LOG

iptables的规则一般是,除非明文允许,否则拒绝,所以经过一系列的规则后如果还没有没ACCEPT的,在最后的都是被DROP了,把这句放最后可以看到究竟是什么包被DROP了。

查看/var/log/everything.log看到这样的记录:

Mar 31 03:17:32 (none) kernel: [17554800.765033] IN=lo OUT= MAC=00:00:00:00:00:0 0:00:00:00:00:00:00:08:00 SRC=106.187.36.50 DST=106.187.36.50 LEN=60 TOS=0x00 PR EC=0x00 TTL=64 ID=31529 DF PROTO=TCP SPT=45594 DPT=80 WINDOW=32792 RES=0x00 SYN URGP=0

这里透露了重要信息:IN=lo SRC=106.187.36.50 DST=106.187.36.50,PHP确实有访问本地网络,送了给lo网卡,SRC和DST都是本地的公网地址。赶紧检查iptables的规则,果然没了对lo设备的允许规则。一般配置机器我都是用记录在自己的Wiki的iptables那套规则的。关于lo的这几句可能一时没仔细想其作用,迁移系统那天脑抽手贱就删掉了。

加入允许lo设备的这句:

-A INPUT -i lo -j ACCEPT

至此,问题解决。

OpenVPN over HTTP 突破Layer 7 QoS的宽带网速限制

March 4, 2012

Why

我家里用的是一家三线的便宜小区宽带,标称有几个M的带宽,虽说有些资源确实能达到这个速度,但发现直连VPN的速度从来都没上去过,大概30k/s,不难猜测到,ISP在链路上做了手脚,即所谓的Layer 7 Priority QoS。因为入线是100M进宅然后PPPoE,不像ADSL那样直接物理链路就限制了连接速度,在链路层做QoS也很合理。平时HTTP打开网页、下载到合适链路的话很容易达到满速,而OpenVPN的TCP/UDP包估计就被当成P2P流量被限制了,所以网速一点都不给力。

用OpenVPN科学上网是最稳定最灵活的方式了,它基于udp/tcp的协议,比pptp、l2tp等直接跑IP包的开销虽说大一点,但好处就是容易把数据流重新封装,避开链路的关卡。显然它也支持HTTP代理,所以把OpenVPN变成HTTP协议,就可以在家里跑上满速的VPN。

How

OpenVPN要过HTTP代理,只能用TCP协议,这个需要服务端和客户端都要稍作修改。

Polipo as Tunnel Proxy

首先在VPS上安装一个http proxy,我选择polipo,比较轻量级。

apt-get install polipo

配置也是很简单的,编辑/etc/polipo/config,原来的配置文件基本全都是注释,直接在文件底部加上:

proxyAddress = "0.0.0.0"
proxyPort = 8128
authCredentials = "user:password3.141592654"
tunnelAllowedPorts = 1194
  • polipo默认只监听本地的127.0.0.1,要拿来做服务就要监听外网
  • polipo默认监听8123,为了不让扫代理的盯上,自己随便写个端口
  • 加上http basic验证,这个密码不要紧,后面让openvpn自动应答
  • 允许管道模式连接openvpn的1194端口

OpenVPN

Server

服务端倒是不需要很多配置,确定是tcp模式监听连接(我是多开一个openvpn server,子网错开,个人喜欢吧)

Client

客户端就要指定使用代理的方式:

remote 127.0.0.1 1194
http-proxy YOUR.VPS.IP.HERE.com 8128 pw.txt
http-proxy-retry

这个pw.txt是上述的HTTP的认证信息,用户名密码各一行。

现在连接OpenVPN,可以看到连接过程的Log有这么几句,基本就确定OpenVPN over HTTP成功了!

Sun Mar  4 19:12:58 2012 Attempting to establish TCP connection with 199.101.103.107:8192 [nonblock]
Sun Mar  4 19:12:59 2012 TCP connection established with [IPADDR]:8128
Sun Mar  4 19:12:59 2012 Send to HTTP proxy: 'CONNECT 127.0.0.1:1194 HTTP/1.0'
Sun Mar  4 19:12:59 2012 Attempting Basic Proxy-Authorization
Sun Mar  4 19:13:00 2012 HTTP proxy returned: 'HTTP/1.1 200 Tunnel established'

Related

最后透露一下我的科学上网环境是在跑OpenWRT的路由器上跑VPN,然后配合chnroute的路由表,当然dnsmasq也经过配置负责把国内常用域名的解释交给国内的114.114.114.114服务器,这样基本一回家手机kindle电脑等全都是翻墙环境,而且速度非常良好。

Reference

x11vnc - 远程工作站桌面

February 5, 2012

Gnome Desktop Environment的VNC服务开启很容易,仅需点开Desktop Sharing,打个勾就完了。

vino-preferences Gnome默认的VNC配置界面

但是有时候某台工作站没有打开这个VNC服务,只有SSH,但是又必须通过桌面来操作,这时候自带的vino server就好像不容易跑起来。

x11vnc这个小工具就可以派上用场,它对当前的x11 session建立VNC服务,很简单直接运行x11vnc,就等你连接5900端口了。当然,如果一定要指定一个session,有个-display参数可以指定,或者看看x11vnc --help,好长好长~~~

Page optimized by WP Minify WordPress Plugin

 
Powered by Wordpress and MySQL. Theme by Shlomi Noach, openark.org