2014-02-09

pythonrc.pyファイル

環境変数「PYTHONSTARTUP」に、python起動時に実行したいファイルを指定できる。

bashなら、~/.bashrcに設定すると楽。

PYTHONSTARTUP=~/.pythonrc.py
export PYTHONSTARTUP

内容は、webで公開している人がいるので参考に。

readlineとrlcompleterの設定、pprintの設定はしとくと便利。

import readline, rlcompleter
readline.parse_and_bind("tab: complete")

import sys, pprint
def my_displayhook(value):
  if value is not None:
    try:
      import __builtin__
      __builtin__._ = value
    except ImportError:
      __builtins__._ = value
    pprint.pprint(value)
sys.displayhook = my_displayhook

2014-02-08

python の csv.writer で日本語(encoding指定)

前回の記事で、csv.readerのencoding指定版を作ってみたので、csv.writerも試してみました。

import csv

class UnicodeCsvWriter:
  def __init__(self, fileobj, dialect='excel', encoding='utf-8', *args, **kwds):
    self.writer = csv.writer(fileobj, dialect, *args, **kwds)
    self.dialect = dialect
    self.encoding = encoding
  def writerow(self, seq):
    row_seq = []
    for elem in seq:
      if isinstance(elem, unicode):
        row_seq.append(elem.encode(self.encoding))
      else:
        row_seq.append(elem)
    self.writer.writerow(row_seq)
  def writerows(self, seq_of_seq):
    for seq in seq_of_seq:
      self.writerow(seq)

class UnicodeDictCsvWriter(csv.DictWriter):
  def __init__(self, f, fieldnames, restval="", extrasaction="raise",
               dialect="excel", encoding='utf-8', *args, **kwds):
    csv.DictWriter.__init__(self, f, fieldnames, restval, extrasaction, dialect, *args, **kwds)
    self.writer = UnicodeCsvWriter(f, dialect, encoding=encoding)

sys.getdefaultencoding()がasciiでも、以下のコードでエラーが出ないはず。
(インタープリター起動時のLANGはen_GB.utf-8)

>>> import StringIO.StringIO
>>> s_io1 = StringIO.StringIO()
>>> u_writer = UnicodeCsvWriter(s_io, encoding='utf-8')
>>> u_writer = UnicodeCsvWriter(s_io1, encoding='utf-8')
>>> u_writer.writerow([1, u'あ', 'い'])
>>> u_writer.writerows([[1, u'あ', 'い'], [2, u'あああ', 22]])
>>> print(s_io1.getvalue())
1,あ,い
1,あ,い
2,あああ,22

>>>
>>> s_io2 = StringIO.StringIO()
>>> ud_writer = UnicodeDictCsvWriter(s_io2, [1,2,3], encoding='utf-8')
>>> ud_writer.writerow({ 1: 1, 2: u'あ', 3: 'い' })
>>> ud_writer.writerows([{ 1: 1, 2: u'あ', 3: 'い' }, {1: 11, 2: u'ああ', 3: 'いい' }])
>>>
>>>
>>> print(s_io2.getvalue())
1,あ,い
1,あ,い
11,ああ,いい

2014-02-06

python の csv.reader で日本語(encoding指定)

python2.7ではcsvモジュールが標準で使えるけど、日本語関係ではまったので、メモ。

CSVファイルを「codecs.open」などでunicode型で読み込んでも、csv.readerでnext()すると、unicode型でなくなって、色々と文字コード関係のエラーになる。

test.csvの中身(UTF-8)

  "a", "あいう", "123"
  "b", "かきく", "987"

pythonの例::

import codecs, csv

csv_reader = csv.reader(codecs.open("test.csv", encoding="utf-8"))
# ここで、unicode型でなくなる
# sys.getdefaultencoding()がasciiとかだと、コンソール表示
# しようとした時などに日本語でエラー
rows = csv.next()

encoding指定できるcsv.readerっぽいクラスを作りました。使い方はcsv.readerとかとだいたい一緒。

class UnicodeCsvReader:
  """csv.reader wrapper which decodes each value with designated encoding"""
  def __init__(self, iterable, dialect='excel', encoding="utf-8", *args, **kwds):
    self.reader = csv.reader(iterable, dialect=dialect, *args, **kwds)
    self.encoding = encoding
    self.dialect = self.reader.dialect
    self.line_num = 0
  def __iter__(self):
    return self
  def decode(self, value):
      return value and value.decode(self.encoding) or value
  def next(self):
    # csv.reader.next returns a list of values of next row
    cols = [ self.decode(x) for x in self.reader.next() ]
    self.line_num = self.reader.line_num
    return cols

class UnicodeCsvDictReader(csv.DictReader):
    def __init__(self, f, fieldnames=None, restkey=None, restval=None,
                 dialect="excel", encoding="utf-8", *args, **kwds):
        csv.DictReader.__init__(
            self, f, fieldnames, restkey, restval, dialect, *args, **kwds)
        self.encoding = encoding
        self.reader = UnicodeCsvReader(f, encoding=encoding)

参考にしたページです。

関係ないけど、前回のsoftetherのSlackBuild、色々と稚拙ですね。SLCKFLAGSを設定だけして、使ってないとか。
まぁ、SlackBuild使うような人なら、気になるところは自分で何とかすると思うので、そのままにします。

*1:UnicodeReaderに__iter_メソッドが抜けていたので追加しました。

2014-02-03

softetherのSlackBuildを作ってみました

開始してから、すっかり放置してしまいましたが…

月も変わったし、これから気分変えて臨みます。

以前耳にしたことのあったSoftEther VPN プロジェクト - SoftEther VPN プロジェクトがオープンソースになていたので、slackware系でビルドできるようにSlackBuild Scripts - SlackWikiっぽいのを作ってみました。debianパッッケージを公開したページもあったので。

ちなみに作業・動作確認環境はWindows 7上のvmware player上のSalixです。

以下の内容を適当な場所で「softether.SlackBuild」として保存して、実行してください。

#!/bin/sh

# Slackware build script for <softether>

# Copyright <2014> <msiz> <Japan>
# All rights reserved.
#
# Redistribution and use of this script, with or without modification, is
# permitted provided that the following conditions are met:
#
# 1. Redistributions of this script must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
#
#  THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY EXPRESS OR IMPLIED
#  WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
#  MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO
#  EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
#  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
#  PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
#  OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
#  WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
#  OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
#  ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

PRGNAM=softether                 # replace with name of program
VERSION=${VERSION:-4.04-9412}    # replace with version of program
PKGVERSION=${PKGVERSION:-4.04_9412} # "-" within ver has side-effects
BUILD=${BUILD:-1}
TAG=${TAG:-_msiz}   # the "_SBo" is required
PKGTYPE=${PKGTYPE:-txz}

# Automatically determine the architecture we're building on:
if [ -z "$ARCH" ]; then
  case "$( uname -m )" in
    i?86) ARCH=i486 ;;
    arm*) ARCH=arm ;;
    # Unless $ARCH is already set, use uname -m for all other archs:
       *) ARCH=$( uname -m ) ;;
  esac
fi

CWD=$(pwd)
SRCTAR="http://jp.softether-download.com/files/softether/v4.04-9412-rtm-2014.01.15-tree/Source%20Code/softether-src-v${VERSION}-rtm.tar.gz"
if [ ! -e `basename $SRCTAR` ] ; then
  if which wget > /dev/null 2>&1 ; then
    wget $SRCTAR
  elif which curl > /dev/null 2>&1 ; then
    curl -O $SRCTAR
  else
    echo "wget or curl is required to download source archive."
    exit 1
  fi
fi
TMP=${TMP:-/tmp/SBo}  # For consistency's sake, use this
PKG=$TMP/package-$PRGNAM
OUTPUT=${OUTPUT:-/tmp}  # Drop the package in /tmp

if [ "$ARCH" = "i486" ]; then
  SLKCFLAGS="-O2 -march=i486 -mtune=i686"
  LIBDIRSUFFIX=""
elif [ "$ARCH" = "i686" ]; then
  SLKCFLAGS="-O2 -march=i686 -mtune=i686"
  LIBDIRSUFFIX=""
elif [ "$ARCH" = "x86_64" ]; then
  SLKCFLAGS="-O2 -fPIC"
  LIBDIRSUFFIX="64"
else
  SLKCFLAGS="-O2"
  LIBDIRSUFFIX=""
fi

set -e # Exit on most errors
# If you prefer to do selective error checking with
#   command || exit 1
# then that's also acceptable.

rm -rf $PKG
mkdir -p $TMP $PKG $OUTPUT
cd $TMP
rm -rf $PRGNAM-$VERSION
#tar xvf $CWD/$PRGNAM-$VERSION.tar.gz
tar xvf $CWD/`basename $SRCTAR`
mv v$VERSION $PRGNAM-$VERSION
cd $PRGNAM-$VERSION
chown -R root:root .
find -L . \
 \( -perm 777 -o -perm 775 -o -perm 750 -o -perm 711 -o -perm 555 \
  -o -perm 511 \) -exec chmod 755 {} \; -o \
 \( -perm 666 -o -perm 664 -o -perm 640 -o -perm 600 -o -perm 444 \
 -o -perm 440 -o -perm 400 \) -exec chmod 644 {} \;

# Your application will probably need different configure flags;
# these are provided as an example only.
# Be sure to build only shared libraries unless there's some need for
# static.
#CFLAGS="$SLKCFLAGS" \
#CXXFLAGS="$SLKCFLAGS" \
#./configure \
#  --prefix=/usr \
#  --libdir=/usr/lib${LIBDIRSUFFIX} \
#  --sysconfdir=/etc \
#  --localstatedir=/var \
#  --mandir=/usr/man \
#  --docdir=/usr/doc/$PRGNAM-$VERSION \
#  --build=$ARCH-slackware-linux

# Compile the application and install it into the $PKG directory

if [ "$ARCH" = "x86_64" ]; then
  MAKFILE=linux_64bit.mak
else
  MAKFILE=linux_32bit.mak
fi
cp src/makefiles/${MAKFILE} ./Makefile
sed -i.bkp -e '/^INSTALL_BINDIR/i DESTDIR = ' Makefile
sed -i.bkp2 -e 's/mkdir -p \$(INSTALL_/mkdir -p $(DESTDIR)$(INSTALL_/g' Makefile
sed -i.bkp3 -e 's/\(cp [^\$]\+\)\$(INSTALL_/\1$(DESTDIR)$(INSTALL_/g' Makefile
sed -i.bkp4 -e 's/\(echo .\+\)> \$(INSTALL_/\1> $(DESTDIR)$(INSTALL_/g' Makefile
sed -i.bkp5 -e 's/\(chmod [^\$]\+\)\$(INSTALL_/\1$(DESTDIR)$(INSTALL_/g' Makefile

if which clang 2> /dev/null >&1 ; then
  CC=clang
else
  CC=gcc
fi
#CC="$CC" \
#CFLAGS="$SLKCFLAGS" \
#CXXFLAGS="$SLKCFLAGS" \
make CC=$CC
mkdir -p $PKG/usr/bin
make install DESTDIR=$PKG/

# Strip binaries and libraries - this can be done with 'make install-strip'
# in many source trees, and that's usually acceptable if so, but if not,
# use this:
find $PKG -print0 | xargs -0 file | grep -e "executable" -e "shared object" | grep ELF \
  | cut -f 1 -d : | xargs strip --strip-unneeded 2> /dev/null || true

# Compress man pages
# If the man pages are installed to /usr/share/man instead, you'll need to either
# add the --mandir=/usr/man flag to configure or move them manually after the
# make install process is run.
if [ -e $PKG/usr/man ]; then
  find $PKG/usr/man -type f -exec gzip -9 {} \;
  for i in $( find $PKG/usr/man -type l ) ; do ln -s $( readlink $i ).gz $i.gz ; rm $i ; done
fi

# Compress info pages and remove the package's dir file
# If no info pages are installed by the software, don't leave this in the script
if [ -e $PKG/usr/info/dir ]; then
  rm -f $PKG/usr/info/dir
  gzip -9 $PKG/usr/info/*.info*
fi

# Remove perllocal.pod and other special files that don't need to be installed,
# as they will overwrite what's already on the system.  If this is not needed,
# remove it from the script.
# Remove 'special' files
find $PKG -name perllocal.pod \
  -o -name ".packlist" \
  -o -name "*.bs" \
  | xargs rm -f

# Copy program documentation into the package
# The included documentation varies from one application to another, so be sure
# to adjust your script as needed
# Also, include the SlackBuild script in the documentation directory
mkdir -p $PKG/usr/doc/$PRGNAM-$VERSION
cp -a \
  AUTHORS.TXT     BUILD_WINDOWS.TXT  LICENSE  THIRD_PARTY.TXT \
  BUILD_UNIX.TXT  ChangeLog          README   WARNING.TXT  \
  $PKG/usr/doc/$PRGNAM-$VERSION
cat $CWD/$PRGNAM.SlackBuild > $PKG/usr/doc/$PRGNAM-$VERSION/$PRGNAM.SlackBuild

# Copy the slack-desc (and a custom doinst.sh if necessary) into ./install
mkdir -p $PKG/install
#cat $CWD/slack-desc > $PKG/install/slack-desc
#cat $CWD/doinst.sh > $PKG/install/doinst.sh
cat > $PKG/install/slack-desc <<END_OF_SLACK_DESC
# HOW TO EDIT THIS FILE:
# The "handy ruler" below makes it easier to edit a package description.  Line
# up the first '|' above the ':' following the base package name, and the '|'
# on the right side marks the last column you can put a character in.  You must
# make exactly 11 lines for the formatting to be correct.  It's also customary
# to leave one space after the ':'.

         |-----handy-ruler------------------------------------------------|
softether: An Open-Source Cross-platform Multi-protocol VPN Program
softether:
softether: SoftEther VPN is developed by SoftEther VPN Project at
softether: University of Tsukuba.  Department of Computer Science has
softether: dozens of overly-enthusiastic geeks.
softether:
softether:
softether:
softether:
softether:
END_OF_SLACK_DESC

# Make the package; be sure to leave it in $OUTPUT
# If package symlinks need to be created during install *before*
# your custom contents of doinst.sh runs, then add the -p switch to
# the makepkg command below -- see makepkg(8) for details
cd $PKG
/sbin/makepkg -l y -c n $OUTPUT/$PRGNAM-$PKGVERSION-$ARCH-$BUILD$TAG.$PKGTYPE

個人的には、fakerootを使うのがお勧め。

fakerootが使えるなら、こんな感じで実行すると、「/tmp/softoether...」というslackwareのパッケージができるはずです。

fakeroot sh -x ./softether.SlackBuild

誰かの役に立てばいいなぁと。

msizの日記

ソフトウェア関係の覚え書きが中心になる予定

pythonrc.pyファイル

python の csv.writer で日本語(encoding指定)

python の csv.reader で日本語(encoding指定)

softetherのSlackBuildを作ってみました