Archive for August 2014

Delete all but the most recent N files(objects) from s3.

2014-08-22 by gihnius, tagged as ruby

When I backup servers and save data to AWS S3, usually just upload entire files to S3. This is done automatic by the backup script. The uploading process looks simple:

S3_PUT backup_data_a s3://org-sys-backup/
S3_PUT backup_data_b s3://org-sys-backup/
S3_PUT backup_data_c s3://org-sys-backup/
S3_PUT backup_data_d s3://org-sys-backup/
S3_PUT backup_data_e s3://org-sys-backup/
S3_PUT backup_data_f s3://org-sys-backup/
...

## or 
S3_PUT backup_data_20140801xxxx s3://org-sys-backup/
S3_PUT backup_data_20140802xxxx s3://org-sys-backup/
S3_PUT backup_data_20140803xxxx s3://org-sys-backup/
S3_PUT backup_data_20140804xxxx s3://org-sys-backup/
S3_PUT backup_data_20140805xxxx s3://org-sys-backup/

I don't care the backups' data older than 1 week, then to free the space in S3 I have to delete those old data files (called objects in S3), just keep the most recent 5 copies. There maybe ready-made tools available, but I want to roll my own. I wrote a ruby script to do that, here is source code:

#!/usr/bin/env ruby

## Delete all but the most recent N files from s3://bucket
## supply a prefix and count of copies and the bucket name

require 'right_aws'
require 'optparse'

opts = {}
OptionParser.new do |o|
  o.on("-a", "--access-key-id X", "aws ec2 access key id") do |x|
    opts[:aws_access_key_id] = x
  end
  o.on("-s", "--secret-access-key X", "aws ec2 secret access key") do |x|
    opts[:aws_secret_access_key] = x
  end
  o.on("-b", "--bucket X", "bucket name") do |x|
    opts[:bucket] = x || ""
  end
  o.on("-p", "--prefix X", "object name prefix") do |x|
    opts[:prefix] = x || ""
  end
  o.on("-c", "--count N", "how many copies to keep") do |x|
    opts[:count] = x || 1
  end
  o.parse!
end

if opts[:bucket].size < 1 || opts[:prefix].size < 1
  puts "bucket or prefix not valid!"
  exit 1
end

s3 = RightAws::S3.new(opts[:aws_access_key_id], opts[:aws_secret_access_key])

object_keys = {}

s3.interface.incrementally_list_bucket(opts[:bucket], {'prefix' => opts[:prefix], 'delimiter' => '/'}) do |item|
  item[:contents].each{|c| object_keys[c[:key]] = c[:last_modified] }
end

## sort obj keys by time(:last_modified)
sorted_keys = object_keys.sort_by{|k,v| Time.parse(v).to_i}.map{|x| x[0]}
puts "Found #{sorted_keys.size} objects with prefix: #{opts[:prefix]}"

if sorted_keys.size > opts[:count].to_i
  old_keys = sorted_keys.first(sorted_keys.size - opts[:count].to_i)
  n = old_keys.inject(0) do |a, k|
    s3.interface.delete(opts[:bucket], k)
    a += 1
  end
  puts "Deleted #{n} old objects."
end

This script get the objects list which object name match the given prefix, and sort them by modified time, then find the old ones and delete them.

Save it as, eg. s3_cleanup.rb.

How to run it?

You need to install a ruby gem 'right_aws' by: gem install right_aws, then

ruby s3_cleanup.rb -a 'access-key-id' -s 'secret-access-key' -b org-sys-backup -p 'backup_data_' -c 5

an example output:

Found 8 objects with prefix: backup_data_
Deleted 3 old objects.

This script is available in Github.

You may want to read about Object Expiration before checking out the script. That maybe a official recommended practices.

Amazon S3 Announces Object Expiration Amazon S3 announced a new feature, Object Expiration that allows you to schedule the deletion of your objects after a pre-defined time period. Using Object Expiration to schedule periodic removal of objects eliminates the need for you to identify objects for deletion and submit delete requests to Amazon S3. You can define Object Expiration rules for a set of objects in your bucket. Each Object Expiration rule allows you to specify a prefix and an expiration period in days.The prefix field (e.g. "logs/") identifies the object(s) subject to the expiration rule, and the expiration period specifies the number of days from creation date (i.e. age) after which object(s) should be removed. Once the objects are past their expiration date, they will be queued for deletion. You will not be billed for storage for objects on or after their expiration date.

Here is another related: Managing Lifecycle Configuration.

This is my desktop... 5 years ago.

2014-08-15 by gihnius, tagged as freebsd, linux

Can you guess which OS?

Emacs, Firefox, Xterm, Rxvt, Fcitx...

and FreeBSD!

A custom power management settings

2014-08-14 by gihnius, tagged as mac

pmset is a command line tool that apple has bundled in their OS X operating system. Its purpose is to display and/or modify the power management settings. Here is a modified settings in order to: 1. Save battery power as much as possible; 2. let you return to work as soon as possible.

battery mode (pmset -b)

Write the hibernation image to disk and powering off memory for Standby after 6 hours.

pmset -b standbydelay 21600

Sleep after 10 minutes idle.

pmset -b sleep 10

Spindown disk after 30 minutes idle.

pmset -b disksleep 30

Set hibernatemode to 3. The system will store a copy of memory to persistent storage (the disk), and will power memory during sleep. The system will wake from memory, unless a power loss forces it to restore from disk image.

pmset -b hibernatemode 3

Turn down display after 5 minutes idle.

pmset -b displaysleep 5

Poweroff after 8 hours to save power.

pmset -b autopoweroffdelay 28800

charger mode (pmset -c)

In charger mode, I would like to change hibernatemode to 0. So that the system can be waked up quickly. In this mode, the system will not back memory up to persistent storage. The system must wake from the contents of memory, the system will lose context on power loss.

pmset -c hibernatemode 0

Standby, the manual said, standby only works if hibernation is turned on to hibernatemode 3 or 25. Here I set it the same as battery mode

pmset -c standbydelay 21600

Sleep after 10 minutes idle.

pmset -c sleep 30

Turn down display after 10 minutes idle.

pmset -c displaysleep 10

Spindown disk after 60 minutes idle.

pmset -c disksleep 60

Poweroff after 12 hours to save power.

pmset -c autopoweroffdelay 43200

Show custom settings for all power sources (pmset -g)

pmset -g custom

Put together and save to a script.

#!/bin/sh

## battery mode
pmset -b standbydelay 21600
pmset -b sleep 10
pmset -b disksleep 30
pmset -b hibernatemode 3
pmset -b displaysleep 5
pmset -b autopoweroffdelay 28800
## charger mode
pmset -c standbydelay 21600
pmset -c sleep 30 
pmset -c hibernatemode 0
pmset -c displaysleep 10
pmset -c disksleep 60
pmset -c autopoweroffdelay 43200

pmset -g custom

for more detail about pmset, please check out the manual page of pmset by man pmset.

tested on osx 10.8 - 10.9.4

Setup a DNS cache server using dnsmasq

2014-08-08 by gihnius, tagged as internet

Dnsmasq is a lightweight, easy to configure, DNS forwarder and DHCP server. It is designed to provide DNS and optionally, DHCP, to a small network. It can serve the names of local machines which are not in the global DNS.

apt-get update
apt-get install dnsmasq

if installed successful, test it by:

dig google.com @localhost

Setup local caching

Edit /etc/dnsmasq.conf (default location in debian linux).

#listen-address=0.0.0.0 ## default 
port=53

no-poll

no-resolv

#forwarding google dns and opendns
server=8.8.8.8
server=8.8.4.4
server=208.67.222.222
server=208.67.220.220

cache-size=1024

neg-ttl=7200

max-ttl=86400

interface=eth0

interface=lo0

In order to configure dnsmasq to act as cache for the host on which it is running, put nameserver 127.0.0.1 in /etc/resolv.conf to force local processes to send queries to dnsmasq.

in /etc/resolv.conf:

nameserver 127.0.0.1

then restart dnsmasq. /etc/init.d/dnsmasq restart.

DNS (UDP) tunneling by SSH with socat.

2014-08-08 by gihnius, tagged as internet

Intro

In China, many "ISP" sucks. Their DNS servers often return incorrect ip address results, is known as DNS poisoning! DNS poisoning is a common and simple way to stop people reaching correct web pages.

Here is a solution to get the correct DNS queries results.

Dependent tools

Server
- A VPS server that can access famous public DNS servers correctly, eg. 8.8.8.8 (google dns) or 208.67.222.222 (opendns).
- SSH server running on that VPS. (Please google: how to setup ssh server)
- socat (Socket Cat). (Please google: how to setup or install socat)
- dnsmasq (Optional, for caching).
Local
- SSH client
- socat (Socket Cat)
- dnsmasq (Optional, for caching).

ssh, socat, dnsmasq are open source softwares which can be found and installed easily.

Samples and Steps

Server
- Setup a DNS caching server using dnsmasq. (Optional)
  - install dnsmasq
  - configure example using google dns and opendns servers. please check out: Setup a DNS cache server using dnsmasq
  - start dnsmasq
- If no local dns server, just use a public dns server instead, eg. 8.8.8.8:53
- Forwarding UDP to TCP by socat (listen on port: 15353)
  - install socat
  - start socat:
    - if use a public dns server, eg. 8.8.8.8:53
      socat tcp4-listen:15353,reuseaddr,fork,bind=127.0.0.1 UDP:8.8.8.8:53
    - if use local dns caching server: 127.0.0.1:5353
      socat tcp4-listen:15353,reuseaddr,fork,bind=127.0.0.1 UDP:127.0.0.1:53
- You can check the forwarding dns server using command line:
  dig +tcp google.com @127.0.0.1 -p 15353
Local
- Setup SSH tunnel
  ssh -N -L 15353:localhost:15353 username@vps.ip
- Forwarding TCP to UDP by socat
  - if no local dns caching server, you can forward to port 53
    socat udp-recvfrom:53,reuseaddr,bind=127.0.0.1,fork tcp:127.0.0.1:15353
  - of cause can forward to any port that can be used.
    socat udp-recvfrom:15353,reuseaddr,bind=127.0.0.1,fork tcp:127.0.0.1:15353
- Setup local dns caching server (Optional but recommend). See the server instruction above.

OK!

Oh not yet!

ssh (tunnel) is not always working well! WTF!

Gih's Blog