Delete all but the most recent N files(objects) from s3.
When I backup servers and save data to AWS S3, usually just upload entire files to S3. This is done automatic by the backup script. The uploading process looks simple:
S3_PUT backup_data_a s3://org-sys-backup/
S3_PUT backup_data_b s3://org-sys-backup/
S3_PUT backup_data_c s3://org-sys-backup/
S3_PUT backup_data_d s3://org-sys-backup/
S3_PUT backup_data_e s3://org-sys-backup/
S3_PUT backup_data_f s3://org-sys-backup/
...
## or
S3_PUT backup_data_20140801xxxx s3://org-sys-backup/
S3_PUT backup_data_20140802xxxx s3://org-sys-backup/
S3_PUT backup_data_20140803xxxx s3://org-sys-backup/
S3_PUT backup_data_20140804xxxx s3://org-sys-backup/
S3_PUT backup_data_20140805xxxx s3://org-sys-backup/
I don't care the backups' data older than 1 week, then to free the space in S3 I have to delete those old data files (called objects in S3), just keep the most recent 5 copies. There maybe ready-made tools available, but I want to roll my own. I wrote a ruby script to do that, here is source code:
#!/usr/bin/env ruby
## Delete all but the most recent N files from s3://bucket
## supply a prefix and count of copies and the bucket name
require 'right_aws'
require 'optparse'
opts = {}
OptionParser.new do |o|
o.on("-a", "--access-key-id X", "aws ec2 access key id") do |x|
opts[:aws_access_key_id] = x
end
o.on("-s", "--secret-access-key X", "aws ec2 secret access key") do |x|
opts[:aws_secret_access_key] = x
end
o.on("-b", "--bucket X", "bucket name") do |x|
opts[:bucket] = x || ""
end
o.on("-p", "--prefix X", "object name prefix") do |x|
opts[:prefix] = x || ""
end
o.on("-c", "--count N", "how many copies to keep") do |x|
opts[:count] = x || 1
end
o.parse!
end
if opts[:bucket].size < 1 || opts[:prefix].size < 1
puts "bucket or prefix not valid!"
exit 1
end
s3 = RightAws::S3.new(opts[:aws_access_key_id], opts[:aws_secret_access_key])
object_keys = {}
s3.interface.incrementally_list_bucket(opts[:bucket], {'prefix' => opts[:prefix], 'delimiter' => '/'}) do |item|
item[:contents].each{|c| object_keys[c[:key]] = c[:last_modified] }
end
## sort obj keys by time(:last_modified)
sorted_keys = object_keys.sort_by{|k,v| Time.parse(v).to_i}.map{|x| x[0]}
puts "Found #{sorted_keys.size} objects with prefix: #{opts[:prefix]}"
if sorted_keys.size > opts[:count].to_i
old_keys = sorted_keys.first(sorted_keys.size - opts[:count].to_i)
n = old_keys.inject(0) do |a, k|
s3.interface.delete(opts[:bucket], k)
a += 1
end
puts "Deleted #{n} old objects."
end
This script get the objects list which object name match the given prefix, and sort them by modified time, then find the old ones and delete them.
Save it as, eg. s3_cleanup.rb.
How to run it?
You need to install a ruby gem 'right_aws' by: gem install right_aws
, then
ruby s3_cleanup.rb -a 'access-key-id' -s 'secret-access-key' -b org-sys-backup -p 'backup_data_' -c 5
an example output:
Found 8 objects with prefix: backup_data_
Deleted 3 old objects.
This script is available in Github.
You may want to read about Object Expiration before checking out the script. That maybe a official recommended practices.
Amazon S3 Announces Object Expiration Amazon S3 announced a new feature, Object Expiration that allows you to schedule the deletion of your objects after a pre-defined time period. Using Object Expiration to schedule periodic removal of objects eliminates the need for you to identify objects for deletion and submit delete requests to Amazon S3. You can define Object Expiration rules for a set of objects in your bucket. Each Object Expiration rule allows you to specify a prefix and an expiration period in days.The prefix field (e.g. "logs/") identifies the object(s) subject to the expiration rule, and the expiration period specifies the number of days from creation date (i.e. age) after which object(s) should be removed. Once the objects are past their expiration date, they will be queued for deletion. You will not be billed for storage for objects on or after their expiration date.
Here is another related: Managing Lifecycle Configuration.
This is my desktop... 5 years ago.
Can you guess which OS?
Emacs, Firefox, Xterm, Rxvt, Fcitx...
and FreeBSD!
A custom power management settings
pmset is a command line tool that apple has bundled in their OS X operating system. Its purpose is to display and/or modify the power management settings. Here is a modified settings in order to: 1. Save battery power as much as possible; 2. let you return to work as soon as possible.
battery mode (pmset -b)
Write the hibernation image to disk and powering off memory for Standby after 6 hours.
pmset -b standbydelay 21600
Sleep after 10 minutes idle.
pmset -b sleep 10
Spindown disk after 30 minutes idle.
pmset -b disksleep 30
Set hibernatemode to 3. The system will store a copy of memory to persistent storage (the disk), and will power memory during sleep. The system will wake from memory, unless a power loss forces it to restore from disk image.
pmset -b hibernatemode 3
Turn down display after 5 minutes idle.
pmset -b displaysleep 5
Poweroff after 8 hours to save power.
pmset -b autopoweroffdelay 28800
charger mode (pmset -c)
In charger mode, I would like to change hibernatemode to 0. So that the system can be waked up quickly. In this mode, the system will not back memory up to persistent storage. The system must wake from the contents of memory, the system will lose context on power loss.
pmset -c hibernatemode 0
Standby, the manual said, standby only works if hibernation is turned on to hibernatemode 3 or 25. Here I set it the same as battery mode
pmset -c standbydelay 21600
Sleep after 10 minutes idle.
pmset -c sleep 30
Turn down display after 10 minutes idle.
pmset -c displaysleep 10
Spindown disk after 60 minutes idle.
pmset -c disksleep 60
Poweroff after 12 hours to save power.
pmset -c autopoweroffdelay 43200
Show custom settings for all power sources (pmset -g)
pmset -g custom
Put together and save to a script.
#!/bin/sh
## battery mode
pmset -b standbydelay 21600
pmset -b sleep 10
pmset -b disksleep 30
pmset -b hibernatemode 3
pmset -b displaysleep 5
pmset -b autopoweroffdelay 28800
## charger mode
pmset -c standbydelay 21600
pmset -c sleep 30
pmset -c hibernatemode 0
pmset -c displaysleep 10
pmset -c disksleep 60
pmset -c autopoweroffdelay 43200
pmset -g custom
for more detail about pmset, please check out the manual page of pmset by man pmset
.
tested on osx 10.8 - 10.9.4
Setup a DNS cache server using dnsmasq
Dnsmasq is a lightweight, easy to configure, DNS forwarder and DHCP server. It is designed to provide DNS and optionally, DHCP, to a small network. It can serve the names of local machines which are not in the global DNS.
apt-get update
apt-get install dnsmasq
if installed successful, test it by:
dig google.com @localhost
Setup local caching
Edit /etc/dnsmasq.conf (default location in debian linux).
#listen-address=0.0.0.0 ## default
port=53
no-poll
no-resolv
#forwarding google dns and opendns
server=8.8.8.8
server=8.8.4.4
server=208.67.222.222
server=208.67.220.220
cache-size=1024
neg-ttl=7200
max-ttl=86400
interface=eth0
interface=lo0
In order to configure dnsmasq to act as cache for the host on which it is running, put nameserver 127.0.0.1
in /etc/resolv.conf to force local processes to send queries to dnsmasq.
in /etc/resolv.conf:
nameserver 127.0.0.1
then restart dnsmasq. /etc/init.d/dnsmasq restart
.
DNS (UDP) tunneling by SSH with socat.
Intro
In China, many "ISP" sucks. Their DNS servers often return incorrect ip address results, is known as DNS poisoning! DNS poisoning is a common and simple way to stop people reaching correct web pages.
Here is a solution to get the correct DNS queries results.
Dependent tools
- Server
- A VPS server that can access famous public DNS servers correctly, eg. 8.8.8.8 (google dns) or 208.67.222.222 (opendns).
- SSH server running on that VPS. (Please google: how to setup ssh server)
- socat (Socket Cat). (Please google: how to setup or install socat)
- dnsmasq (Optional, for caching).
- A VPS server that can access famous public DNS servers correctly, eg. 8.8.8.8 (google dns) or 208.67.222.222 (opendns).
- Local
- SSH client
- socat (Socket Cat)
- dnsmasq (Optional, for caching).
- SSH client
ssh, socat, dnsmasq are open source softwares which can be found and installed easily.
Samples and Steps
- Server
- Setup a DNS caching server using dnsmasq. (Optional)
- install dnsmasq
- configure example using google dns and opendns servers. please check out: Setup a DNS cache server using dnsmasq
- start dnsmasq
- If no local dns server, just use a public dns server instead, eg. 8.8.8.8:53
- Forwarding UDP to TCP by socat (listen on port: 15353)
- You can check the forwarding dns server using command line:
dig +tcp google.com @127.0.0.1 -p 15353
- Setup a DNS caching server using dnsmasq. (Optional)
- Local
- Setup SSH tunnel
ssh -N -L 15353:localhost:15353 username@vps.ip
- Forwarding TCP to UDP by socat
- if no local dns caching server, you can forward to port 53
socat udp-recvfrom:53,reuseaddr,bind=127.0.0.1,fork tcp:127.0.0.1:15353
- of cause can forward to any port that can be used.
socat udp-recvfrom:15353,reuseaddr,bind=127.0.0.1,fork tcp:127.0.0.1:15353
- if no local dns caching server, you can forward to port 53
- Setup local dns caching server (Optional but recommend). See the server instruction above.
- Setup SSH tunnel
OK!
Oh not yet!
ssh (tunnel) is not always working well! WTF!