Tip for using ddclient on AWS EC2 instances

I have a long-running EC2 instance on which I host a few simple services. On this instance I configured ddclient to update my public IP for my no-ip hostname.

Recently I started getting warnings from no-ip that if I don't update the IP, my hostname will be deleted.

After checking the logs on my instance, I found this:

Jun 29 13:04:09 hostname ddclient[1676]: WARNING:  cannot connect to checkip.dyndns.com:80 socket: IO::Socket::INET: Bad hostname 'checkip.dyndns.com'

I googled for that error message and found several bug reports, mostly from last year, and the bug was supposedly fixed.

Regardless, I was using dyndns to check for my IP, and updating my ip on the no-ip service. That feels kind of wrong. I wanted to kill two flies with one blow, so I tried using EC2 metadata to update the IP.

My new /etc/ddclient.conf:

protocol=noip

use=web, web=http://169.254.169.254/latest/meta-data/public-ipv4

server=dynupdate.no-ip.com

login=<mylogin>

password='mypassword'

myhostname.no-ip.org

The only thing changed in my config is this:

web=http://169.254.169.254/latest/meta-data/public-ipv4

Testing time:

service ddclient stop

ddclient 

here...<mylogin> --> <REDACTED>

SUCCESS:  updating myhostname.no-ip.org: good: IP address set to <my-instance-public-ip>

service ddclient start

A success story of cloning Windows 7 from SSD to HDD using Linux

BIOS Drive0 - Kingston / SSD / 120GB
BIOS Drive1 - WD / HDD / 160GB

Both drives are SATA, and both are configured as AHCI in BIOS:

• Kingston SSD: This was the primary, bootable drive which held Windows 7.
• WD HDD had a test Linux distribution with LVM, which I did not use or need. It was ready for nuking down.

I needed the SSD for Linux - to use as my primary OS. Also, I needed that computer to stil boot Windows by default when I'm not around, for other people using it. At the sime time I wanted to avoid reinstalling Windows, because this installation was perfectly fine. Seeing that the disks were similar in size, I wanted to clone SSD to HDD, and then boot Windows off the HDD. I would only boot from the SSD when I choose the boot device at BIOS startup. I did not want to bother with maintaining a bootloader across disks because I might move that SSD to another computer one day. Best to keep those two hard drives and their OS/bootloader separate.

My weapon of choice when doing low level disk stuff or recovery is SystemRescueCd. It's a nice little gentoo-based bootable live CD/USB that has most of the tools you would ever need for recovery tasks.

While booted into SystemRescueCd, the SSD would be sda, and the HDD would be sdb.

A little digression: I used to check for physical devices by using fdisk -l. This is no longer the case. The reason is because fdisk -l also prints a LOT of spam, especially if you use LVM. Lack of support for GPT partition tables is also another reason. Sure, grep is always handy, but the more elegant solution is to use parted. The output below is from my laptop, not from the system I was working on at the time.

# parted
(parted) print devices
/dev/sda (320GB)
/dev/mapper/sys-root (26.1GB)
/dev/mapper/sys-swap (2047MB)
/dev/mapper/sda5_crypt (114GB)
(parted) quit
#

I find this list to be much more readable. The drawback is that you can't have a universal oneliner. I always use this method, because 'parted /dev/sda print devices' might not work on every system.

In some of my previous experiences, I would 'dd if=/dev/zero of=/dev/drive-to-nuke bs=1M count=20' (nuke 20MB where disk begins). This would, of course, destroy the partition table and MBR. However, when you re-create the partition table (at least with fdisk), the LVM data creeps back up. I did not go in depth into researching why this is, but rather undo any LVM config on that drive before nuking.

This is why for the HDD, I first removed all lvs/vgs/pvs, and then recreated the partition table using parted. I'm not saying this is the correct way to do this, or that parted wouldn't have done this automatically. I was just using the method I know works, because of lazyness :)

After I've prepared the HDD, it was time:
dd if=/dev/sda of=/dev/sdb bs=1M

I could have used bs=4096, to match the block size of both drives, but I just used that times 256, to see if the performance would be different. It turns out that the performance is the same. Disk dump speeds ran at about 50-60MB/s, which was the maximum for the HDD writes.

Afer the dump. I've turned off the computer and used the BIOS-select-boot-device at startup to use the HDD.

This is when something interesting happened: Windows started booting off the HDD (that WD drive churns like crazy), and then the drive got silent, but Windows kept booting. Weird. I logged in... pretty fast. No peep from the HDD. This required investigation. I opened the "Disk Management" in Windows, and saw that the HDD was "ignored" because it had the same ID as the SSD. My guess is that the kernel started loading from the HDD, and then switched over to loading from the SSD once it found a problem, or because they had the same ID.

I shut down the computer, and disconnected the SSD. When I turned the computer back on, the boot sequence completed, HDD churning all the way, and I was able to log in. Success! The only problem was that the hard drive was heavily fragmented (20% according to Windows), which contributed to even more noise from the HDD. After the defrag, which took a couple of hours, the noise from the hard drive reduced significantly.

After the successful test, I installed Linux on the SSD - Debian Jessie. I hit a snag there. Grub did not do it's job at the end of the Debian installer. I had to once again resort to SystemRescueCd, chroot and re-run grub-install.

As a finishing touch, I swapped the SATA cables. I wanted the HDD to be Device0 in BIOS. The priority can be changed in BIOS, but I like things being as low level as  possible. Booting into Windows, and looking at the "Disk Management" no longer ignored any of the drives, because our SSD had a brand new partition table, and non-Windows partitions.

Early 90s graphics card helped me reduce noise on my headless server

At home, I have this Debian Wheezy headless server - a Dell, not sure about the model. It has a P4 @ 3.0GHZ, 1GB ram, and I threw in 2 working hard drives in there. One of them is system/storage, the other one takes rsync backups.

It also had a Nvidia Geforce 4xx card, with passive cooling. I sold that to a guy who needed that card, and he gave me a semi-working AMD card from the same era. The picture wobbles on that one at higher resolutions, and the cooler is kind of noisy.

Lately, the noise started to get to me, and I decided to dive into my boxes of leftover hardware. I managed to find a S3 Trio 64v+ similar to the one pictured below, and I threw it in, to see if it would work. 

I plugged my monitor in, booted the system with only that graphics card, and the kernel started to boot, but as soon as modules started loading, i lost my picture. Figured it probably has something to do with an old module loading or something similar.

Lucky for me, the system was actually running, and I could SSH in and investigate. The only video related module that was loaded was i915, but i'm pretty sure i need a module that is called s3<something>. I wanted to try and blacklist the i915 module, so I opened up my /etc/modprobe.d/fbdev-blacklist.conf . To my surprise, the s3fb was commented. I uncommented that guy, rebooted, and everything worked :)

No wobbly picture 
No noise

Great success

Edit:

As it turns out, this home server can run pure headless without a graphics card! Good people of Reddit, specifically /u/ghost_upstairs, insisted that it was possible, while I was almost certain that this cannot be done on common PC hardware. I'm glad I was wrong!

Installing Debian Wheezy from a Debian rescue environment using debootstrap

Intro

You might find this guide useful if you need complete control over your Debian install process, but your housing provider can only provide you with rescue mode boot. For example: your housing provider has no Debian install images which do LVM by default. If they do offer some kind of rescue system (especially if it's Debian) then you are in luck!

This guide has been tested with Debian Wheezy 7.0.0 net-install image (Rescue)

Notice

I'm not responsible for any and all damage done by following this guide. A fair level of GNU/Linux familiarity is expected of the reader. Extra Debian knowledge can't hurt either. Practicing this guide on a VirtualBox guest (with bridged networking) before trying it live is strongly advised.

Prerequisites

You are somewhat experienced with GNU/Linux and familiar with Debian installation process. You read manpages before you Google, and have at least heard of debootstrap.

You know your networking configuration for your machine in question (Virtual or otherwise).

You know that if you mess up, you might end up being cut off from your remote machine.

Ok? Let's start.

Debian rescue environment

If you are trying this on VirtualBox/VMware, you need to boot your CD. When the menu pops up, choose: Advanced Options -> Rescue Mode

In case you have a hosting provider, you might just be given access to it straight away.

Debian installer options

Configure your networking via prompts

Locale, and other options are not relevant and won't be included in our newly installed system. We will configure almost everything by hand.

You might get a warning that there are "No partitions found". Either way, make sure you pick: "Execute a shell in the installer environment"

Partitioning

The following partitioning scheme will be used for the purpose of this guide:

/dev/sda1, type Linux, size 512MB

/dev/sda2, type LVM, size <everything else>

As I said in the introduction, I expect the reader to be somewhat GNU/Linux ready. This means that you know how to use fdisk to do this basic type of partitioning.

To get you started:

fdisk /dev/sda

Press "h" for help, read the manpages, and use Google as your last resource (for examples)

LVM

Just like partitioning, this LVM setup will be used for the remainder of this guide

vgcreate mainvg /dev/sda2

lvcreate mainvg -L 1G -n root

lvcreate mainvg -L 1G -n swap

lvcreate mainvg -L 1G -n usr

lvcreate mainvg -L 1G -n tmp

lvcreate mainvg -L 1G -n srv

lvcreate mainvg -L 1G -n var

lvcreate mainvg -L 1G -n home

You can adjust the sizes (1G), volume group name (mainvg) and logical volume names (root,swap...) as you see fit... example:

lvcreate reallycoolvg -L 512M -n myhome

File systems

mkswap /dev/mainvg/swap

mkfs.ext2 /dev/sda1

mkfs.ext4 /dev/mainvg/root

mkfs.ext4 /dev/mainvg/usr

mkfs.ext4 /dev/mainvg/tmp

mkfs.ext4 /dev/mainvg/srv

mkfs.ext4 /dev/mainvg/var

mkfs.ext4 /dev/mainvg/home

You can use whatever filesystem you like. Just be careful to use whatever partition names you chose before and to set it accordingly in fstab later!

We have to mount root first

mount /dev/mainvg/root /mnt  

Basic dir structure

mkdir /mnt/boot

mkdir /mnt/usr

mkdir /mnt/tmp

mkdir /mnt/srv

mkdir /mnt/var

mkdir /mnt/home

Activate swap

swapon -a

Mount our partitions

mount /dev/sda1 /mnt/boot

mount /dev/mainvg/usr /mnt/usr

mount /dev/mainvg/tmp /mnt/tmp

mount /dev/mainvg/srv /mnt/srv

mount /dev/mainvg/var /mnt/var

mount /dev/mainvg/home /mnt/home

Bootstrap

We want locales set up, unless you like seeing Perl errors everywhere. Those editors (vim, joe) are just my preference. You will have nano installed by default.

Use whatever working mirror you like. Don't go overboard with the initial packages. Better save your list of favorite packages for later, when you can copy and paste with ease.

debootstrap --include=openssh-server,vim,joe,locales wheezy /mnt http://ftp.de.debian.org/debian

Chroot

cd /mnt

mount -t proc proc proc/

mount -t sysfs sys sys/

mount -o bind /dev dev/

mount -t devpts pts dev/pts/

chroot /mnt /bin/bash

Basic stuff

Root password:

passwd

Create your non-root user:

useradd -c "This is my account" -m -s /bin/bash -d /home/userguy userguy

Don't forget to set a password

passwd userguy

Optional

vim /etc/ssh/sshd_config

Find the following directive and set it like this:

PermitRootLogin no

Now that we are somewhat safer, it's time to start sshd in our chroot environment, so we can copy and paste with ease.

service ssh start

Now it's time to SSH in. Use the "userguy" user you created earlier, and switch to root:

su -

Initial system configuration

Let's edit our sources.list. If you know what you are doing, edit this to your satisfaction:

vim /etc/apt/sources.list

This is the contents of our example content:

deb http://ftp.de.debian.org/debian/ wheezy main contrib non-free

deb-src http://ftp.de.debian.org/debian wheezy main contrib non-free

deb http://security.debian.org/ wheezy/updates main contrib non-free

deb-src http://security.debian.org/ wheezy/updates main contrib non-free

We need to configure our locales. If you don't know what to pick, then go with en_US.UTF-8. You will get a second question, and again select your desired locale (en_US.UTF-8)

dpkg-reconfigure locales

Configure your timezone:

dpkg-reconfigure tzdata

Set your hostname

vim /etc/hostname

We configured networking for our rescue environment, but those settings don't get picked up by debootstrap. You need to configure your network manually here.

Use your own IP address / subnet / gateway

vim /etc/network/interfaces

Example content:

# interfaces(5) file used by ifup(8) and ifdown(8)

auto lo

iface lo inet loopback

auto eth0

iface eth0 inet static

        address 192.168.1.10

        netmask 255.255.255.0

        gateway 192.168.1.1

Set your DNS servers here

vim /etc/resolv.conf

Content:

nameserver 8.8.8.8

nameserver 8.8.4.4

Edit your fstab

vim /etc/fstab

Content:

# /etc/fstab: static file system information.

#

# Use 'blkid' to print the universally unique identifier for a

# device; this may be used with UUID= as a more robust way to name devices

# that works even if disks are added and removed. See fstab(5).

#

# <file system> <mount point>   <type>  <options>       <dump>  <pass>

/dev/mapper/mainvg-root  /   ext4 defaults                 0       1

# /boot was on /dev/sda1 during installation

/dev/sda1                /boot  ext2         defaults      0       2

/dev/mapper/mainvg-home  /home  ext4         nosuid,nodev  0       2

/dev/mapper/mainvg-srv   /srv   ext4         nosuid,nodev  0       2

/dev/mapper/mainvg-tmp   /tmp   ext4         nosuid,nodev  0       2

/dev/mapper/mainvg-usr   /usr   ext4         defaults      0       2

/dev/mapper/mainvg-var   /var   ext4         nosuid,nodev  0       2

/dev/mapper/mainvg-swap  none   swap         nosuid,nodev  0       0

/dev/sr0        /media/cdrom0   udf,iso9660  user,noauto   0       0

You might consider using UUID= for your swap partition. Use blkid /dev/mainvg/swap

Let's update our system

apt-get update && apt-get upgrade

Install some basic stuff

apt-get install lvm2 grub firmware-linux-nonfree firmware-linux-free sudo ntp linux-image-amd64 less

Note:

"firmware-*" is optional, but some hardware components might need it. 

"sudo", "less" and "ntp" are also optional. You can get rid of them if you don't need them.

Use the appropriate kernel architecture for your arch.

GRUB

Note: I hope you tested this procedure on a VirtualBox/VMware box first. If you have other partitions on a live system, be VERY careful how you type the following commands:

mkdir /boot/grub

update-grub

grub-install --root-directory=/ /dev/sda

Log out from your SSH session

Log out from your chroot environment

Reboot from rescue shell

Don't boot from CD

Hopefully, you are now looking at your Wheezy loading up. 

Post 1st boot

Take notes of any errors. 

As soon as the system boots, ssh in and check system logs for errors:

less /var/log/syslog

Check if all mounts are good:

mount

Check if swap is there:

free

Some people like to see what else they can configure with dpkg-reconfigure:

Run this command if you'd like to know what else you can reconfigure with it:

ls /var/lib/dpkg/info/*.templates|xargs -n 1 basename|sed -e "s/.templates$//"

Get your firewall up and running, and do the stuff you usually do.

I'm happy to take any suggestions and further improve this guide on your input.

Thank you for reading :)