Bash 101

So are you new to Linux?  Are you the kinda person who's like "Awesome, Linux!" and you're also like "Awesome, Command Line Interface (CLI)!", but you're also like "I don't know how to use the command line effectively and desire to know more on this topic in order to advance my knowledge of the Unix world!"

If so, this article is for you.  It's really just to get you started.

This isn't really a technical guide.  This is a list of practical tips that, looking back, I wish I had seen somewhere when I started to make myself use the CLI on a regular basis.  Once you get used to the CLI, you start to use it a lot; on my Linux box, the only window I usually have open that isn't a terminal is Firefox.  Everything else - with a few exceptions - can, in my opinion, be done effectively from the command line.

So here's my list of tips for the CLI beginner:

1. Make use of auto-completion
This is a small tip, but useful if you don't already know it.  If you're using the bash shell, you can usually press [ Tab ] to do this.  In other words, say I have a directory I want to cd (change directory) to:

• user@localhost:~$ ls
  
• Directory_1 Directory_2 Directory_3 Other_Directory
  
• user@localhost:~$ cd Directory_2
  

Instead of taking the extra few seconds to type in "Directory_2", you can simply start the word and press tab.  In this case, suppose you type "Dire[ Tab ]".  As soon as you press [ Tab ], the shell will complete the word "Directory_" for you.  It won't, of course, put the number, because there's more than one choice.  If you had typed "Oth[ Tab ]" or even just "O[ Tab ]", it would completely fill the word "Other_Directory" for you.

This same tip will work if you are starting to type an executable name in your $PATH (the variable that knows where to find all of your executable files).  Suppose you wanted to run the program gnome-system-properties from the command line (which would actually launch a GUI, but can be useful anyway).  To speed things up, type a few letters and hit [ Tab ].  Bash will complete the string as far as it can until there is any uncertainty, which probably means up to 'gnome-'.  Perhaps you've forgotten the exact name of what you're trying to run?  Hit [ Tab ] again and you'll be given a list of possible completions.  Start typing again hit [ Tab ] to complete if you get lazy.

You will still, however, have to press [ Enter ].

P.S. - If this isn't working, you might not be using the bash shell (type set | grep SHELL or something reasonably equivalent to find out what you're using).  This doesn't mean all hope is lost.  If you're just in plain old sh, for example, you can use the wildcard character '*' to complete your phrase.  In the first example above, then, you could type 'cd Dire*2' to go to Directory_2.

2. Understand I/O Redirection
As many people will tell you, piping and I/O redirection is perhaps the most powerful part of the nature of the CLI.  Take this example:

Suppose you had to build a toy car.  You need someone to carve the right shape, someone to put on the wheels, someone to screw everything in, and someone to put it in the correct package.  In one case, you could give the car to the person who carves it, and then take it back.  You would put the car in a box and go find the person to put on the wheels.  When they were done, you would package it and take it to the person adding the screws.  Finally, you would put it back in a package and take it to the person who puts it in the package for marketing.

In another example, though, you could setup an assembly line and use a conveyor belt.  This removes your role as the messenger and removes the intermediate packaging.  This second example is analogous to the CLI.  Whereas in the GUI example one has to use a program, save the result, and then open a whole new program, the CLI-master can simply tell each program in line to pass on their result to the next program.  You might write, for example:

• user@localhost:~$ carver | wheeler | carpenter > correctBox
  

Here, the carver program sends it's output to the wheeler program which sends it to the carpenter program which then writes all of its output to the file correctBox.

As a more useful example, consider this command in my .bashrc file:

• alias dl = 'ls -laht | sed -n /^d/p'
  

After the next two tips, you'll understand what that does.

3. Use Stream Editors
More specifically, I mean that you should try to understand the power of commands like sed and grep.  Programs like these can help you get the data you want from other commands.  Say, for example, that I wanted to find all of the C source files in my ~/Code directory.  I might write something like:

• user@localhost:~$ ls Code | grep '\.c'
  

First of all, do you understand why simply grep .c doesn't give the same results in general?  If not, we'll get to that in a second.  Right now just realize that the period is a special symbol (a metacharacter).

The above command does this:  first, it uses the ls command to print out all of the regular files and folders in the directory.  The grep command then receives these filenames (just the names, not actually the files) as they are generated, and it selectively decides which ones to output.  In this case, it only prints out the strings which contain the substring '.c', which is more-likely-than-not a C source file.  Of course, if you happen to have a file called "a.b.c.d.e.txt", it will catch this as well.

For dealing with more advanced structured, we can use sed (among other tools); that, however, is a topic beyond the scope of this article.  As a basic primer, it can be said that sed usually operates on text using regular expressions, a powerful tool that the Perl hacker may already be familiar with.  This is what it does in the above example; for everything output by ls -laht (which, by the way, is a listing of the current directory in Long form with All files listed (as opposed to only non-hidden ones) in Human-readable file sizes, sorted by Time modified), sed only lets through lines that start with the letter 'd', which, using ls's long form, means that they are directories.  That's why I called this alias dl - it stands for directory listing (in my head, at least).

It should be noted that this is only a basic usage of sed, and in fact this exact same thing can be done with grep (ls -laht | grep ^d).  That '^' in both the sed and grep expressions is called a metacharacter.  Just like the period that we were searching for in the first grep example above, we would have to backslash-escape the character like this if we actually wanted to search for the character '^':

• user@localhost:~$ ls -laht | sed -n /\^d/p
  

In this case, '^' would no longer mean 'beginning of line'; instead, it tries to find the caret symbol within the inputted text.

4. Force yourself to use the CLI by trying to do everything with it until it gets to the point where you know where you want to be.
This is an interesting point that other people might not suggest, but I did this and it seemed to work.  If you really do have a desire to use the CLI but don't know how, just try forcing yourself to use it.  I did this by installing Fluxbox as my desktop environment and making the only two things in my menu 'firefox' and 'gnome-terminal'.  I use Alpine for my mail, VIM as my editor, Irssi as my IRC client, CenterIM as for Google Talk, and then I usually have one or two other terminals open just for kicks.  One probably has Nethack open.  The only non-terminal based program I use in Linux is Firefox, because the the Web really has evolved beyond the reasonably capacity of, say, Lynx (some might argue that point).  I also use VLC occasionally to watch movies, which obviously can't be done in the CLI.  I used to even use the CLI for playing music, but since my music started being centralized on my Mac, I usually just plug my speakers into my iPod.

So I suggest you do something similar - just keep Googling things like "best CLI programs" or "favorite terminal applications" until you know a reasonable amount about what's out there (and what's good) for the command line.  Make yourself use them for a month and see how you feel.  I decided not to go back (although I can't say that very well now that I have a Macbook, but it was true for a long time up until a few months ago).

Best of luck!

5. Links & Resources

Bash Beginner's Guide
Advanced Bash Scripting
Linux Shell Scripting Tutorial
CLI Apps
Favorite Command Line Programs

SCTA-1

Since this is my personal blog, I thought I should share a bit about a personal project.  Sherwin and I have just finished what we think is a stable build of our encryption algorithm SCTA-1 (Stream Cipher Test Algorithm 1).  You can read about and download it at our SourceForge.net site.

Don't trust it for your encrypted data just yet though.  We're going to see if we can find any cryptanalysts willing to find weaknesses before we get too excited.

It's definitely been a fun project to work on, though.  Lots of brainstorming, mathematics, and even some physics.  I wouldn't mind doing something like this for a living.

f[8,9,10,11] user guide

We (the Fedora Documentation team) just finished reorganizing the User Guide for Fedora 8 (we're kinda behind).  It can be found at https://fedoraproject.org/wiki/F8_User_Guide.

I recently volunteered to take the lead on the User Guide, and one thing we're trying to do is catch up to F10 while also cleaning up our style and organization (this is something the entire team is looking at for all our docs).  One thing that we are particularly pushing is new user friendliness.

So if anyone is willing to, please take the time to look at that user guide above, browse through a few sections, and send me some suggestions on... whatever.  Does it say something that perhaps you have no clue about?  Something that an experienced Linux user writing it might understand, but that you, the new user, can't decode?  Help me out here by showing me the weak points in our communication.

And hey, if anyone is feeling really volunteery, you can join the Fedora Docs team and help edit these documents.  No experience is necessary, and in some cases (like making things more new-user friendly) a lack of experience can be pseudo-desirable.  Just let me know if you're interested.

music out of math

So one day, I was talking to this guy about coding projects I've been doing, and out of nowhere an idea popped into my head.  Anyone who has ever seen the electricsheep screensaver project knows how awesome that is, and I realized that perhaps the same thing could be done with music.  I'm sure people have tried this before, but I've never seen any good results or examples, so now it's my turn to try.

So I talked to my trusty friend Sherwin, with whom I attack all random coding projects.  In the past, we've tried things like Sifu (for using reliable proxies distributed across a torrent-like network in order to bypass school internet restrictions), libtion (a framework for implementing abstract network structures), and Craft (an AJAX delivered course management system; I'm actually still working actively on this one, but on my own because Sherwin doesn't like Java), among other things.

So this time, we're writing our project as follows (so far): we've envisioned several abstract structures, namely signals, engrams, and sequences.  A signal is simply a waveform.  An engram is a set of signals (like in the figure above, shamelessly taken from a random page I googled) with a set duration, and a sequence is just all the engrams in a row.  Since we know so little about music theory, we're trying to make them pretty mathematically rooted:

Signal: s ≡ ( ν, A, Φ )

Engram: E ≡ { s₁, s₂, s₃, ... ,  s_n },

and E has a property of length in time E_l

Sequence: S ≡ < E₁, E₂, E₃, ... ,  E_n >

From here, we have some pseudo-mathematical descriptions of methods we will use to mutate our initial sequence and its descendants so that they can undergo natural selection against each other.  These mutations include signal-base mutations, engram-base mutations, and sequence-base mutations; basically, each type of mutation affects the type of object for which it is named.  So signal-base mutations affect signals themselves; this means that their properties, namely frequency, amplitude, and phase shift (as seen in the ordered triplet defined above) are fundamentally altered.  Examples of some engram-base mutations include insertion (where a signal is added to an engram), deletion (the opposite of insertion), propagation (where a signal replicates and spreads to nearby engrams), and nondisjunction (where a signal is moved from one engram to an adjacent one).  Sequence base mutations involve adding engrams, deleting them, moving them around, or inverting the order of some partition of engrams.

The plan is to start with a single waveform - that's to say, a single signal with some defined parameters which is the lone signal in a lone engram in a lone sequence - and then let it "reproduce" via fusion where both of its children are run through mutation engines.  The population will then be played for human observers who will rate each sequence according to its relative beauty, and the most favored sequences will have more chances to reproduce (just like in natural selection).

Perhaps in a year or so I'll have my hands on a pretty awesome song I can market.  Sherwin and I are willing to take investment money right now in return for a percentage of the profit we make off of our world-famous scores of the future.  Donations can be made via hard cash or checks, or in some material form that's useful to us (like servers, Star Trek episodes, dice, coffee, swords, or other useful stuff).

when you give a moron a printer...

There's a serious issue facing us these days. There's too many people who are computer illiterate.

Now, let me be quite frank about this. I don't give a fuck if some kid knows how to use Microsoft Excel. It'd be great if he did, especially if he's in my chemistry lab group, but to be quite realistic you can probably get by without knowing if you had to. And if you needed to learn, well, you could learn.

In that case, ignorance is okay. It's not like people can go around using Excel haphazardly, and it's not like they can mess anything up too bad by trying to use it.

When I speak about computer literacy (right now, at least), I'm talking more understanding the basic nature of how a computer, and especially a network, operates (the latter may be more important, in actuality). I say this - and I'm writing this post - because I had to clean up for some poor girl. You see, I came to the library - which is relatively empty (relatively) on a Friday afternoon - to print out some notes for Chemistry last week, as well as a paper my dad wanted me to proofread.

So I hit print, and go over to the printer. The time is 5:25.

Well, lo and behold, the printer is out of paper. No big deal... I load in some more paper and printing resumes.

The printing resumed with documents printed at 4:45.

Can nobody else load a fucking paper tray?

You know what, though? I can deal with it. I don't really care if people lost their documents. Yeah, I'm sorta upset that I had to wait 5 or 6 minutes for my document to be printed, but whatever. I'm pretty disgusted that nobody else can load paper into a printer, but whatever.

As an act of good will, in case these kids ever want to come retrieve their printouts, I decided to sort them all and lay them out on a table.

All documents across a Novell network are printed with cover sheets showing the username, time of printing, and document source. It helps a lot when people are swarming over the printers at lunch hour on Monday. But sorting them, I notice this one girl who printed out the same document thrice. I glance at the document source... the first thing I notice is that it's a secure site. The second thing I noticed was that it was from carolinafirst.com.

The last thing I noticed, when I turned the page, was that it contained her full name, address, phone, email, social security number, mother's maiden name, account listings and account numbers.

So I decided to sell it all online!

Of course not, but I know kids who might. So here's a lesson to everyone:

(1) To load a printer, open the drawer and put the paper in the container that's conveniently shaped like paper.

(2) If you execute a print command multiple times, whether you're using lpr or just clicking an icon in MS Word, your document will be printed multiple times. Computer's aren't hard of hearing; they do exactly as you command. Just be patient while they do things one at a time.

(3) Don't print information like your social security number and bank number on a public printer over a public network; and if god forbid that you do, make sure the fucking paper is in your hands before you leave!

As a side note, I went to carolinafirst.com. I would like to make it well known that I could have gotten into her account. In fact, I suppose it stands to reason that I still can, now that I have her information. But being the good person I am, I'm going to rip the information up like I did to the other two copies and throw it away. She'll get away with it this time.

But if I find that banking information again, I may just give her a bit more of a scare.

perlmwdtut(0)

A tutorial for those new to Perl but not to Programming: Part 0

Introduction

When you don't have much to blog about, you start trying to think about things you already know.  Well... I know Perl, and I have some friends who would like to learn it, or at least be comfortable with it.  I'll present this tutorial in progressively more advanced parts, showingcasing a little at a time the high and low points of Perl and how to use it.

What is Perl?

Perl is lots of things to lots of people, but to me - and many other people - it's a swiss army knife language (if you will).  When you pull a bunch of random data out of an excel spreadsheet and you need to do some stuff with it real quick (like, maybe, remove invalid email addresses and then send a mass message), you don't compile a huge C application and you don't need to power up your Java VM.  You just write a few lines of Perl.

Perl has had other purposes, too.  Sometimes referred to as the glue that holds the web together, Perl has been used extensively throughout the years in CGI scripting, although it is my believe (but not quotable experience) that it is falling out of use with the evolution of the internet into a far mor

e interactive and m

ultimedia-oriented experience... although I also know that there is still plenty, plenty, plenty of Perl out there.

Perl is also great for just hacking things together and making things work.  If you had a round peg, a square hole, and a pocketknife, Perl would be that pocketknife.  Observe this comic from xkcd.com (released under Creative Commons; kudos to Randall Munroe for his amazing webcomic):

(Click for larger view.)

Anyway, that's about what Perl is.

Jumping Right Into It

No tutorial would be complete without...

#!/usr/bin/perl

use warnings;

use strict;

print "Hello, world!";

There's our familiar Hello, world! program.  The first three lines are unnecessary, generally.  You could write the program as simply print "Hello, world!";, execute perl and it would work just fine.  But if you're curious, here's what those first lines are all about.

Line 1: #!/usr/bin/perl

This line is often called a shebang, and is common to many Unix scripts.  In fact, if you go find a bash script on your computer, chances are that the first line is something like #!/bin/sh.  That's because when you run chmod +x script.pl to turn your script into an executable file, it looks as that first line to determine what needs to be called to run it.

Of course, if you're running Windows, none of this will make any sense.  If, for some reason, you are in a Windows system, this first line won't help you out much.  Just run your scripts using the perl syntax and everything will be okay.  Depending on your Perl interpreter (I know that ActivePerl works this way), you could also just double-click on your script in Explorer and it should run.  Be wary that if you're script isn't actually doing anything functional - for example, if it's just the Hello, World! script above - then you probably won't see anything happen besides a terminal appear for a few milliseconds and disappear again, because the terminal's process will die along with the Perl script.

But be warned that if you're running Windows, you probably won't be able to use a lot of the things in this tutorial.  Go use Visual Basic or something.

Anyway...

Line 2/3: use warnings; use strict;

The use of the warnings module alerts us when we do silly things.  It won't stop our program from running, but it might let me know if, say, I declared a variable I never used.  Sounds stupid?  Not really.  Perl is extremely nice to its user and tries to do exactly as it is told... which can be a problem.  Say I have the variable $rest = 5; and then later on I mistype it: for ($x=0; $x<$breast; $x++)....  Normally, an interpreter/compiler would catch this Freudian slip (perhaps you had some inappropriate browser windows open while coding), but Perl will do its best to do exactly as you say, so it'll instantiate $breast for you with the default value of 0.  This will not function correctly.  Using warnings will probably give you a little wake up call if you knew this variable was supposed to be in use.  This is only an example, though; the warnings module is extensive and highly useful for debugging.  The use strict flag makes you declare all variables with a some sort of scope modifier, which we'll talk about later, but it is important to note that, unlike many languages, a variable declared in any scope will, by default, become global.  This often causes trouble if not handled with care.

If you can't figure out line 4, you probably shouldn't be reading a tutorial for people who already have some programming experience.  In fact, you should probably understand line 4 whether you have programming experience or not.

Variables

Watch this:

$variable = 5;

$variable = "Hello, Janet!";

$variable = \$referenced_var

$variable = \&referenced_subroutie

$variable = -234.3

$variable = 0x0000003f;

$variable = "Some other data type!";

print $variable; # prints out "Some other data type!"... and yes, this is a valid program.

# By the way, this is a comment!

Perl has a what's called a dynamic typing system.  Several other languages have this as well (Ruby and Python, to name two) but it isn't typical of what you see in C, C++, or Java.  Whereas in those languages you must declare a variable's data type (either beforehand or on-the-fly), Perl variables can hold any type of data, any time... the only restriction is on memory (which, these days, is considerable).

Note: Perl 6 (which will be similar but also considerably different from Perl 5 and earlier) will have an exciting new hybrid typing system which allows you to assign a value if you want to.  For example, you could write the traditional, perlish: $variable = 5;, but you can also write the sometimes useful and more C-ish: int $variable = 5;, which will treat $variable just like C would.

More to come later!