UNIX: Shell Programming

• Model
  • data = stream of bytes
  • text as common represent
    • allow external tools
    • human readable
    • conversion to other type is not a hard issue
  • interpreted(解释) development environment
    • command is interpreter & single program
  • shell script: combine shell commands in a file and let shell to run it
• feature
  • very effective for certain type of problem(very strange syntax)
  • simple constructs
  • command set = building blocks
  • manipulating(操作) text files
  • sh scripts
• simple script
  • create file
    • any editor
    • #! + (path of interpreter)
    • comments in scripts begin with #
    • line terminated by \n
  • save file
  • run file
• execute file
  • sh fileName(argument to shell as command)
  • ./fileName(as command)
  • sh < fileName(stdin)
• Variables
  • variable start with $
  • EnvVar(already set)
  • predefined variable
    • ? exit status of last command
    • $ PID for current process
    • …
    • argument to command are variables 1,2,3…..number represent number of parameter on command line
    • @ – all parameter will use
    • # – number of parameter
    • 0 – name of script file
  • local variable,user-defined
    • variableName= value(string)
      • create is variable not exist
    • empty string refer to a variable not exist
    • variable name
      • all capital letter
      • not use EnvVar name(HOME,PATH…)
  • quoting
    • “”: enclosed string
      • still perform some substitution(ex. variable substitutions)
        • no filename wildcard expansion
    • ”: enclosed string
      • no variable substitution or file name wildcard expansion
      • not same as “
    • \: quote a single character(remove function)
    • ${} = variable substitution
  • shift command
    • var 1 get value of var 2
    • var 2 get value of var 3
    • etc…
  • a shell command can run as a child of your script
  • export an environment variable in shell can be then used in another script
    • use export or env command
    • local shell variable (not created with export,env ) won`t work
  • env variable can be changed inside script and won`t known by parent process
  • .(dot) before a script file means use this file as a source file
    • ./ script = source script
• Debugging Script
  • set -v
    • verbose mode
      • provide additional detail to what computer is doing
      • turn off variable expansion in “”
  • set -x
    • execution mode on
      • show what command is and it`s output next line
• Simple I/O
  • expr command

    • expr prints the value of EXPRESSION to standard output. A blank line below separates increasing precedence groups.

      EXPRESSION may be:

      ARG1 | ARG2	ARG1 if it is neither null nor 0, otherwise ARG2.
      ARG1 & ARG2	ARG1 if neither argument is null or 0, otherwise 0.
      ARG1 < ARG2	ARG1 is less than ARG2.
      ARG1 <= ARG2	ARG1 is less than or equal to ARG2.
      ARG1 = ARG2	ARG1 is equal to ARG2.
      ARG1 != ARG2	ARG1 is unequal to ARG2.
      ARG1 >= ARG2	ARG1 is greater than or equal to ARG2.
      ARG1 > ARG2	ARG1 is greater than ARG2.
      ARG1 + ARG2	arithmetic sum of ARG1 and ARG2.
      ARG1 – ARG2	arithmetic difference of ARG1 and ARG2.
      ARG1 * ARG2	arithmetic product of ARG1 and ARG2.
      ARG1 / ARG2	arithmetic quotient of ARG1 divided by ARG2.
      ARG1 % ARG2	arithmetic remainder of ARG1 divided by ARG2.
      STRING : REGEXP	anchored pattern match of regular expression REGEXP in STRING.
      match STRING REGEXP	same as STRING : REGEXP.
      substr STRING POSLENGTH	substring of STRING, POS counted from 1.
      index STRING CHARS	index in STRING where any CHARS is found, or 0.
      length STRING	length of STRING.
      + TOKEN	interpret TOKEN as a string, even if it is a keyword like ‘match‘ or an operator like ‘/‘.
      ( EXPRESSION )	value of EXPRESSION.

    • from:http://www.computerhope.com/unix/uexpr.htm
    • usage:
      • put expr and expression in script, use input as value need to compare, check return value
    • exit status also important(show input vaild or not)
      – 0 if expression is neither null nor 0
      – 1 if expression is null or 0
      – 2 if expression is syntactically invalid
  • exit
    • end process
    • default return 0(success)
  • command substitution
    • in script3.sh. expr have a exit status 0 and a output 2, the 2 used as value of output; in echo command, the value of out used as a argument
    • the command inside “ can be any command
      • can change to $() in bash
    • child command inherit stdin and stdout from parent
      • in script6.sh, the stdin used inside “
    • in script7.sh, shows how command substitution works one by one
  • control-flow constructs
    • conditional
      • if-then-elif-then…-else-fi
        • example of conditional commands
        • exit status: 0-> true; else -> false
        • test command:
          • compare or check status and return to if
    • multi-way
      • case;case;case…..
        • value syntax same to filename wildcard
        • *
        • ?
        • [] a set of character
        • | alternative pattern
    • loop
      • for(iterative)
        • script11.sh
        • foobar in not inside current workspace:
      • while/until(conditional)
        • script12.sh
    • Modularization
      • function(){}
      • () must present with no argument
      • use $1….$n for parameters
      • use exit/return for exit status(0,1,2…..)
    • redirection (output)
      • break
      • continue
      • exit
    • operacors
      • https://www.tutorialspoint.com/unix/unix-basic-operators.htm
  • awk
    • higher level than perl, python and C/C++
    • lower level than shell
    • for text procession and report generation
    • process text one record(default:line) each time
    • weakly typed
      • var as string
      • convert to number when contain number in processing
    • interpreted language(run directly without compile(not like c))
    • 
    • run
      • awk ‘script’ (infile)
      • awk -f scriptfile (infile)
        • if not infile, read from stdin
      • awk ‘/pattern/{action}’ [filename]
      • awk ‘condition{action}’ [filename]
        • $0 means everything in record
        • $1….$n field 1…n, diff by field delimiters(space,tab…)
        • 1. read record
        • 2.break record
        • 3. check if satisfy condition
        • 4. perform action
      • same arithmetic operator as, all do in float point(numbers) except %
      • statements
        • if-else
        • while-do
        • do-while
        • for
        • break continue
        • printf
      • build-in variables
        • NR: number of records input thus far
        • NF: # of fields in current record
        • FILENAME: name of current input file
        • FS: field separator characters
          • string
          • • defaults to any/all whitespace
          • can be a regular expression
          • can also be set on command line with -F option
        • RS: record separator
          • defaults to newline
          • can be a regular expression
        • OFS: output field separator •
          • string printed between elements in print •
          • default: space
        • ORS: output record separator; string printed at end of print
          •  default: newline
        • OFMT: format code for numeric output. •
          • default: %.6g ‣
        • variables can be set within script or on invocation of awk with
          • -v var=val
      • selector
        • (!)/regex/ : (false)true if the record matches regex
        • $N~(!)/regex/ : (false)true if field N matches regex
        • $N op value
          • op in { == , !=, <, >, <=, >= }
          • value is a string or number
        • boolean ops: &&, ||, !
        • use built-in variables
      • make it a shell command
        • #!/bin/awk -f
        • chmod u+x