Details

Published: Wednesday, 29 July 2020 13:45

Hits: 908

Python Advanced features: Whatever is mentioned below works for python 3.6 and later (unless mentioned specifically)

Modules:

We saw functions in the previous section. However, since these built in functions are limited, python provides modules which have more functions defined in them. These modules can be imported in other python pgm, so that we don't have to write our own functions for commonly used tasks. There are modules written for almost everything in python, so search python modules before you start writing your function for something complicated. We can create our own modules with our functions in them.

when we want to implement any functionality in module, we can write them as functions. However, if we are working in OOP, we can write them as methods. Most modules provide both function and method support for a given functionality, so you can use any one depending on your preference.

module: are python files (*.py) consisting python code. Any python file can be referenced as module (there's nothing special about a module). Python file called "hello.py" has module name of "hello" that can be imported into other python files. Modules can define functions, classes and variables, and can have any additional code outside of func/classes (i.e calling these functions, running test cases, etc). All of the code within file file1.py is part of the module "file1".

Packages:

We can bundle multiple modules (or multiple files) together to form a package. A package is basically a directory with Python files and a file with the name __init__.py (init with 2 underscore _ on both sides of init). This means that every directory inside of the Python path, which contains a file named __init__.py, will be treated as a package by Python. It's possible to put several modules into a Package. Package is just for convenience where several modules are bundles into one: modules in a package behave exactly like any other module. Package is treated like a higher level module.

ex: we create dir pkg1, and put several files file1.py, file2.py etc in it. Additionally, we need to put init file "__init__.py:, which can be an empty file (but is needed to identify dir pkg1 as a package and not just a dir).  Now the whole dir is treated as package "pkg1". Now we can use dotted module names, i.e A.B is used to refer to module B in package A

Import modules in python:

Once we have downloaded the modules, we can call it in any other python pgm. To do that, first we have to import that module using "import" cmd.

ex: import hello => imports hello module (hello.py has module name of hello, so here hello.py is being included in our current pgm). Any print stmt etc or any function called inside hello.py will be executed, as importing module or file is same as doing an "include file" or putting the code of that imported file in our current file at that location, where import is being called.

ex:
name1.py: python script that contains func "print_def".
main.py: python script thats calls func from name1.py

Import from any dir:

So contents of main.py will look like this:
import name1 => import entire module from name1.py. It first searches for builtin module named "name1", and on not finding a built in module, searches for file named "name1.py" in current dir and then dir specified in "sys.path" var or in environment variable "PYTHONPATH" (from your operating system). By default, "sys.path" includes python system dir only, but we can add user specified dir in it too. If name1 module is in subdir, then we can specify the relative path here as "import subdir1.dir2.name1". We can go up the dir structure by using 1 dot to go one dir above, but it doesn't seem to work for me (i.e import ...dir1/file3 should take you 3 levels higher to dir1, but it gives an error as invalid syntax).

ex: name1.print_def("zara"); => now any func, etc can be called from that module, by specifying name of the module (i.e file), followed by a dot and then the name of func/method/attribute. This is assuming name1 module is in current dir.

ex: subdir1.dir2.name1.print_def("zara"); => This is needed if we imported module from some subdir in current dir (i.e "import subdir1.dir2.name1").

To import modules in any dir (not just subdir starting from current dir), we can't to do in above way. We need to add dir path to sys.path

sys.path.append('/home/user1/dir1/') => Now this dir is added to python sys.path var, so that python will look in this dir also, when searching for modules

sys.path.append('../../dir1/') => Relative paths (relative to current dir where script is running) can also be provided

One very easy way to include modules in any dr is to make soft links to other dir containing these modules in current dir (i.e ln -s ../../dir1 .). Now, all modules are available in subdir of current dir, which can now easily be accessed by doing "import dir1.dir2.name1".

NOTE: many modules are automatically included with python distribution, so you do not need to downlaod them, you call just import them, i,e import math => this imports math module w/o downloading math (as math module is already included with python)

alias: we can also modify the name of module,

ex: import math as m => here math is renamed as "m". So, now we can call math as m. We can also use package.module name i.e import mathplotlib.pyplot as plt. As a convention most modules are imported with a "given" alias name.

print (m.pi); => instead of using math.pi, we used m.pi (as we used alias above)

import specific function: Instead of importing the whole module which imports everything from that module, we can also import specific function from that module:

from random import randint => This is another style of importing func. Here func "randint" is moved from "random" namespace to current namespace, so that dot is no longer required. So, now randint can be called directly, i.e instead of random.randint(), we can call randint(). If we try to call random.randint() now, we'll get an error as "random" namespace is not present anymore. Note: only randint func is imported here (and any other func inside random module is not imported)

from random import * => This imports all functions from module random. Most of the times, we just use * as it's convenient, however if there are too many functions inside that module, then it may affect performance, as a lot of memory will be used up loading unneeded functions.

Import packages in python:

Importing packages is done in same way as importing modules. NOTE: pkg name is no different than dir name, so we can use usual "import from subdir" style that we used for modules above (i.e init files are not needed). Looks like for python3.6 and above, init files are not needed at all, as all ex below work without init file too.

ex: import pkg1 => Here pkg1 is imported, but not the modules inside it. So, if we try to access any module file1 or file2 (i.e pkg1.file1.func1() or pkg1/file2, etc), we'll get an error "name file1 not defined". We have to import modules explicitly.

ex: import pkg1.file1 => Here we import each module explicitly. This is no diff than importing file1.py from dir pkg1 as we saw above for importing modules for subdir. init file is not needed in this case. Here pkg1 namespace is still maintained for these func.  This is helpful in cases where there are functions with same names in different modules

pkg1.file1.func1() => Here full path is needed as file1 is still in namespace "pkg1"

We can also use the "from" style to import modules.Here "from" may have name of module or name of pkg.

ex: from pkg1 import file1, file2 => This is the most common way. This imports both file1.py and file2.py modules from package "pkg1" into current pgm. "from pkg1 import *" doesn't work (errors out), as * is for func/method within modules and not for modules themselves ?? Not sure??

file1.func1() => This function inside file1 module can now be called.

ex: from pkg1.file1 import * => This is valid syntax as we are importing all func from module file1 in pkg1.

pip:

To install a module which is not present on your python installation, simply use pip. The most common usage of pip is to install from the Python Package Index using a requirement specifier. Generally speaking, a requirement specifier is composed of a project name followed by an optional version specifier. There are many ways to install various modules in python, but pip is the preferred installer program. Starting with Python 3.4 for Python3 and Python 2.7.9 for Python2, it is included by default with the Python binary installers. So, always use pip to install modules.

To check if pip is installed, run cmd:

pip --version => Since we didn't specify python version here, this cmd runs with python version installed and pointing to python, i.e "python -m pip --version".

sudo apt install python3-pip => This installs pip for python3. if we use "sudo apt install python-pip", then it installs pip for python2.

python3 -m pip => Now running this cmd will either show options which means pip is installed correctly for python3, or it will say "module pip not found" which means pip didn't get installed correctly for python3. Or we can also type "python3 -m pip --version" to see version number of pip.

Install modules using pip:

Once pip is installed, we can use it to install other Python modules. cmds are same on both CentOS and LinuxMint.

ex: python3.6 -m pip install SomePackage => This installs package named "SomePackage". Here, instead of running pip directly, we prepend it with "python3.6 -m pip". This runs appropriate copy of pip for python3.6. This is helpful when there are multiple python versions installed on your system. Here pip is called for python3.6, so that SomePackage is installed for python3.6. This is preferred way, since there is no ambiguity on which version of python was this package installed for.

If module gets installed for python3, we see all those modules in this dir: /usr/local/lib64/python3.6/site-packages/*

However, if it's gets installed for python2, we see those modules here: /usr/lib64/python2.7/site-packages/*

We can use unix "locate" cmd to find out where these libs are. i.e locate python2 => shows all libs and other dir for python2.

Below are some very helpful modules that you should install as they may be required for many projects.

1.pygame: pygame is very popular gui package for python for building games. See on pygame.org for more details. It's very easy for kids to get started on this and build sleek games. To install pygame module on python3, run this:

$ pip install pygame => This installs pygame module on your system, so that it can be imported in any python file. However, here since we didn't provide specific python version, it uses copy of pip for python. This is equiv to "python -m pip install pygame". However, python here may be pointing to python2, python3 or python3.6, This may install pygame only for python version pointed to (if python points to python2, then pygame is only installed for python2. If we import pygame in python3 pgm, it will error out saying module not found), so better to specify python version

$ sudo python3 -m pip install -U pygame => this install pygame for python3. -U option updates pygame module if already installed. Here pygame package is installed for python3 (or python3.6 as python3 is pointing to python3.6). Or, we can run this cmd "python3 .6-m pip install pygame", it's the same thing. 

Important modules: Below are few modules that are already included with distribution, and are useful for carrying out many tasks.

Module random: It generates random numbers

import random => importing built in module named random. All methods or functions in random module are imported here

print (random.randint(1,30)) => calls func returning random int b/w range 1 to 30. Functions within modules are called by separating them via dot (to indicate that the function is in the namespace "random".

from random import randint => If we import randint function this way, then we just do print (randint(1,30)).

Module os: One  of the important modules is "os" module. It provides operating system i/f. exec* functions loads new pgm with optional args.

from os import execvp, envron => imports these 2 methods from os module. execvp loads new pgm, while environ function provides shells environment var.

exexvp(file, args) => this executes a new pgm specified by "file" (with args as specified), replacing the current process, they do not return. The new executable is loaded into current process.

environ['HOME'] => This provides the value of HOME environment var found in bash, csh, etc.

ex: os.rmdir(/home/scratch) => runs unix cmd rmdir, so causes /home/scratch dir to be removed

Module sys: This module provides  "argv" func to get access to cmd line args. sys.argv is list of cmd line args. We can also access individual args as sys.argv[0], sys.argv[1] etc

ex: file fil1.py

from sys import argv # or just "import sys"

print ("Args are", argv[0], argv[1], ...)

run file1.py abc 179 my_name => here argv[0]= file.py, argv[1]=abc, ...

Standard library: python's std lib contains many built in modules (written in C) that provides access to system functionality such as file I/O (which would otherwise be inaccessible to python programmers) as well as many python modules for a variety of simple/complicated tasks. This std lib contains data types, built in functions (so that no import is needed), and tons of modules. This std lib is included in any python pgm by default.

A. Date and Time:
import time;
ticks = time.time() => ticks is floating point num. 1st time is time module, while 2nd time is time method working on time module
print "Ticks=",ticks => prints ticks since Jan 1, 1970
localtime = time.localtime(time.time()) => gets local time

import calendar
cal = calendar.month(2008, 1) => gets calender for Jan 2008. Here calender is module and month is the method

regular expression:  helps match or find patterns using special syntax. This is very useful feature as most of the times scripts are used for matching, replacing, etc words in documents. module re along with it's many functions used for this.

1. module re: module re provides full support for Perl-like regular expressions in Python. The re module raises the exception re.error if an error occurs while compiling or using a regular expression.

match func:
re.match => re.match function returns a match object on success, None on failure.
We use group(num) (returns entire match or specific match num if specified) or groups() (returns all matching subgroups in tuple) function of match object to get matched expression.
syntax: re.match(pattern, string, flags=0)
Since various characters have special meaning when they are used in regular expression, to avoid any confusion while dealing with regular expressions, we would use Raw Strings as r'expression'.
control char: decides how to match
. => Matches any single character except newline. Using m option allows it to match newline as well.
[...] => Matches any single character in brackets. [abc] matches any of a or b or c
[^...] => Matches any single character not in brackets. ex: [^0-9] => match anything other than digit
re* => Matches 0 or more occurrences of preceding expression.
re+ => Matches 1 or more occurrence of preceding expression.
re? => Matches 0 or 1 occurrence of preceding expression.
a|b => Matches either a or b.
(re) => Groups regular expressions and remembers matched text. (.*) me => matches 0 or more char until "space me" matches
\d+ => match any num of digits \D => match any nondigit, \d{3,5} => match 3,4 or 5 digits
\s => match whitespace, \S => match non whitespace
\w => match single word char

ex:
<.*> => matches in greedy way. So will match full <python>perl> in <python>perl>
<.*?> => matches in non greedy way (since ? means 0 or 1 occurrence only). So will match <python> in <python>perl>
\bpyth(on|off)\b => match puthon or pythoff
ex:
import re => import re module
line = "Cats are smarter than dogs"
matchObj = re.match( r'(.*) are (.*?) .*', line, re.M|re.I) => (.*) matches char in greedy fashion (matches smallest possible), while (.*?) matches char in non-greedy fashion, and stored in group. re.M, re.I are optional flags. re.I performs case insensitive matching, while re.M makes $ match the end of a line (not just the end of the string) and makes ^ match the start of any line (not just the start of the string). multiple flas are provided by using exclusive OR (|)
if matchObj: => match obj returned on match, else None returned
   print "matchObj.group() : ", matchObj.group() => returns entire match since no num specified => Cats are smarter than dogs
   print "matchObj.group(1) : ", matchObj.group(1) => returns 1st match within () => Cats
   print "matchObj.group(2) : ", matchObj.group(2) => returns 2nd match within () => smarter. using (.*) will match in non greedy fashion, so it will match as much as possible until it finds spce and few char after it. so (.*) will match "smartar than" as there space after "than".
else:
   print "No match!!"

search func: similar to match in perl, where it looks for match anywhere in string (as opposed to match which looks for match only in beginning of string)
ex:
matchObj = re.match( r'dogs', line, re.M|re.I) => returns None since dogs is not at beginning of line
searchObj = re.search( r'dogs', line, re.M|re.I) => returns dogs since it appears somewhere in line

search and replace func:

ex:
phone = "2004-959-559 # This is Phone Number"
num = re.sub(r'#.*$', "", phone) => substitute anything after # until end of string with blank, so num = 2004-959-559
num = re.sub(r'\D', "", phone) => substitute any non digit with blank, so num =  2004959559. we don't need \D+ as sub does global replacement on that string for every occurnce of that pattern.

CGI, database access, webpage, sending email all can be achieved via python by importing particular modules