GitHub Repository: suyashi29/python-su
Path: blob/master/Python core/More Topics.ipynb
³⁰⁷⁴ views

Kernel: Python 3

1. Garbage Collection in python

Python’s memory allocation and deallocation method is automatic. The user does not have to preallocate or deallocate memory similar to using dynamic memory allocation in languages such as C or C++.

Python uses two strategies for memory allocation:

Reference counting
Garbage collection

Automatic Garbage Collection of Cycles

Because reference cycles take computational work to discover, garbage collection must be a scheduled activity. Python schedules garbage collection based upon a threshold of object allocations and object deallocations. When the number of allocations minus the number of deallocations is greater than the threshold number, the garbage collector is run. One can inspect the threshold for new objects (objects in Python known as generation 0 objects) by importing the gc module and asking for garbage collection thresholds

In [1]:

# loading gc 
import gc 

# get the current collection 
# thresholds as a tuple 
print("Garbage collection thresholds:", 
					gc.get_threshold())

Out[1]:

Garbage collection thresholds: (700, 10, 10)

Here, the default threshold on the above system is 700. This means when the number of allocations vs. the number of deallocations is greater than 700 the automatic garbage collector will run. Thus any portion of your code which frees up large blocks of memory is a good candidate for running manual garbage collection.

Manual Garbage Collection

Invoking the garbage collector manually during the execution of a program can be a good idea on how to handle memory being consumed by reference cycles. The garbage collection can be invoked manually in the following way

In [2]:

# Importing gc module 
import gc 

# Returns the number of 
# objects it has collected 
# and deallocated 
collected = gc.collect() 

# Prints Garbage collector 
# as 0 object 
print("Garbage collector: collected", 
		"%d objects." % collected)

Out[2]:

Garbage collector: collected 0 objects.

In [5]:

import gc 
i = 0

# create a cycle and on each iteration x as a dictionary 
# assigned to 1 
def create_cycle(): 
	x = { } 
	x[i+1] = x 
	print(x )

# lists are cleared whenever a full collection or 
# collection of the highest generation (2) is run 
collected = gc.collect() # or gc.collect(2) 
print ("Garbage collector: collected %d objects." % (collected) )

print ("Creating cycles...")
for i in range(10): 
	create_cycle() 

collected = gc.collect() 

print ("Garbage collector: collected %d objects." % (collected) )

Out[5]:

Garbage collector: collected 41 objects.
Creating cycles...
{1: {...}}
{2: {...}}
{3: {...}}
{4: {...}}
{5: {...}}
{6: {...}}
{7: {...}}
{8: {...}}
{9: {...}}
{10: {...}}
Garbage collector: collected 10 objects.

2. API Invocation(SOAP)

What is SOAP ?

Without entering to much into details, SOAP is an RPC (Remote Procedure Call) object-oriented protocol used by web services. In contrary to the REST architecture, SOAP allows applications to call functions directly. One of its disadvantages is that it uses a verbose XML format, so first it's slower than using simple JSON, and then SOAP support within Python is not as strong as within Java or .NET.

Weather SOAP Service using plain requests lib

In [10]:

import requests
url="http://wsf.cdyne.com/WeatherWS/Weather.asmx?WSDL"
#headers = {'content-type': 'application/soap+xml'}
headers = {'content-type': 'text/xml'}
body = """<?xml version="1.0" encoding="UTF-8"?>
         <SOAP-ENV:Envelope xmlns:ns0="http://ws.cdyne.com/WeatherWS/" xmlns:ns1="http://schemas.xmlsoap.org/soap/envelope/" 
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
            <SOAP-ENV:Header/>
              <ns1:Body><ns0:GetWeatherInformation/></ns1:Body>
         </SOAP-ENV:Envelope>"""

response = requests.post(url,data=body,headers=headers)
print (response.content)

Out[10]:

b'<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><soap:Fault><faultcode>soap:Server</faultcode><faultstring>Server was unable to process request. ---&gt; A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)</faultstring><detail /></soap:Fault></soap:Body></soap:Envelope>'

Secure Shell (SSH)

SSH, also known as Secure Shell or Secure Socket Shell, is a network protocol that gives users, particularly system administrators, a secure way to access a computer over an unsecured network. SSH also refers to the suite of utilities that implement the SSH protocol. Secure Shell provides strong authentication and encrypted data communications between two computers connecting over an open network such as the internet. SSH is widely used by network administrators for managing systems and applications remotely, allowing them to log into another computer over a network, execute commands and move files from one computer to another.

In python SSH is implemented by using the python library called fabric. It can be used to issue commands remotely over SSH.

In the below example we connect to a host and issue the command to identify the host type. We capture the result in and display it as a formatted text.

first install : conda install -c conda-forge fabric

In [ ]:

from fabric import Connection
result = Connection('xyz.com').run('uname -s')
msg = "Ran {.command!r} on {.connection.host}, got stdout:\n{.stdout}"
print(msg.format(result))

SSH ,Paramiko & FTP

Paramiko is a Python (2.7, 3.4+) implementation of the SSHv2 protocol, providing both client and server functionality. While it leverages a Python C extension for low level cryptography (Cryptography), Paramiko itself is a pure Python interface around SSH networking concepts.

install : $ pip install paramiko --user

Paramiko to execute Remote Commands:

We will use paramiko module in python to execute a command on our remote server.
Client side will be referenced as (side-a) and Server side will be referenced as (side-b

In [ ]:

## python code
import paramiko

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname='192.168.10.10', username='ubuntu', key_filename='/home/ubuntu/.ssh/mykey.pem')

stdin, stdout, stderr = ssh.exec_command('lsb_release -a')

for line in stdout.read().splitlines():
    print(line)

ssh.close()

Execute our Command Remotely:

Now attempt to establish the ssh connection from side-a, then run lsb_release -a on our remote server, side-b : by following command

$ python execute.py

FTP

The ftplib module in Python allows you to write Python programs that perform a variety of automated FTP jobs. You can easily connect to a FTP server to retrieve files and process them locally. Steps :

To use the ftplib module in Python, you first have to import it into your script.
Open a Connection : To "open" a connection to the FTP Server, you have to create the object.
Once the connection is made (opened), you can use the methods in the ftplib module.

Example

This program will first connect to a FTP server (ftp.cwi.nl) and then list thefiles and directories in the FTP server root directory using the LIST() method.

In [ ]:

from ftplib import FTP

ftp = FTP('ftp.cwi.nl')   # connect to host, default port

ftp.login()               # user anonymous, passwd anonymous@

ftp.retrlines('LIST')     # list directory contents

Our second program opens a connection to 'ftp.sunet.se' as the user 'default' with an email address of '[email protected]'
It then lists the files and directories on the FTP server by using the dir()method.
The output is saved to the 'files' variable.
use print to see the files on screen.
To change directoryjust use ftp.cwd(path).
To close the FTP connection, use the quit() method.

In [ ]:

import ftplib

ftp = ftplib.FTP('ftp.sunet.se', 'default', '[email protected]')

print "File List: "

files = ftp.dir()

print files

ftp.cwd("/pub/unix") #changing to /pub/unix

Meta characters in python & re module , matching

Regular expressions (called REs, or regexes, or regex patterns) are essentially a tiny, highly specialized programming language embedded inside Python and made available through the re module.
Using this little language, you specify the rules for the set of possible strings that you want to match; this set might contain English sentences, or e-mail addresses, or TeX commands, or anything you like.
Here’s a complete list of the metacharacters : ^ $ * + ? { } \ | ( )
The first metacharacters we’ll look at are [ and ]. They’re used for specifying a character class, which is a set of characters that you wish to match. Characters can be listed individually, or a range of characters can be indicated by giving two characters and separating them by a '-'. For example, [abc] will match any of the characters a, b, or c; this is the same as [a-c], which uses a range to express the same set of characters. If you wanted to match only lowercase letters, your RE would be [a-z].

Metacharacters are not active inside classes. For example, [akm $] will match any of the characters 'a', 'k', 'm', or '$ '; '$' is usually a metacharacter, but inside a character class it’s stripped of its special nature.

You can match the characters not listed within the class by complementing the set. This is indicated by including a '^' as the first character of the class. For example, [^5] will match any character except '5'. If the caret appears elsewhere in a character class, it does not have special meaning. For example: [5^] will match either a '5' or a '^'.

Perhaps the most important metacharacter is the backslash, . As in Python string literals, the backslash can be followed by various characters to signal various special sequences. It’s also used to escape all the metacharacters so you can still match them in patterns; for example, if you need to match a [ or , you can precede them with a backslash to remove their special meaning: [ or \.

Some of the special sequences beginning with '' represent predefined sets of characters that are often useful, such as the set of digits, the set of letters, or the set of anything that isn’t whitespace.

Matching Characters

Most letters and characters will simply match themselves. For example, the regular expression test will match the string test exactly. (You can enable a case-insensitive mode that would let this RE match Test or TEST as well; more about this later.)

There are exceptions to this rule; some characters are special metacharacters, and don’t match themselves. Instead, they signal that some out-of-the-ordinary thing should be matched, or they affect other portions of the RE by repeating them or changing their meaning.

Compliling regular expressions using re module

In [13]:

import re
p = re.compile('ab*')
p

Out[13]:

re.compile(r'ab*', re.UNICODE)

re.compile() also accepts an optional flags argument, used to enable various special features and syntax variations

In [14]:

p = re.compile('ab*', re.IGNORECASE)

matching various strings against the RE [a-z]+. An empty string shouldn’t match at all, since + means ‘one or more repetitions’. match() should return None in this case, which will cause the interpreter to print no output. You can explicitly print the result of match() to make this clear.

In [15]:

import re
p = re.compile('[a-z]+')
p

Out[15]:

re.compile(r'[a-z]+', re.UNICODE)

In [16]:

p.match("")
print(p.match(""))

Out[16]:

None

let’s try it on a string that it should match, such as tempo. In this case, match() will return a match object, so you should store the result in a variable for later use.

In [17]:

m = p.match('tempo')
m

Out[17]:

<re.Match object; span=(0, 5), match='tempo'>

Function Polymorphism

Polymorphism in Python The word polymorphism means having many forms. In programming, polymorphism means same function name (but different signatures) being uses for different type.

In [18]:

# Python program to demonstrate in-built poly- 
# morphic functions 

# len() being used for a string 
print(len("Data Science")) 

# len() being used for a list 
print(len([10, 20, 30]))

Out[18]:

12
3

In [19]:

# A simple Python function to demonstrate 
# Polymorphism 

def add(x, y, z = 0): 
	return x + y+z 

# Driver code 
print(add(2, 3)) 
print(add(2, 3, 4))

Out[19]:

5
9

Polymorphism with class methods

Below code shows how python can use two different class types, in the same way. We create a for loop that iterates through a tuple of objects. Then call the methods without being concerned about which class type each object is

In [20]:

class India(): 
	def capital(self): 
		print("New Delhi is the capital of India.") 

	def language(self): 
		print("Hindi the primary language of India.") 

	def type(self): 
		print("India is a developing country.") 

class USA(): 
	def capital(self): 
		print("Washington, D.C. is the capital of USA.") 

	def language(self): 
		print("English is the primary language of USA.") 

	def type(self): 
		print("USA is a developed country.") 

obj_ind = India() 
obj_usa = USA() 
for country in (obj_ind, obj_usa): 
	country.capital() 
	country.language() 
	country.type()

Out[20]:

New Delhi is the capital of India.
Hindi the primary language of India.
India is a developing country.
Washington, D.C. is the capital of USA.
English is the primary language of USA.
USA is a developed country.

Polymorphism with Inheritance: Overidding

In Python, Polymorphism lets us define methods in the child class that have the same name as the methods in the parent class. In inheritance, the child class inherits the methods from the parent class. However, it is possible to modify a method in a child class that it has inherited from the parent class. This is particularly useful in cases where the method inherited from the parent class doesn’t quite fit the child class. In such cases, we re-implement the method in the child class. This process of re-implementing a method in the child class is known as Method Overriding.

In [23]:

class Bird: 
 def intro(self): 
    print("There are many types of birds.") 
 def flight(self): 
    print("Most of the birds can fly but some cannot.") 
class sparrow(Bird): 
 def flight(self): 
    print("Sparrows can fly.") 

class ostrich(Bird): 
 def flight(self): 
    print("Ostriches cannot fly.") 

obj_bird = Bird() 
obj_spr = sparrow() 
obj_ost = ostrich() 

obj_bird.intro() 
obj_bird.flight() 

obj_spr.intro() 
obj_spr.flight() 

obj_ost.intro() 
obj_ost.flight()

Out[23]:

There are many types of birds.
Most of the birds can fly but some cannot.
There are many types of birds.
Sparrows can fly.
There are many types of birds.
Ostriches cannot fly.

Method Overloading

Like other languages (for example method overloading in C++) do, python does not supports method overloading. We may overload the methods but can only use the latest defined method

In [24]:

# First product method. 
# Takes two argument and print their 
# product 
def product(a, b): 
	p = a * b 
	print(p) 
	
# Second product method 
# Takes three argument and print their 
# product 
def product(a, b, c): 
	p = a * b*c 
	print(p) 

# Uncommenting the below line shows an error	 
# product(4, 5) 

# This line will call the second product method 
product(4, 5, 5)

Out[24]:

100

Class & Instance Attributes in Python

Class attributes

Class attributes belong to the class itself they will be shared by all the instances. Such attributes are defined in the class body parts usually at the top, for legibility.

In [26]:

# Write Python code here 
class sampleclass: 
	count = 0	 # class attribute 

	def increase(self): 
		sampleclass.count += 1

# Calling increase() on an object 
s1 = sampleclass() 
s1.increase()		 
print (s1.count )

# Calling increase on one more 
# object 
s2 = sampleclass() 
s2.increase() 
print (s2.count )

print (sampleclass.count )

Out[26]:

1
2
2

Instance Attributes

Unlike class attributes, instance attributes are not shared by objects. Every object has its own copy of the instance attribute (In case of class attributes all object refer to single copy). To list the attributes of an instance/object, we have two functions:-

vars()- This function displays the attribute of an instance in the form of an dictionary.
dir()- This function displays more attributes than vars function,as it is not limited to instance. It displays the class attributes as well.

In [30]:

# Python program to demonstrate 
# instance attributes. 
class emp: 
	def __init__(self): 
		self.name = 'xyz'
		self.salary = 4000

	def show(self): 
		print (self.name )
		print (self.salary )

e1 = emp() 
print ("Dictionary form :", vars(e1) )
print (dir(e1) )

Out[30]:

Dictionary form : {'name': 'xyz', 'salary': 4000}
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'name', 'salary', 'show']

Using third party modules – Boto3,pysnow

https://pysnow.readthedocs.io/en/latest/ : pynow module
https://boto3.amazonaws.com/v1/documentation/api/latest/py-modindex.html : Boto3

An overview on __builtin and future modules

You can use future to help to port your code from Python 2 to Python 3 today – and still have it run on Python 2.If you already have Python 3 code, you can instead use future to offer Python 2 compatibility with almost no extra work. The easiest way is to start each new module with these lines:

from future import (absolute_import, division,

                    print_function, unicode_literals)

from builtins import *

link for built in modules : https://www.tutorialsteacher.com/python/python-builtin-modules

Recursive directory iteration using os.walk

When you use a scripting language like Python, one thing you will find yourself doing over and over again is walking a directory tree, and processing files. While there are many ways to do this, Python offers a built-in function that makes this process a breeze.

example that walks a directory tree, printing out the name of each directory and the files contained :

os.walk takes care of the details, and on every pass of the loop, it gives us three things:

dirName: The next directory it found.
subdirList: A list of sub-directories in the current directory.
fileList: A list of files in the current directory.

In [31]:

# Import the os module, for the os.walk function
import os
 
# Set the directory you want to start from
rootDir = '.'
for dirName, subdirList, fileList in os.walk(rootDir):
    print('Found directory: %s' % dirName)
    for fname in fileList:
        print('\t%s' % fname)

Out[31]:

Found directory: .
	1.log
	Advanced_IP_Scanner_2.5.3646.exe
	DAY 1Basic Implementation in Python.ipynb
	DAY 2.ipynb
	Day 3.ipynb
	Day 4.ipynb
	Day5.ipynb
	demo.txt
	demo.xml
	Few Topics.ipynb
	fibo.py
Found directory: .\.ipynb_checkpoints
	DAY 1Basic Implementation in Python-checkpoint.ipynb
	DAY 2-checkpoint.ipynb
	Day 3-checkpoint.ipynb
	Day 4-checkpoint.ipynb
	Day5-checkpoint.ipynb
	Few Topics-checkpoint.ipynb
Found directory: .\Flask
Found directory: .\Flask_App
	app.py.ipynb
Found directory: .\Flask_App\.ipynb_checkpoints
	app.py-checkpoint.ipynb
Found directory: .\Flask_App\templates
	app.py.ipynb
	main.py.ipynb
	new.html
Found directory: .\Flask_App\templates\.ipynb_checkpoints
	app.py-checkpoint.ipynb
	main.py-checkpoint.ipynb
Found directory: .\__pycache__
	fibo.cpython-37.pyc
	M.cpython-37.pyc

Changing the Way the Directory Tree is Traversed

By default, Python will walk the directory tree in a top-down order (a directory will be passed to you for processing), then Python will descend into any sub-directories. We can see this behaviour in the output above; the parent directory (.) was printed first, then its 2 sub-directories.

Sometimes we want to traverse the directory tree bottom-up (files at the very bottom of the directory tree are processed first), then we work our way up the directories. We can tell os.walk to do this via the topdown parameter:

In [32]:

import os
 
rootDir = '.'
for dirName, subdirList, fileList in os.walk(rootDir, topdown=False):
    print('Found directory: %s' % dirName)
    for fname in fileList:
        print('\t%s' % fname)

Out[32]:

Found directory: .\.ipynb_checkpoints
	DAY 1Basic Implementation in Python-checkpoint.ipynb
	DAY 2-checkpoint.ipynb
	Day 3-checkpoint.ipynb
	Day 4-checkpoint.ipynb
	Day5-checkpoint.ipynb
	Few Topics-checkpoint.ipynb
Found directory: .\Flask
Found directory: .\Flask_App\.ipynb_checkpoints
	app.py-checkpoint.ipynb
Found directory: .\Flask_App\templates\.ipynb_checkpoints
	app.py-checkpoint.ipynb
	main.py-checkpoint.ipynb
Found directory: .\Flask_App\templates
	app.py.ipynb
	main.py.ipynb
	new.html
Found directory: .\Flask_App
	app.py.ipynb
Found directory: .\__pycache__
	fibo.cpython-37.pyc
	M.cpython-37.pyc
Found directory: .
	1.log
	Advanced_IP_Scanner_2.5.3646.exe
	DAY 1Basic Implementation in Python.ipynb
	DAY 2.ipynb
	Day 3.ipynb
	Day 4.ipynb
	Day5.ipynb
	demo.txt
	demo.xml
	Few Topics.ipynb
	fibo.py

In [ ]: