Loading [MathJax]/jax/output/CommonHTML/fonts/TeX/fontdata.js

Python training UGA 2017

A training to acquire strong basis in Python to use it efficiently

Pierre Augier (LEGI), Cyrille Bonamy (LEGI), Eric Maldonado (Irstea), Franck Thollard (ISTerre), Christophe Picard (LJK), Loïc Huder (ISTerre), Colin Thomas (ISTERRE)

Standard types and basic statements

Function calls

There are built-in functions and the developers can of course define other functions. To call a function:

print("hello")

hello

Some functions return a result.

round(1.2)

1

It's common to store the result in a variable:

my_var = round(1.2)

which can then be used:

print(my_var)

1

Few standard types

• Simple types (int, float, bool, complex)
• Standard type str
• Standard type list
• Standard type tuple

int (integers)

a = 4
c = -10

# binary notation (base 2)
b = 0b010

# octal notation (base 8)
o = 0o011

# hexadecimal (base 16)
h = 0x1cf0

a = int('1')       # base 10
a = int('111', 2)  # base 2
a = int('70', 8)   # base 8
a = int('16', 16)  # base 16

Remark: int in Python 3 are impressive! No limit! See https://docs.python.org/3.1/whatsnew/3.0.html#integers

Arithmetic operations

print(10 + 3)
print(10 - 3)
print(10 * 3)
print(10 / 3)  # float division
print(10 // 3)  # integer division
print(10 % 3)

13
7
30
3.3333333333333335
3
1

bool (booleans)

b = bool('1')
b = False
b = True

Comparison operations (bool)

• == equal
• != différent
• < inferior
• <= inferior or equal
• > superior
• >= superior or equal

Keyword is: check identity

a = None
print(a is None)
print(a is not None)

True
False

Keywords and and or

True and True

True

True and False

False

False and False

False

True or True

True

True or False

True

False or False

False

float (real, double precision) and complex

# float
a = float('1')
a = 1.234
a = 1e2
a = -1e-2
a = .2

# complex (2 floats)
c = complex('1')
c = 1 + 2j
print(c, c.real, c.imag)

(1+2j) 1.0 2.0

Remark: notation var_name.attr_name to access to an attribute of an object.

Warning about floating-point arithmetic and numerical errors!

b = 1e16
c = 1.2 + b
d = c - b
print(d)

2.0

Very general issue (not Python):

See https://en.wikipedia.org/wiki/Floating-point_arithmetic and https://docs.python.org/3/tutorial/floatingpoint.html

Standard type str

s = 'hello'
s = "hello"

s = ('How is it possible to write a very very '
     'very long string with lines limited to 79 characters?')

s = """Strings on 
more thand 
one line.
"""
print(s)

Strings on 
more thand 
one line.

Warning: big difference between Python 2 and Python 3. In Python 3, str are unicode and there is another type bytes.

Methods of the type str

Objects of built-in types have methods associated with their type (object oriented programming). The built-in function dir returns a list of name of the attributes. For a string, these attributes are python system attributes (with double-underscores) and several public methods:

s = 'abcdef'
print(dir(s))

['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'capitalize', 'casefold', 'center', 'count', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'format_map', 'index', 'isalnum', 'isalpha', 'isascii', 'isdecimal', 'isdigit', 'isidentifier', 'islower', 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']

To access an attribute of an object (here, the method str.startswith), we use the dot:

s.startswith('a')

True

standard type str

function str.format

Docstring:
S.format(*args, **kwargs) -> str

Return a formatted version of S, using substitutions from args and kwargs.
The substitutions are identified by braces ('{' and '}').

a = 1.23456789
'a = {}'.format(a)

'a = 1.23456789'

'a = {:.4f}'.format(a)

'a = 1.2346'

'a = {:8.4f}'.format(a)

'a =   1.2346'

'a = {:.4e} (scientific notation)'.format(a)

'a = 1.2346e+00 (scientific notation)'

print('{}\t{}\t{}'.format(1, 2, 3))

1	2	3

standard type str

Since python 3.6: format strings

a = 1.23456789
f'a = {a}'

'a = 1.23456789'

f'a = {a:.4f}'

'a = 1.2346'

f'a = {a:8.4f}'

'a =   1.2346'

f'a = {a:.4e} (scientific notation)'

'a = 1.2346e+00 (scientific notation)'

print(f'{1}\t{1+1}\t{2+1}')

1	2	3

standard type str

Strings are immutable "sequences".

• lookup

s = 'abcdef'
print('a' in s)
print('hello' not in s)

True
True

• We can get an element of a string (index starts from 0):

print(s[0])

a

• since strings are immutable, they can not be modified inplace. If we try, we get an error:

s[0] = 'b'

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-55620f378bce> in <module>()
----> 1 s[0] = 'b'

TypeError: 'str' object does not support item assignment

• since strings are sequences, they can be "sliced" (we will soon study in details this powerful notation):

s[1:3]

'bc'

• it is very simple to manipulate strings in many ways:

print((s.capitalize() + ' ' + s.upper() + '\n') * 4 ) 

Abcdef ABCDEF
Abcdef ABCDEF
Abcdef ABCDEF
Abcdef ABCDEF

Slicing

Very general, can be used on all sequences as str, list, etc... Not simple for beginners but very powerfull (see here and here).

Python indexes and slices for a six-element str. Indexes enumerate the elements, slices enumerate the spaces between the elements.

Index from rear:    -6  -5  -4  -3  -2  -1
Index from front:    0   1   2   3   4   5      
                   +---+---+---+---+---+---+
                   | a | b | c | d | e | f |    
                   +---+---+---+---+---+---+    
Slice from front:  0   1   2   3   4   5   6    
Slice from rear:  -6  -5  -4  -3  -2  -1   None

s = 'abcdef'
# s[start:stop:step]
s[2:6:2]

'ce'

# s[start:stop]
s[2:6]

'cdef'

# s[start:]
s[1:]

'bcdef'

# s[:stop]
s[:2]

'ab'

# step = -1: goes through the string in reverse order
s[::-1]

'fedcba'

Do it yourself

Suppose we have a string representing a header line of the form:

s = ' wind;temperature;;pressure '

• Remove leading and ending blanks (see str.replace and str.strip)
• Extract the first field (e.g. using find method and slicing)
• Extract the last field (e.g. using rfind method and slicing)
• Check for empty field (e.g. find ";;" pattern)
• Remove empty field (e.g. using replace method)
• BONUS : Extract the second field (tip :find can take an argument that tells where to start the search)

A possible solution:

s = ' wind;temperature;;pressure '
# remove leading blanks
s = s.strip()
print(f"--{s}--")
# extract the first field
idx = s.find(";")
s0 = s[0:idx]
print(s0)
# extract the second field
idx1 = s.find(";", idx+1) # start the search after the first ";"
s1 = s[idx+1:idx1]
print(s1)
# extract the last field
idx2 = s.rfind(";")
s2 = s[idx2+1:]
print(s2)
idx_first_empty_field = s.find(";;")
print(idx_first_empty_field)
# remove empty field
s_no_empty = s.replace(";;", ";")
print(s_no_empty)

--wind;temperature;;pressure--
wind
temperature
pressure
16
wind;temperature;pressure

standard type list

A list is a mutable sequence of (possibly inhomogeneous) elements.

type([0, 'a'])

list

# create an empty list
l = []
# fill the list (with the function append)
l.append('2')
# fill the list (with the function extend)
l.extend([6, 3.])
print(l)

['2', 6, 3.0]

# concatenate lists with the operator +
print(l + ['hello', 3])

['2', 6, 3.0, 'hello', 3]

# get values
print(l[0], l[2], l[-2])
# slicing
print(l[0:2])

2 3.0 6
['2', 6]

standard type tuple

A tuple is a immutable sequence of (possibly inhomogeneous) elements.

Remarks:

• When you need a sequence that won't be modified, tuple is usually more efficient than list.
• Use tuple instead of list for immutable sequences so that python will raise an error if you attempt to modify it

t = 0, 'a', 1.2
t1 = (5, 'hello')
t2 = tuple([1.1, 2])
type(t)

tuple

t[1]  # indexing

'a'

t[1:]  # slicing

('a', 1.2)

a, b = t1  # tuple assigment
print(b)

hello

Mutable and immutable objects

The objects of type str, int, float, bool are immutable. They can not be modified. Of course, a name that points towards an integer can point towards a different integer.

i = 1
i = i + 2  # (or i += 2)
print(i)
i = 10
print(i)

3
10

Here, the objects 1 and 3 have not been modified.

Mutable and immutable objects

An object of type list is mutable:

l = [0, 5]
print(l)
l.append('hello')
print(l)

[0, 5]
[0, 5, 'hello']

Here, the object list tagged by the name l has been modified inplace.

References and del keyword

del removes a reference. If an object in not binded to any names, Python can delete it from its internal memory.

l = ['a', 'b']
del l[1]
print(l)

['a']

More on slicing

Very general, can be used on all sequences as str, list, etc... Not simple for beginners but very powerfull (see here and here).

Python indexes and slices for a six-element str. Indexes enumerate the elements, slices enumerate the spaces between the elements.

Index from rear:    -6  -5  -4  -3  -2  -1
Index from front:    0   1   2   3   4   5      
                   +---+---+---+---+---+---+
                   | a | b | c | d | e | f |    
                   +---+---+---+---+---+---+    
Slice from front:  0   1   2   3   4   5   6    
Slice from rear:  -6  -5  -4  -3  -2  -1   0

s = 'abcdef'
# s[start:stop:step]
s[2:6:2]

'ce'

More on slicing

Assigment to mutable object

l = [0, 1, 2, 3, 4, 5]
l1 = l  # assigment to a new name l1 (no copy of the object).
# the names l and l1 points towards the same object.
l1.append('a')
print(l1)
print(l)

[0, 1, 2, 3, 4, 5, 'a']
[0, 1, 2, 3, 4, 5, 'a']

Shallow copy

l = [0, 1, 2, 3, 4, 5]
l1 = l[:] # shallow copy of l
l1.append('a')
print(l1)
print(l)

[0, 1, 2, 3, 4, 5, 'a']
[0, 1, 2, 3, 4, 5]

More on slicing

Other examples of slices for a six-element list. Indexes enumerate the elements, slices enumerate the spaces between the elements.

a = [0, 1, 2, 3, 4, 5]  
all([
    len(a) == 6,
    a[1:] == [1, 2, 3, 4, 5],
    a[:5] == [0, 1, 2, 3, 4],
    a[0] == 0,
    a[:-2] == [0, 1, 2, 3],
    a[5] == 5,
    a[1:2] == [1],
    a[-1] == 5,
    a[1:-1] == [1, 2, 3, 4],
    a[-2] == 4,
])

True

Do it yourself

Suppose we have the string containing header line.

s = 'wind;temperature;pressure'

• Extract the list of items (i.e. "wind", "temperature", "pressure"; see str.split).
• Add "Snow level" to the list of items (e.g. using append)
• Build a new header such that items are capitalized (e.g. using the methods str.join and str.capitalize and iterating on the list)

A possible solution:

s = 'wind;temperature;pressure'
list_items = s.split(";")
print(list_items)
list_items.append("snow level".capitalize())
list_items[0] = list_items[0].capitalize()
list_items[1] = list_items[1].capitalize()
list_items[2] = list_items[2].capitalize()

print(list_items)
";".join(list_items)

['wind', 'temperature', 'pressure']
['Wind', 'Temperature', 'Pressure', 'Snow level']

'Wind;Temperature;Pressure;Snow level'

The function range

The function returns a range object:

# start, stop, step
range(1, 8, 2)

range(1, 8, 2)

We can make a list with the range object:

# start, stop, step
list(range(1, 8, 2))

[1, 3, 5, 7]

# start, stop (step=1)
list(range(2, 8))

[2, 3, 4, 5, 6, 7]

# stop argument (start=0, step=1)
list(range(8))

[0, 1, 2, 3, 4, 5, 6, 7]

Do it yourself

Build a list of odd numbers in decreasing order.

A possible solution

maximum = 21
print(f"type of range is: {type(range(maximum, 0, -2))}")
my_list1 = list(range(maximum, 0, -2))
print(my_list1)

type of range is: <class 'range'>
[21, 19, 17, 15, 13, 11, 9, 7, 5, 3, 1]

Lookup in a list (and more generally in a sequence) :

Checking if an element belongs to the list can be done with the in keyword. In the worst case, this can iterate other all the elements (in particular if the element searched does not belong to the list).

l = [ 1, 2, 4, 9]
1 in l

True

Conditions: if, elif, else

if expression:
   statement(s)
else:
   statement(s)

The statement contains the block of code that executes if the conditional expression in the if statement resolves to 1 or a TRUE value.

a = 0
if a == 0:
    print('a is equal to 0.')

a is equal to 0.

a = 1
if a < 0:
    print('a is negative.')
elif a == 0:
    print('a is equal to 0.')
elif a > 0:
    print('a is positive.')
else:
    print("I don't know.")

a is positive.

Loops

Loops with the keyword while

i = 0
while i < 4:
    i += 1
print(f'i = {i}')

i = 4

i = 0
while i < 4:
    i += 1
    print(f'i = {i}')
    

i = 1
i = 2
i = 3
i = 4

Do it yourself

• Edit a script with your editor that calculates the average of a set of numbers. For example numbers = [67, 12, 2, 9, 23, 5]

  • using the functions sum and len
  • manually, using the keyword while

• Run the script

  • in spyder,
  • in a ipython session opened from another terminal,
  • with the command python.

A possible solution:

numbers = [67, 12, 2, 9, 23, 5]
local_sum = 0
i = 0
while i < len(numbers):
    local_sum = local_sum + numbers[i]
    i = i+1
avg = local_sum / len(numbers)
print(avg)
assert avg > 19.6666666 and avg < 19.7

19.666666666666668

Loops with the keyword for

for index in range(5):
    print(index, end=', ')

0, 1, 2, 3, 4, 

cities = ["grenoble", "paris", "berlin", "london"]
for city in cities:
    print(city, end=", ")

grenoble, paris, berlin, london, 

Print elements of a list and its indexes

index = 0 
cities = ["grenoble", "paris", "berlin", "london"]
for city in cities:
    print(f"({index}, {city})")
    index = index+1

(0, grenoble)
(1, paris)
(2, berlin)
(3, london)

This is a classical snippet, hence the built-in function enumerate

# the built-in function enumerate is very useful
cities = ["grenoble", "paris", "berlin", "london"]
for index, city in enumerate(cities): 
    print(f'({index}, {city})')
    cities[index] = city.capitalize()

(0, grenoble)
(1, paris)
(2, berlin)
(3, london)

cities

['Grenoble', 'Paris', 'Berlin', 'London']

Iterate synchoneously in many lists

The two following piece of code are equivalent

cities = ["grenoble", "paris", "berlin", "london"]
areas = [18.1, 105.4, 891.7, 1500.]
populations = [158198, 2165423, 3748148, 14372596]

for index, city in enumerate(cities):
    print(f"density({city}) = {populations[index]/areas[index]:.2f}")

density(grenoble) = 8740.22
density(paris) = 20544.81
density(berlin) = 4203.37
density(london) = 9581.73

cities = ["grenoble", "paris", "berlin", "london"]
areas = [18.1, 105.4, 891.7, 1500.]
populations = [158198, 2165423, 3748148, 14372596]

for city, area, population in zip(cities, areas, populations):
    print(f"density({city}) = {population/area:.2f}")

density(grenoble) = 8740.22
density(paris) = 20544.81
density(berlin) = 4203.37
density(london) = 9581.73

DIY Compute square of the L2 distance between two vectors

V1 = [ 1, 6, 9, 10, 4]

and

V2 = [ 0, 4, 1, 9, 6]

A possible solution:

V1 = [ 1, 6, 9, 10, 4]
V2 = [ 0, 4, 1, 9, 6]

sum_square_dists = 0
for v1, v2 in zip(V1, V2):
    sum_square_dists += (v1-v2)*(v1-v2)
print(f"sum square dists = {sum_square_dists}")

sum square dists = 74

Note on efficiency:

computing a^2 can be done using the power function (i.e. a***2) but is by far less efficient than computing ax2

Loops: keywords continue and break

• continue: passes the block in the loop and continues the loop.

for x in range(1, 8):
    if x == 5:
        continue
    print(x, end=', ')

1, 2, 3, 4, 6, 7, 

• break: stop the loop.

for x in range(1, 11):
    if x == 5:
        break
    print(x, end=', ')

1, 2, 3, 4, 

Do it yourself

• Modify your script to print the average of the first n elements for each n.

A possible solution:

l = [67, 12, 2, 9, 23, 5]
local_sum = 0
for index, elements in enumerate(l):
    local_sum += elements
    print(f"local average={local_sum}/{index+1}={local_sum/(index+1):.2f}")

local average=67/1=67.00
local average=79/2=39.50
local average=81/3=27.00
local average=90/4=22.50
local average=113/5=22.60
local average=118/6=19.67

Do it yourself

We build a list:

from random import shuffle, randint

n = 20
i = randint(0, n-1)
# i = 19 #special case for testing, element at the end of the list

print('integer remove from the list:', i)
l = list(range(n))
l.remove(i)
shuffle(l)
print('shuffled list: ', l)

integer remove from the list: 4
shuffled list:  [18, 11, 12, 13, 10, 3, 5, 17, 2, 9, 1, 19, 16, 6, 8, 0, 15, 14, 7]

One element has been removed:

• Find this element (given that you can change the ordering of l).
• Find this element (given that you cannot change the ordering of l).

A possible solution:

# we can change ordering, let's sort
print(l)
l_sorted = sorted(l)
print(l_sorted)
found = None
for idx, elem in enumerate(l_sorted):
    if elem != idx:
        found = idx
        break
if found is None:
    found = len(l)
print("missing ", found)
    

[18, 11, 12, 13, 10, 3, 5, 17, 2, 9, 1, 19, 16, 6, 8, 0, 15, 14, 7]
[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
missing  4

# we cannot sort -> higher complexity
for elem in range(len(l)+1):
    if elem not in l:
        break
print("missing ", elem)

missing  4

# another solution
actual_sum = sum(l)
len_l = len(l)
original_sum = (len_l + 1) * (len_l) // 2
print("missing ", original_sum - actual_sum)

missing  4

Exceptions and try, except syntax

Exceptions and errors are common in all codes. There is a good system to handle them in Python. Let's first see what gives such buggy code

l = ['a']
i = 1
print(l[i])

When these lines are executed, Python stops its execution and print a traceback:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-30-8df9cec1a0ec> in <module>()
      1 l = ['a']
      2 i = 1
----> 3 print(l[i])

IndexError: list index out of range

Exceptions and try, except syntax

An exception is something unexpected but that the code can/should handle.

Handling exception:

import math

num = 10
denom = 2
try:
    div = num / denom
except ZeroDivisionError as e:
    print(f"division by zero -> infinity {e}")
    div = math.inf
print(f"{num} / {denom} = {div}")
    

10 / 2 = 5.0

Remark: never use

except:

It means "except all errors and exceptions". A user Control-C is an exception (KeyboardInterrupt) so it would be caught and have no effect.

Full syntax

try:
    ...
except <exception1> as e1:
    # Handling of exception of type exception1
except <exception2> as e2:
    # Handling of exception of type exception2
else:
    # execute if no exception
    
finally:
    # always executed

Non exhaustive error list:

• ArithmeticError
• ZeroDivisionError
• IndexError
• KeyError
• AttributeError
• IOError
• ImportError
• NameError
• SyntaxError
• TypeError

Do it yourself:

Let us come back to the computation of the average of a vector. Let us consider that in a normal setting, the vector is not empty. If this happens, it means that a problem appears beforehand.

Modify the code that computes the average considering that an empty vector is an exception.

numbers = []
local_sum = 0

nb_elems = len(numbers)
for elem in numbers:
    local_sum += elem
average = local_sum / nb_elems    

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
Cell In[27], line 7
      5 for elem in numbers:
      6     local_sum += elem
----> 7 average = local_sum / nb_elems    

ZeroDivisionError: division by zero

A possible solution

numbers = []
local_sum = 0

nb_elems = len(numbers)
for elem in numbers:
    local_sum += elem
try:
    average = local_sum / nb_elems
except ZeroDivisionError as excpt:
    print(f"size of numbers is not supposed to be 0 !!!")
    local_sum = 0
    nb_elems = 1
else:
    print(f"size of numbers is {nb_elems}")
finally:
    average = local_sum / nb_elems
print(f"average of {numbers} is {average}")

size of numbers is not supposed to be 0 !!!
average of [] is 0.0

Do it yourself:

For each line of this string, append a variable of correct type in a list (i.e. "hello" should stay hello, 2 should become an int and 1.5 a float). Do it by catching errors.

Hints:

• int("a") and float("a") raise ValueError
• the above str_variable can be split using "\n"

str_variables = """hello
1.5
2"""

the_list_you_should_get = ["hello", 1.5, 2]

A possible solution:

split_list = []
for value in str_variables.split("\n"):
    try:
        value = int(value)
    except ValueError: 
        print(value, "is not a int")
        try:
            value = float(value)
        except ValueError:
            print(value, 'is not a float')
    split_list.append(value)

            
print(split_list)

hello is not a int
hello is not a float
1.5 is not a int
['hello', 1.5, 2]

Bonus: the Easier to Ask for Forgiveness than Permission (EAFP) principle

There are several ways to perform the previous task:

• In the solution presented above, we try first the conversion of the string in number and ask for forgiveness if we encounter an error (by using the except).
• Or we can test if the string is a number, using for example str.isdigit() or str.isdecimal() (asking for permission) and do the conversion only in this case.

Both approches will lead to the same result so it is rather a question of principle. In Python, it is generally prefered to use try and except to follow the EAFP principle. In this particular case, using try and except is far easier !

split_list = []
for value in str_variables.split("\n"):
    if value.isdecimal():
        value = int(value)
    else: 
        print(value, "is not a int")
        int_part, sep, dec_part = value.partition('.')
        if int_part.isdecimal() and dec_part.isdecimal():
            value = float(value)
        else:
            print(value, 'is not a float')
    split_list.append(value)

            
print(split_list)

hello is not a int
hello is not a float
1.5 is not a int
['hello', 1.5, 2]