# ---
# jupyter:
#   jupytext:
#     formats: notebooks//ipynb,python_scripts//py:percent
#     text_representation:
#       extension: .py
#       format_name: percent
#       format_version: '1.2'
#       jupytext_version: 1.2.1
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# %% [markdown]
# In this notebook, we will look at necessary steps that happen before any machine learning takes place.
# * load the data
# * look at the variables in the dataset, in particular make the difference
#   between numerical and categorical variables, which need different
#   preprocessing in most machine learning workflows
# * visualize the distribution of the variables to gain some insights into the dataset.

# %%
# Inline plots
# %matplotlib inline

# plotting style
import seaborn as sns

# %% [markdown]
# ## Loading the adult census dataset

# %% [markdown]
# We will use data from the "Current Population adult_census" from 1994 that we
# downloaded from [OpenML](http://openml.org/).

# %%
import pandas as pd

adult_census = pd.read_csv("https://www.openml.org/data/get_csv/1595261/adult-census.csv")

# Or use the local copy:
# adult_census = pd.read_csv('../datasets/adult-census.csv')

# %% [markdown]
# We can look at the OpenML webpage to know more about this dataset.

# %%
from IPython.display import IFrame
IFrame('https://www.openml.org/d/1590', width=1200, height=600)

# %% [markdown]
# ## Look at the variables in the dataset
# The data are stored in a pandas dataframe.

# %%
adult_census.head()

# %% [markdown]
# The column named **class** is our target variable (i.e., the variable which
# we want to predict). The two possible classes are `<= 50K` (low-revenue) and
# `> 50K` (high-revenue).

# %%
target_column = 'class'
adult_census[target_column].value_counts()

# %% [markdown]
# Note: classes are slighly imbalanced. Class imbalance happens often in
# practice and may need special techniques for machine learning. For example in
# a medical setting, if we are trying to predict whether patients will develop
# a rare disease, there will be a lot more sane patients than ill patients in
# the dataset.

# %% [markdown]
# The dataset contains both numerical and categorical data. Numerical values
# can take continuous values for example `age`. Categorical values can have a
# finite number of values, for exemple `native-country`.

# %%
numerical_columns = ['age', 'education-num', 'capital-gain', 'capital-loss',
                     'hours-per-week']
categorical_columns = ['workclass', 'education', 'marital-status', 'occupation',
                       'relationship', 'race', 'sex', 'native-country']
all_columns = numerical_columns + categorical_columns + [target_column]

adult_census = adult_census[all_columns]

# %% [markdown]
# Note that for simplicity, we have ignored the "fnlwgt" (final weight) column
# that was crafted by the creators of the dataset when sampling the dataset to
# be representative of the full census database.

# %% [markdown]
# ## Inspect the data
# Before building a machine learning model, it is a good idea to look at the
# data:
# * maybe the task you are trying to achieve can be solved without machine
#   learning
# * you need to check that the data you need for your task is indeed present in
# the dataset
# * inspecting the data is a good way to find peculiarities. These can can
#   arise in the data collection (for example, malfunctioning sensor or missing
#   values), or the way the data is processed afterwards (for example capped
#   values).

# %% [markdown]
# Let's look at the distribution of individual variables, to get some insights
# about the data. We can start by plotting histograms, note that this only
# works for numerical variables:

# %%
adult_census.hist(figsize=(20, 10));

# %% [markdown]
# We can already make a few comments about some of the variables:
# * age: there are not that many points for 'age > 70'. The dataset description
# does indicate that retired people have been filtered out (`hours-per-week > 0`).
# * education-num: peak at 10 and 13, hard to tell what it corresponds to
# without looking much further. We'll do that later in this notebook.
# * hours per week at 40, this was very likely the standard of working hours at
# the time of the data collection
# * most values of capital-gain and capital-loss are close to zero

# %% [markdown]
# For categorical variables, we can look at the distribution of values:

# %%
adult_census['sex'].value_counts()

# %%
adult_census['education'].value_counts()

# %% [markdown]
# `pandas_profiling` is a nice tool for inspecting the data (both numerical and
# categorical variables).

# %%
import pandas_profiling
adult_census.profile_report()

# %% [markdown]
# As noted above, `education-num` distribution has two clear peaks around 10
# and 13. It would be reasonable to expect that 'education-num' is the number of
# years of education. Let's look at the relationship between education and
# education-num.
# %%
pd.crosstab(index=adult_census['education'], columns=adult_census['education-num'])

# %% [markdown]
# This shows that education and education-num are redundant. For
# example, `education-num=2` is equivalent to `education='1st-4th'`. In
# practice that means we can remove `education-num` without losing information.
# Note that having redundant (or highly correlated) columns can be a problem
# for machine learning algorithms.

# %% [markdown]
# Another way to inspect the data is to do a pairplot and show how variable
# differ according to the class. In the diagonal you can see the distribution
# of individual variables. The plots on the off-diagonal can reveal interesting
# interactions between variables.

# %%
n_samples_to_plot = 5000
columns = ['age', 'education-num', 'hours-per-week']
sns.pairplot(data=adult_census[:n_samples_to_plot] , vars=columns,
             hue=target_column, plot_kws={'alpha': 0.2}, height=4,
             diag_kind='hist');

# %%
sns.pairplot(data=adult_census[:n_samples_to_plot], x_vars='age', y_vars='hours-per-week',
             hue=target_column, markers=['o', 'v'], plot_kws={'alpha': 0.2}, height=12);

# %% [markdown]
#
# By looking at the data you could infer some hand-written rules to predict the
# class:
# * if you are young (less than 25 year-old roughly), you are in the `<= 50K` class.
# * if you are old (more than 70 year-old roughly), you are in the `<= 50K` class.
# * if you work part-time (less than 40 hours roughly) you are in the `<= 50K` class.
#
# These hand-written rules could work reasonably well without the need for any
# machine learning. Note however that it is not very easy to create rules for
# the region `40 < hours-per-week < 60` and `30 < age < 70`. We can hope that
# machine learning can help in this region. Also note that visualization can
# help creating hand-written rules but is limited to 2 dimensions (maybe 3
# dimensions), whereas machine learning models can build models in
# high-dimensional spaces.
#
# Another thing worth mentioning in this plot: if you are young (less than 25
# year-old roughly) you tend to work less and if you are old (more than 70
# year-old roughly). This is a non-linear relationship between age and hours
# per week. Some machine learning models can only capture linear interaction so
# this may be a factor when deciding which model to chose.
#
# In a machine-learning setting, we will use an algorithm to automatically
# decide what should be the "rules" in order to predict on new data. We can
# check which set of simple rule a decision tree would grasp using the same
# data.

# %%
def plot_tree_decision_function(tree, X, y, ax):
    """Plot the different decision rules found by a `DecisionTreeClassifier`.

    Parameters
    ----------
    tree : DecisionTreeClassifier instance
        The decision tree to inspect.
    X : dataframe of shape (n_samples, n_features)
        The data used to train the `tree` estimator.
    y : ndarray of shape (n_samples,)
        The target used to train the `tree` estimator.
    ax : matplotlib axis
        The matplotlib axis where to plot the different decision rules.
    """
    import numpy as np
    from scipy import ndimage

    h = 0.02
    x_min, x_max = 0, 100
    y_min, y_max = 0, 100
    xx, yy = np.meshgrid(
        np.arange(x_min, x_max, h), np.arange(y_min, y_max, h)
    )

    Z = tree.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]
    Z = Z.reshape(xx.shape)
    faces = tree.tree_.apply(
        np.c_[xx.ravel(), yy.ravel()].astype(np.float32)
    )
    faces = faces.reshape(xx.shape)
    border = ndimage.laplace(faces) != 0
    ax.scatter(
        X.iloc[:, 0], X.iloc[:, 1], c=np.array(['tab:blue', 'tab:red'])[y],
        s=60, alpha=0.7
    )
    ax.contourf(xx, yy, Z, alpha=.4, cmap='RdBu_r')
    ax.scatter(xx[border], yy[border], marker='.', s=1)
    ax.set_xlabel(X.columns[0])
    ax.set_ylabel(X.columns[1])
    ax.set_xlim([x_min, x_max])
    ax.set_ylim([y_min, y_max])
    sns.despine(offset=10)

# %%
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rcParams.update(mpl.rcParamsDefault)  # reset the plotting style

# %%
from sklearn.preprocessing import LabelEncoder

# select a subset of data
data_subset = adult_census[:n_samples_to_plot]
X = data_subset[["age", "hours-per-week"]]
y = LabelEncoder().fit_transform(data_subset[target_column].to_numpy())

# %% [markdown]
# We will create a decision tree which we will keep really simple on purpose.
# We will only allow a maximum of 2 rules to interpret the results.

# %%
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import plot_tree

max_leaf_nodes = 3
tree = DecisionTreeClassifier(max_leaf_nodes=max_leaf_nodes, random_state=0)
tree.fit(X, y)

# %% [markdown]
# We can now first check the set of rules learnt by the tree and check
# visually what theses rules look like.

# %%
# plot the tree structure
fig, ax = plt.subplots()
plot_tree(tree, ax=ax)

# plot the decision function learned by the tree
fig, ax = plt.subplots()
plot_tree_decision_function(tree, X, y, ax=ax)

# %% [markdown]
# Allowing only 3 leaves in the tree, we get similar rules than the one
# designed by hand:
# * the persons younger than 28.5 year-old will be considered in the class
#   earning `<= 50K`.
# * the persons older than 28.5 and working more than 40.5 hours-per-week
#   will be considered in the class earning `> 50K`, while the one working
#   below 40.5 hours-per-week, will be considered in the class
#   earning `<= 50K`.

# %% [markdown]
#
# In this notebook we have:
# * loaded the data from a CSV file using `pandas`
# * looked at the kind of variables in the dataset, and make the difference
#   between categorical and numerical variables.
# * inspected the data with `pandas`, `seaborn` and `pandas_profiling`. Data inspection
#   can allow you to decide whether using machine learning is appropriate for
#   your data and to notice potential peculiarities in your data.


################################################################################
## COMP3122 - Home assignment 2 - Due: Tuesday Nov 20, 23:59                  ##
################################################################################

# Read the instructions below and complete the function for each task.
#  - The __main__ section at the bottom contains test cases which will also
#    provide an estimate of your grade.
#  - Submit the completed assignment by emailing it as an attachment to
#    graderbot@kamrik.org (as a .py file, no zip)
#  - You can submit up to 5 times, the best submissions will be used.
#
# Scoring: The score calculated here is just an estimate, actual score will be
# calculated by other code (otherwise it would be too easy to cheat). For the
# score to be calculated properly, importing this file must not crash with an
# unhandled exception. Submissions where the estimate differs from the actual
# grade significantly will be inspected manually.
#
# General hints:
#  - Brush up your memory about Python data types: List, Dict and Set
#  - Feel free to define helper functions if you want to
#  - Experiment with your code - computer is much faster than you at
#    interpreting code
#  - Don't overthink
#
# Useful references
# - Quick Python reference: https://learnxinyminutes.com/docs/python3/

assignment_name = 'assignment2'
assignment_version = '1.0'

## Task 1 ######################################################################
# Replace 999 with your student id
student_id = 100649804

## Task 2 - Average-by #########################################################
# You are only allowed to use the Python standard libraries for this task:
# No NumPy, no Pandas !!!
#
# Given a table of data represented as a list of lists, compute the averages
# of a specified column `col`, within groups specified by column `by`
# `col` and `by` are column indices.
# The input table is a list lists, each inner list is a row of the table.
# The output should be a dictionary.
#
# Example:
# Calculating the average of col 0, grouped by col 3
# [[15 29  6  2]
#  [16  9  8  0]
#  [ 7 27 16  0]]
# OUT - {0: 11.5, 2: 15.0}
#     - OK
#
# Make sure to try edge cases, such as an empty table.
# No need to check input validity.
def avgby(tbl, col, by):

    averages = {}
    counts = {}

    if range(len(tbl)) == 0:
        return {}

    for x in range(len(tbl)):
        if tbl[x][by] in averages.keys():
            averages[tbl[x][by]] += tbl[x][col]
            counts[tbl[x][by]] += 1
        if tbl[x][by] not in averages.keys():
            averages[tbl[x][by]] = tbl[x][col]
            counts[tbl[x][by]] = 1

    for x in averages:
        averages[x] = averages[x]/counts[x]
    return averages

## Task 3 - Merge ##############################################################
# You are only allowed to use the Python standard libraries for this task:
# No NumPy, no Pandas !!!
#
# Write a merge function to implement a _inner_ join of two data tables each
# represented as a list of rows where each row is a list.
#
# Inputs:
#  - `left` and `right` - data tables
#  - `left_on` and `right_on` - the column indices containing the key to merge
#     on in each of the tables.
#
# Example 1:
# Merging on column 0 of each table (person name)
# left = [['Bob', 'Accounting'],
#         ['Jake', 'Engineering'],
#         ['Lisa', 'HR']]
# right = [['Lisa', 2004],
#         ['Bob', 2008],
#         ['Jake', 2012],
#         ['Sue', 2014]]
# merged =   [['Bob', 'Accounting', 'Bob', 2008],
#             ['Jake', 'Engineering', 'Jake', 2012],
#             ['Lisa', 'HR', 'Lisa', 2004]]
#
# Example 2:
# Merging on columns 1 and 0, note that 'B' appears twice in the right table.
# left = [[1, 'A'], [2, 'B'], [3, 'C']]
# right = [['A', 2004], ['B', 2008], ['C', 2012], ['B', 2014]]
# merged =   [[1, 'A', 'A', 2004],
#             [2, 'B', 'B', 2008],
#             [2, 'B', 'B', 2014],
#             [3, 'C', 'C', 2012]]
#
# "Inner join" means that only rows with data from both tables are returned.
# Each row in the merged table should be the concatenation of corresponding
# rows from the left and right tables.
# In terms of Python lists:
#     row_in_merged_tbl = row_from_left + row_from_right
# where the following is true
#     row_from_left[left_on] == row_from_right[right_on]
# This merge is essentially equivalent to pd.merge(tbl1, tbl2, left_on=?, right_on=?)
# You can read about Pandas merge function in the book:
# https://jakevdp.github.io/PythonDataScienceHandbook/03.07-merge-and-join.html#Categories-of-Joins
# Note what happens when a key appears multiple times in both tables
# https://jakevdp.github.io/PythonDataScienceHandbook/03.07-merge-and-join.html#Many-to-many-joins

def merge(left, right, left_on, right_on):

    merged = []
    for x in left:
        for y in right:
            if y[right_on] == x[left_on]:
                merged.append(y+x)
    return merged

################################## Tests #######################################
if __name__ == '__main__':
    # Convenience variables for printouts
    OUT = 'OUT ╦'
    OK  = '    ╚ OK'
    ERR = '    ╚ ERROR:'

    total_score = 0

    print('####### Task1: student ID ########################################')
    print(OUT, 'Student ID:', student_id)
    if student_id == 999 or student_id is None or type(student_id) != int:
        print(ERR, 'student_id not set')
    else:
        print(OK)

    # This is for tests only, don't use NumPy in your solutions.
    import numpy as np
    import pandas as pd

    print()
    print('####### Task 2: average by ########################################')
    task_score = 0

    # Test the case of empty table
    if avgby([], 1, 0) != {}:
        print(ERR, "avgby([],...) should return and empty dict")
    task_score = 5  # 5 points for avgby() not crashing so far.

    # A helper function for comparing dicts with floating point values.
    def cmp_dict(d1, d2):
        if set(d1.keys()) != set(d2.keys()):
            return False
        for k in d1:
            if d1[k] - d2[k] > 1e-6:
                return False
        return True

    # Run a bunch of randomly generated cases
    # Once your solution works for 10 cases, try running it with a 1000 cases.
    n_cases = 10
    rng = np.random.RandomState(seed=1) # Settin the seed for repeatability
    for test_case in range(n_cases):
        # Generating random input data
        nrows = rng.randint(15)
        ncols = rng.randint(2, 6)
        # The input table will be of shape (nrows, ncols) containing integers
        arr = rng.randint(30, size=(nrows, ncols))
        by = rng.randint(ncols)

        # Randomly decide which column to average
        avg_col = rng.randint(ncols)

        # Change the 'by' column so that it contains multiple copies of the same
        # number, this way group by makes more sense.
        groupmod = rng.randint(1, nrows+2)
        arr[:, by] %= groupmod

        print("Case number ", test_case)
        print("Calculating the average of col %d, grouped by col %d" % (avg_col, by))
        print(arr)

        # Compute the answer using NumPy
        by_arr = arr[:, by]
        avg_arr = arr[:, avg_col]
        test_ans = dict((k, avg_arr[by_arr==k].mean()) for k in set(by_arr))

        # Convert NumPy array into list of lists
        lst = arr.tolist()
        # Run the solution function
        avgs = avgby(lst, avg_col, by)

        # Compare outputs
        print(OUT, avgs)
        if cmp_dict(avgs, test_ans):
            print(OK)
            task_score += 50 / n_cases
        else:
            print(ERR, 'Expected: ', test_ans)

    # End of test case loop
    total_score += task_score

    print()
    print('####### Task 3: merge #############################################')
    task_score = 0
    if merge([], [], 0, 0) != []:
        print(ERR, "merge([], [],...) should return and empty list")

    # 5 points for merge() not crashing so far.
    task_score = 5

    # Function to compare your merge with Pandas merge :)
    def check_merged(merged, tbl1, tbl2, left_on, right_on):
        df1 = pd.DataFrame(tbl1)
        df2 = pd.DataFrame(tbl2)
        n1 = len(df1.columns)
        # Make sure we have distinct column names between the to data frames.
        # Column names here are integers starting from 0, add n1 to each of them.
        df2.columns += n1
        # Pandas merge
        dfpd = pd.merge(df1, df2, left_on=left_on, right_on=(right_on+n1))
        # Your answer converted to DataFrame
        dfm = pd.DataFrame(merged)
        print(dfm)
        # Special case of empty merge and 0 length tables.
        if len(dfpd) == 0 and len(merged) == 0:
            return True
        if len(dfpd) == 0 and len(merged) != 0:
            print(ERR, 'Expected empty list as output')
            return False
        if len(merged) == 0:
            print(ERR, 'Got empty table as output. Expected table:')
            print (dfpd)
            return False
        # Sort by all columns to ensure the same order
        dfpd = dfpd.sort_values(by=list(dfpd.columns))
        dfm = dfm.sort_values(by=list(dfm.columns))
        # Compare shapes first
        if dfpd.shape != dfm.shape:
            print(ERR, 'Wrong shape, expected table:\n', dfpd)
            return False
        # Compare content
        if not dfpd.equals(dfm):
            print(ERR, 'Correct shape, but not contents. Expected table:\n', dfpd)
        return True

    #### Test cases ####
    print()
    print('Case 1:')
    tbl1 = [['Bob', 'Accounting'], ['Jake', 'Engineering'], ['Lisa', 'HR']]
    tbl2 = [['Lisa', 2004], ['Bob', 2008], ['Jake', 2012], ['Sue', 2014]]
    left_on = 0
    right_on = 0
    merged = merge(tbl1, tbl2, left_on, left_on)

    if check_merged(merged, tbl1, tbl2, left_on, right_on):
        print(OK)
        task_score += 10

    print()
    print('Case 2:')
    tbl1 = [[1, 'A'], [2, 'B'], [3, 'C']]
    tbl2 = [['A', 2004], ['B', 2008], ['C', 2012], ['B', 2014]]
    left_on = 1
    right_on = 0
    merged = merge(tbl1, tbl2, left_on, right_on)
    if check_merged(merged, tbl1, tbl2, left_on, right_on):
        print(OK)
        task_score += 10

    print()
    print('Case 3 - empty merge:')
    tbl1 = [[1, 'A']]
    tbl2 = [['B', 2004]]
    left_on = 1
    right_on = 0
    merged = merge(tbl1, tbl2, left_on, right_on)
    if check_merged(merged, tbl1, tbl2, left_on, right_on):
        print(OK)
        task_score += 10

    print()
    print('Case 4:')
    tbl1 = [[1, 'A'], [2, 'A'], [3, 'A']]
    tbl2 = [['A', 2004, True], ['A', 2008, False]]
    left_on = 1
    right_on = 0
    merged = merge(tbl1, tbl2, left_on, right_on)
    if check_merged(merged, tbl1, tbl2, left_on, right_on):
        print(OK)
        task_score += 10

    print()
    print('Case 5:')
    tbl1 = [[1, 'A'], [2, 'A'], [3, 'A']]
    tbl2 = [['A', 2004, 1], ['A', 2008, 2]]
    left_on = 0
    right_on = 2
    merged = merge(tbl1, tbl2, left_on, right_on)
    if check_merged(merged, tbl1, tbl2, left_on, right_on):
        print(OK)
        task_score += 10

    total_score += task_score

    ### Estimated total score
    print('ESTIMATED SCORE: %d (if above 100 will be set to 100)' % total_score)


from resemblyzer import VoiceEncoder, preprocess_wav
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
import pandas as pd

def get_top_similar(file, top):
    top_names = []

    wav = preprocess_wav(file)
    encoder = VoiceEncoder()
    embed = encoder.embed_utterance(wav)

    cs = cosine_similarity(X=embeded_voices, Y=embed.reshape(1, -1))
    cs_sorted = np.argsort(cs, axis=0)[::-1][:, 0]
    top_similarity = cs[cs_sorted[:top]]
    top_ids = ids[cs_sorted[:top]]

    for each in top_ids:
        top_names.append(mydict[each])

    return top_names, top_similarity, top_ids

# IMPORT DATA

df = pd.read_csv("ids_names.csv")
mydict = dict(zip(df.id,df.name)) #Creates dictionary id - celebrity
embeded_voices = np.load("embeded_voices.npy")
ids = np.load("ids_voices.npy")

top = 5

names, scores, res_ids = get_top_similar("your_audio.wav", top = top)
print("\nYour best match is {} with {:.0f}% similarity.".format(names[0].replace("_"," "),scores[0][0]*100))

print("\n Your top {} is the following:".format(top))

for name in names:
    print("\n" + name.replace("_", " "))


import numpy as np
from keras.preprocessing.image import img_to_array
from keras.models import load_model
import imutils
import cv2

cascade_path = 'harrcascade_files/haarcascade_frontalface_default.xml'
model_weight = 'borneel_xception.hdf5'

face_detection = cv2.CascadeClassifier(cascade_path)
emotion_classifier = load_model(model_weight, compile=False)
face_emotions = ["ANGRY" ,"DISGUSTED","FEARFUL", "HAPPY", "SAD", "SURPRISED", "NEUTRAL"]
feeling_faces = []

for (index, emotion) in enumerate(face_emotions):
    feeling_faces.append(cv2.imread('emojis/' + emotion + '.png', -1))

camera = cv2.VideoCapture(0)
while True:
    frame1 = camera.read()[1]
    frame1 = imutils.resize(frame1,width=700)
    gray = cv2.cvtColor(frame1, cv2.COLOR_BGR2GRAY)
    faces = face_detection.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=5,minSize=(30,30),flags=cv2.CASCADE_SCALE_IMAGE)

    chart = np.zeros([300, 500, 3], dtype="float32")
    frame2 = frame1.copy()
    if len(faces) > 0:
        faces = sorted(faces, reverse=True,
        key=lambda x: (x[2] - x[0]) * (x[3] - x[1]))[0]
        (face_x, face_y, face_width, face_height) = faces

        roi = gray[face_y:face_y + face_height, face_x:face_x + face_width]
        roi = cv2.resize(roi, (64, 64))
        roi = roi.astype("float") / 255.0
        roi = img_to_array(roi)
        roi = np.expand_dims(roi, axis=0)

        predicts = emotion_classifier.predict(roi)[0]
        emotion_probability = np.max(predicts)
        label = face_emotions[predicts.argmax()]
    else:
        continue

    for (i, (emotion, prob)) in enumerate(zip(face_emotions, predicts)):
                text = "{}: {:.2f}%".format(emotion, prob * 100)
                emoji_face = feeling_faces[np.argmax(predicts)]
                width = int(prob * 500)
                cv2.rectangle(chart, (7, (i * 35) + 5), (width, (i * 35) + 35), (0, 255, 0), -1)
                cv2.putText(chart, text, (10, (i * 35) + 23),
                cv2.FONT_HERSHEY_SIMPLEX, 0.45,(255, 0, 0), 2)
                cv2.putText(frame2, label, (face_x, face_y - 10),
                cv2.FONT_HERSHEY_SIMPLEX, 0.45, (255, 0, 0), 2)
                cv2.rectangle(frame2, (face_x, face_y), (face_x + face_width, face_y + face_height), (0, 255, 0), 2)

    cv2.imshow('Emotion Detection', frame2)
    cv2.imshow("Emotion Probabilities", chart)
    cv2.imshow("Emoji Representation", emoji_face)
    if cv2.waitKey(1) & 0xFF == 27:
        break

camera.release()
cv2.destroyAllWindows()


#  Version: 2020.02.21
#
#  MIT License
#
#  Copyright (c) 2018 Jiankang Deng and Jia Guo
#
#  Permission is hereby granted, free of charge, to any person obtaining a copy
#  of this software and associated documentation files (the "Software"), to deal
#  in the Software without restriction, including without limitation the rights
#  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
#  copies of the Software, and to permit persons to whom the Software is
#  furnished to do so, subject to the following conditions:
#
#  The above copyright notice and this permission notice shall be included in all
#  copies or substantial portions of the Software.
#
#  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
#  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
#  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
#  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
#  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
#  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
#  SOFTWARE.
#

from __future__ import division
import collections
import mxnet as mx
import numpy as np
from numpy.linalg import norm
import mxnet.ndarray as nd
from ..model_zoo import model_zoo
from ..utils import face_align

__all__ = ['FaceAnalysis',
           'Face']

Face = collections.namedtuple('Face', [
        'bbox', 'landmark', 'det_score', 'embedding', 'gender', 'age', 'embedding_norm', 'normed_embedding'])

Face.__new__.__defaults__ = (None,) * len(Face._fields)

class FaceAnalysis:
    def __init__(self, det_name='retinaface_r50_v1', rec_name='arcface_r100_v1', ga_name='genderage_v1'):
        assert det_name is not None
        self.det_model = model_zoo.get_model(det_name)
        if rec_name is not None:
            self.rec_model = model_zoo.get_model(rec_name)
        else:
            self.rec_model = None
        if ga_name is not None:
            self.ga_model = model_zoo.get_model(ga_name)
        else:
            self.ga_model = None

    def prepare(self, ctx_id, nms=0.4):
        self.det_model.prepare(ctx_id, nms)
        if self.rec_model is not None:
            self.rec_model.prepare(ctx_id)
        if self.ga_model is not None:
            self.ga_model.prepare(ctx_id)

    def get(self, img, det_thresh = 0.8, det_scale = 1.0, max_num = 0):
        bboxes, landmarks = self.det_model.detect(img, threshold=det_thresh, scale = det_scale)
        if bboxes.shape[0]==0:
            return []
        if max_num>0 and bboxes.shape[0]>max_num:
            area = (bboxes[:,2]-bboxes[:,0])*(bboxes[:,3]-bboxes[:,1])
            img_center = img.shape[0]//2, img.shape[1]//2
            offsets = np.vstack([ (bboxes[:,0]+bboxes[:,2])/2-img_center[1], (bboxes[:,1]+bboxes[:,3])/2-img_center[0] ])
            offset_dist_squared = np.sum(np.power(offsets,2.0),0)
            values = area-offset_dist_squared*2.0 # some extra weight on the centering
            bindex = np.argsort(values)[::-1] # some extra weight on the centering
            bindex = bindex[0:max_num]
            bboxes = bboxes[bindex, :]
            landmarks = landmarks[bindex, :]
        ret = []
        for i in range(bboxes.shape[0]):
            bbox = bboxes[i, 0:4]
            det_score = bboxes[i,4]
            landmark = landmarks[i]
            _img = face_align.norm_crop(img, landmark = landmark)
            embedding = None
            embedding_norm = None
            normed_embedding = None
            gender = None
            age = None
            if self.rec_model is not None:
                embedding = self.rec_model.get_embedding(_img).flatten()
                embedding_norm = norm(embedding)
                normed_embedding = embedding / embedding_norm
            if self.ga_model is not None:
                gender, age = self.ga_model.get(_img)
            face = Face(bbox = bbox, landmark = landmark, det_score = det_score, embedding = embedding, gender = gender, age = age
                    , normed_embedding=normed_embedding, embedding_norm = embedding_norm)
            ret.append(face)
        return ret


#!/usr/bin/env python
import argparse
import os
import sys

import csv
import h5py
import tensorflow.keras as keras
import tensorflow as tf
import numpy as np
from tqdm import tqdm
import cv2
import SimpleITK as sitk
import time

tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)
tf.compat.v1.enable_v2_behavior()

if __name__ == "__main__" and __package__ is None:
    sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..'))
    import fl_covid.bin  # noqa: F401
    __package__ = "fl_covid.bin"

# Change these to absolute imports if you copy this script outside the fl_covid package.
from ..utils.anchors import compute_overlap
from .. import models
from ..preprocessing.csv_generator import CSVGenerator
from ..utils.eval import _compute_ap, _get_annotations, _get_annotations_and_img_path
from ..utils.config import read_config_file, parse_anchor_parameters
from ..utils.keras_version import check_keras_version
from ..utils.visualization import draw_detections, draw_annotations
from keras_retinanet.utils.visualization import draw_box, label_color, draw_caption
from keras_retinanet.bin.train_edit import create_models
from keras_retinanet.layers.filter_detections import filter_detections
from keras_retinanet.utils.image import preprocess_image, resize_image
from keras_retinanet.bin.evaluate_internal_patient_wise import draw_colorful_result, evaluate_from_npy

def get_session():
    """ Construct a modified tf session.
    """
    config = tf.ConfigProto()
    config.gpu_options.allow_growth = True
    return tf.Session(config=config)

def draw_detections(image, boxes, scores, labels, color=None, label_to_name=None, slice_id=None, bbox_writer=None, score_threshold=0.4):  # score_threshold used to be 0.5
    """ Draws detections in an image.

    # Arguments
        image           : The image to draw on.
        boxes           : A [N, 4] matrix (x1, y1, x2, y2).
        scores          : A list of N classification scores.
        labels          : A list of N labels.
        color           : The color of the boxes. By default the color from keras_retinanet.utils.colors.label_color will be used.
        label_to_name   : (optional) Functor for mapping a label to a name.
        score_threshold : Threshold used for determining what detections to draw.
    """
    selection = np.where(scores > score_threshold)[0]
    for i in selection:
        c = color if color is not None else label_color(labels[i])

        if bbox_writer is not None and slice_id is not None:
            tar_path = 'slice_{}.png'.format(slice_id)
            b = np.array(boxes[i, :]).astype(int)
            bbox_writer.writerow([tar_path]+ [b[0],b[1],b[2],b[3]]+['lesion'])

        draw_box(image, boxes[i, :], color=c)

        # draw labels
        caption = (label_to_name(labels[i]) if label_to_name else str(labels[i])) + ': {0:.2f}'.format(scores[i])
        draw_caption(image, boxes[i, :], caption)

def create_generator(args):
    """ Create generators for evaluation.
    """

    if args.dataset_type == 'csv':
        validation_generator = CSVGenerator(
            args.annotations,
            args.classes,
            image_min_side=args.image_min_side,
            image_max_side=args.image_max_side,
            config=args.config,
            shuffle_groups=False
        )
    else:
        raise ValueError('Invalid data type received: {}'.format(args.dataset_type))

    return validation_generator

def _seg_filter(bboxes,scores_sort,seg):
    # print('scores_sort')
    # print(scores_sort.shape)
    # print(scores_sort)
    # print('indices')
    # print(type(indices[scores_sort]))
    # print(indices[scores_sort])
    # print(indices)
    # # select detections
    # image_boxes      = boxes[0, indices[scores_sort], :]
    # print('seletec_boxes', image_boxes.shape)
    # print(type(image_boxes))
    # print(image_boxes)
    # image_boxes_filtered = []
    # seg = cv2.imread('/Users/jemary/Data/DataSet/COVID-19/COVID-19 image data collection(public1)/007_seg/slice_seg_{}.png'.format(img_idx))
    # print(seg.shape)
    image_boxes = bboxes
    inner = np.asarray([],dtype=np.bool)
    flag = False
    for i in range(image_boxes.shape[0]):
        x1 = int(image_boxes[i][0])
        y1 = int(image_boxes[i][1])
        x2 = int(image_boxes[i][2])
        y2 = int(image_boxes[i][3])
        x1 = 511 if x1 > 511 else x1
        y1 = 511 if y1 > 511 else y1
        x2 = 511 if x2 > 511 else x2
        y2 = 511 if y2 > 511 else y2
        # print(scores_sort)
        # print(scores_sort.shape)
        if (seg[y1,x1,:] == 0).all() and (seg[y2,x2,:] == 0).all() and  (seg[y1,x2,:] == 0).all() and  (seg[y2,x1,:] == 0).all():
            inner = np.append(inner,False)
            flag=True
            # scores_sort = np.delete(scores_sort,i,axis=0)
        else:
            inner = np.append(inner, True)
    # print(inner)
    # cnt = 1
    # if flag:
    #     if cnt > 0:
    #         print("FP out of lung filtered")
    #     cnt -= 1
    scores_sort = scores_sort[inner]
    # print('scores_sort after filter')
    # print(scores_sort.shape)
    # print(scores_sort)
    return scores_sort

def _print_ensemble_detections_to_npy(args, generator, model_list, client_idx, client_name, patient_name, score_threshold=0.05, max_detections=100, save_path=None):
    all_detections = [[None for i in range(generator.num_classes()) if generator.has_label(i)] for j in range(generator.size())]

    detection_out = np.zeros([generator.size(),512,512,3])
    # detection_out = np.zeros([generator.size(),512,512])
    attention_out = np.zeros([generator.size(),512,512])
    mask_out = np.zeros([generator.size(),512,512])

    results = open(os.path.join(save_path, '{}_{}_output_bbox.csv'.format(client_name, patient_name)), 'w', newline='')
    result_writer = csv.writer(results, delimiter=',')

    for i in tqdm(range(generator.size()), desc='Running network on {} {}: '.format(client_name, patient_name)):
        raw_image    = generator.load_image(i)
        # image = np.expand_dims(raw_image.copy(), axis=-1)
        # image = np.repeat(image, 3, axis=-1)
        # image        = generator.preprocess_image(image)
        image        = generator.preprocess_image(raw_image.copy())
        image, scale = generator.resize_image(image)

        if keras.backend.image_data_format() == 'channels_first':
            image = image.transpose((2, 0, 1))

        # run network
        # boxes, scores, labels = model.predict_on_batch(np.expand_dims(image, axis=0))[:3]
        all_boxes = np.empty([1,0,4],dtype=np.float32)
        all_scores = np.empty([1,0], dtype=np.float32)
        all_labels = np.empty([1,0], dtype=np.int32)
        masks = np.zeros([1,512,512,1], dtype=np.float32)
        attention_map = np.zeros([1,512,512,1], dtype=np.float32)
        for site_model in model_list:
            site_boxes, site_scores, site_labels, site_masks, site_attention_map = site_model.predict_on_batch(np.expand_dims(image, axis=0))
            site_boxes = site_boxes.numpy()
            site_scores = site_scores.numpy()
            site_labels = site_labels.numpy()
            site_masks = site_masks.numpy()
            site_attention_map = site_attention_map.numpy()
            if np.squeeze(site_attention_map).shape[0] !=512:
                resized_attn_map = cv2.resize(np.squeeze(site_attention_map),(512,512))
                site_attention_map = np.expand_dims(resized_attn_map,axis=-1)
                site_attention_map = np.expand_dims(site_attention_map,axis=0)
            all_boxes = np.concatenate([all_boxes, site_boxes],axis=-2)
            all_scores = np.concatenate([all_scores, site_scores],axis=-1)
            all_labels = np.concatenate([all_labels, site_labels],axis=-1)
            masks = np.add(masks, site_masks/len(model_list))
            attention_map = np.add(attention_map, site_attention_map/len(model_list))
            # print('-------------EACH MODEL----------')
            # print('----boxes----')
            # print(type(site_boxes))
            # print(site_boxes.shape)
            # print(site_boxes.dtype)
            # print('----scores----')
            # print(type(site_scores))
            # print(site_scores.shape)
            # print(site_scores.dtype)
            # print('----labels----')
            # print(type(site_labels))
            # print(site_labels.shape)
            # print(site_labels.dtype)
            # print('----masks----')
            # print(type(site_masks))
            # print(site_masks.shape)
            # print(site_masks.dtype)
            # print('----attn map----')
            # print(type(site_attention_map))
            # print(site_attention_map.shape)
            # print(site_attention_map.dtype)
            # print('total_boxes')
            # print(all_boxes.shape)
            # print(all_boxes.dtype)
            # print('total_scores')
            # print(all_scores.shape)
            # print(all_scores.dtype)
            # print('total_labels')
            # print(all_labels.shape)
            # print(all_labels.dtype)
        # print(np.squeeze(all_boxes).shape)
        # print(np.expand_dims(np.squeeze(all_scores),axis=-1).shape)
        out_boxes, out_scores, out_labels = filter_detections(
            np.squeeze(all_boxes),
            np.expand_dims(np.squeeze(all_scores),axis=-1),
            other=[],
            class_specific_filter=True,
            nms=True)
        out_boxes = np.expand_dims(np.squeeze(out_boxes),axis=0)
        out_scores = np.expand_dims(np.squeeze(out_scores),axis=0)
        out_labels = np.expand_dims(np.squeeze(out_labels),axis=0)
        # print('boxes:', out_boxes.shape)
        # print('scores:', out_scores.shape)
        # print('labels',out_labels.shape)
        boxes = out_boxes.copy()
        scores = out_scores.copy()
        labels = out_labels.copy()

        # correct boxes for image scale
        boxes /= scale

        # select indices which have a score above the threshold
        indices = np.where(scores[0, :] > -1)[0]
        # print('indices', indices)
        # print(type(scores))
        if type(scores) is not np.ndarray:
            scores = scores.numpy()
            boxes = boxes.numpy()
            labels = labels.numpy()
            masks = masks.numpy()
            attention_map = attention_map.numpy()
        # select those scores
        scores = scores[0][indices]

        # find the order with which to sort the scores
        scores_sort = np.argsort(-scores)[:max_detections]

        image_boxes = boxes[0, indices[scores_sort], :]
        # print('seletec_boxes',image_boxes.shape)
        # print(image_boxes)

        # filter out of lung
        if args.lung_filter:
            client_paths = ['private_1', 'private_2', 'private_3']
            # client_paths = ['private_4/B']
            lung_filter_path = '/research/dept8/qdou/data/covid/{}/lung_seg_png/'.format(client_paths[client_idx])
            # lungfilter = '/covid/private_2/lung_seg_png/
            # print('---img path---')
            img_path = generator.image_path(i)
            patient = img_path.split('/')[-2]
            slice_idx = img_path.split('/')[-1].replace('slice_', '').replace('.h5', '')
            # print('patient:', patient)
            # print('slice:', slice_idx)
            seg_path = os.path.join(lung_filter_path,'{}_slice_{}.png').format(patient,slice_idx)
            # print(seg_path)
            seg = cv2.imread(seg_path)
            scores_sort = _seg_filter(image_boxes,scores_sort,seg)
            image_boxes = boxes[0, indices[scores_sort], :]

        image_scores     = scores[scores_sort]
        image_labels     = labels[0, indices[scores_sort]]
        image_detections = np.concatenate([image_boxes, np.expand_dims(image_scores, axis=1), np.expand_dims(image_labels, axis=1)], axis=1)

        # copy detections to all_detections
        for label in range(generator.num_classes()):
            if not generator.has_label(label):
                continue

            all_detections[i][label] = image_detections[image_detections[:, -1] == label, :-1]

        if args.save_result == 1:
            img_path = generator.image_path(i)
            img_path = img_path.replace('h5_normalize', 'h5')
            # print(img_path)
            with h5py.File(img_path, "r") as hf:
                h5_raw_image = hf['arr'][:]

            draw_annotations(h5_raw_image, generator.load_annotations(i), label_to_name=generator.label_to_name)
            # draw_detections(raw_image, image_boxes, image_scores, image_labels, score_threshold=args.score_threshold, label_to_name=generator.label_to_name)
            draw_detections(h5_raw_image, image_boxes, image_scores, image_labels, slice_id=i, bbox_writer=result_writer, score_threshold=args.score_threshold)
            # if args.lung_filter:
            # slice_idx = generator.image_path(i).split('/')[-1].replace('slice', '').replace('.png', '')
            # cv2.imwrite('../COVID/slice_{}.png'.format(slice_idx),raw_image)

                # print("Shape of load Image")
                # print(arr.shape)

            detection_out[i, :, :] = h5_raw_image

            attention_map[np.where(attention_map < args.attention_threshold)] = 0
            # attention_out[i, :, :] = cv2.flip( cv2.resize(np.squeeze(np.uint8(attention_map * 255)), (origin_shape[1], origin_shape[0])), 0)
            attention_out[i, :, :] = cv2.resize(np.squeeze(np.uint8(attention_map * 255)), (512, 512))

            masks[masks < args.segmentation_threshold] = 0
            masks = cv2.resize(np.squeeze(np.uint8(masks * 255)), (512, 512))

            mask_out[i, :, :] = masks

    if save_path is not None and args.save_result == 1:
        print('Writing Results...')
        # detection_out = sitk.GetImageFromArray(detection_out)
        # sitk.WriteImage(detection_out, os.path.join(save_path, '{}_{}_detection_result.nii.gz'.format(client_name, patient_name)))

        # attention_out = sitk.GetImageFromArray(attention_out)
        # sitk.WriteImage(attention_out, os.path.join(save_path, '{}_{}_attention_result.nii.gz'.format(client_name, patient_name)))

        mask_out = sitk.GetImageFromArray(mask_out)
        sitk.WriteImage(mask_out, os.path.join(save_path, '{}_{}_masks_result.nii.gz'.format(client_name, patient_name)))

    np.save(os.path.join(save_path, '{}_{}_prediction.npy'.format(client_name, patient_name)), all_detections)
    all_annotations, all_annotations_img_path = _get_annotations_and_img_path(generator)
    np.save(os.path.join(save_path, '{}_{}_annotations.npy'.format(client_name, patient_name)), all_annotations)
    np.save(os.path.join(save_path, '{}_{}_annotations_img_path.npy'.format(client_name, patient_name)), all_annotations_img_path)

    return 0

def parse_args(args):
    """ Parse the arguments.
    """
    parser     = argparse.ArgumentParser(description='Evaluation script for a RetinaNet network.')
    # subparsers = parser.add_subparsers(help='Arguments for specific dataset types.', dest='dataset_type')
    # subparsers.required = True

    # csv_parser = subparsers.add_parser('csv')
    # csv_parser.add_argument('annotations', help='Path to CSV file containing annotations for evaluation.')
    # csv_parser.add_argument('classes', help='Path to a CSV file containing class label mapping.')

    parser.add_argument('--ensemble', help='Path to RetinaNet model.', default=False, action='store_true')
    parser.add_argument('--model', help='Path to RetinaNet model.', default=None)
    parser.add_argument('--weights', help='only load weights.', default=None)
    parser.add_argument('--nii',              help='path to nii files.')
    parser.add_argument('--convert-model',    help='Convert the model to an inference model (ie. the input is a training model).', action='store_true')
    parser.add_argument('--backbone',         help='The backbone of the model.', default='vgg19')
    parser.add_argument('--gpu',              help='Id of the GPU to use (as reported by nvidia-smi).')
    parser.add_argument('--multi-gpu', help='Number of GPUs to use for parallel processing.', type=int, default=0)
    parser.add_argument('--score-threshold',  help='Threshold on score to filter detections with (defaults to 0.05).', default=0.4, type=float)
    parser.add_argument('--iou-threshold',    help='IoU Threshold to count for a positive detection (defaults to 0.5).', default=0.5, type=float)
    parser.add_argument('--max-detections',   help='Max Detections per image (defaults to 100).', default=100, type=int)
    parser.add_argument('--detection-threshold', help='Threshold used for determining what detections to draw.', default=0.4, type=int)
    parser.add_argument('--segmentation-threshold', help='Threshold used for filter segmentation map.', default=0.1, type=int)
    parser.add_argument('--attention-threshold', help='Threshold used for filter attention map.', default=0.8, type=int)
    parser.add_argument('--save-path',        help='Path for saving images with detections (doesn\'t work for COCO).',default=None)
    parser.add_argument('--get_predicted_bbox',        help='Save predicted bbox to csv.', action='store_true')
    parser.add_argument('--save-result',        help='Save result or not.', type=int, default=0)
    parser.add_argument('--lung-filter',        help='Path for lung seg filter images', default=False, action='store_true')
    parser.add_argument('--draw-colorful', help='draw difficult type of predict with color', default=False, action='store_true')
    parser.add_argument('--reduce-fp', help='reduce fp, must use after completing first evaluation', default=False, action='store_true')
    parser.add_argument('--log',        help='Path for saving log file', default=None)
    parser.add_argument('--image-min-side',   help='Rescale the image so the smallest side is min_side.', type=int, default=512)
    parser.add_argument('--image-max-side',   help='Rescale the image if the largest side is larger than max_side.', type=int, default=512)
    parser.add_argument('--config',           help='Path to a configuration parameters .ini file (only used with --convert-model).')
    parser.add_argument('--weighted-average', help='Compute the mAP using the weighted average of precisions among classes.', action='store_true')
    parser.add_argument('--dataset_type', help='Path to CSV file containing annotations for evaluation.', default='csv')
    parser.add_argument('--annotations', help='Path to CSV file containing annotations for evaluation.')
    parser.add_argument('--classes', help='Path to a CSV file containing class label mapping.', default='mapping.csv')

    return parser.parse_args(args)

def main(args=None):
    # parse arguments
    if args is None:
        args = sys.argv[1:]
    args = parse_args(args)

    # make sure keras is the minimum required version
    check_keras_version()

    # optionally choose specific GPU
    if args.gpu:
        os.environ['CUDA_VISIBLE_DEVICES'] = args.gpu
    # keras.backend.tensorflow_backend.set_session(get_session())

    # make save path if it doesn't exist
    if args.save_path is not None and not os.path.exists(args.save_path):
        os.makedirs(args.save_path)

    # optionally load config parameters
    if args.config:
        args.config = read_config_file(args.config)

    anno_base_dir = '/research/dept8/qdou/data/covid/'
    args.classes = os.path.join(anno_base_dir, args.classes)

    # create the generator

    # print model summary
    # print(model.summary())

    client_name = ['Dataset1', 'Dataset2', 'Dataset3']
    data_path = ['private_1/h5_normalize', 'private_2/h5_normalize', 'private_3/h5_normalize']
    # data_path = ['private_1/h5_normalize_-1050_800', 'private_2/h5_normalize_-1050_800','private_3/h5_normalize_-1050_800']
    # data_path = ['private_4/B/h5_normalize']
    private_1 = ['P5_annotations_h5_whole_vol.csv']
    private_2 = ['case1_annotations_h5_whole_vol.csv', 'case4_annotations_h5_whole_vol.csv']
    private_3 = ['case19_annotations_h5_whole_vol.csv', 'case23_annotations_h5_whole_vol.csv',
                 'case40_annotations_h5_whole_vol.csv', 'case42_annotations_h5_whole_vol.csv',
                 'case46_annotations_h5_whole_vol.csv', 'case49_annotations_h5_whole_vol.csv',
                 'case51_annotations_h5_whole_vol.csv', 'case54_annotations_h5_whole_vol.csv',
                 'case58_annotations_h5_whole_vol.csv', 'case60_annotations_h5_whole_vol.csv',
                 'case61_annotations_h5_whole_vol.csv', 'case62_annotations_h5_whole_vol.csv']
    private_4 = ['001_annotations_h5_whole_vol.csv', '005_annotations_h5_whole_vol.csv',
                 '006_annotations_h5_whole_vol.csv', '008_annotations_h5_whole_vol.csv',
                 '009_annotations_h5_whole_vol.csv', '010_annotations_h5_whole_vol.csv',
                 '011_annotations_h5_whole_vol.csv', '012_annotations_h5_whole_vol.csv',
                 '013_annotations_h5_whole_vol.csv', '014_annotations_h5_whole_vol.csv']
    # test_data_list = ['test_private_1_all.csv', 'test_mos_all.csv']
    test_data_list = [private_1, private_2, private_3]

    assert len(client_name) == len(data_path) == len(test_data_list)

    # generate patient name based on csv
    patient_names = {}
    for i in range(len(client_name)):
        for j in range(len(test_data_list[i])):
            if client_name[i] not in patient_names:
                patient_names[client_name[i]] = []
                patient_names[client_name[i]].append(test_data_list[i][j].split('_')[0])
            else:
                patient_names[client_name[i]].append(test_data_list[i][j].split('_')[0])

    # start evaluation
    log_path = args.log if args.log else './evaluate_ensemble_internal_patient_wise_3_June.txt'
    logfile = open(log_path,'a')

    # save prediction to npy
    if args.get_predicted_bbox == 1:
        logfile.write('*********************************\n')
        logfile.write('Save prediction of ensemble model to .npy file\n'.format(args.model))

        backbone = models.backbone(args.backbone)

        # optionally load anchor parameters
        anchor_params = None
        if args.config and 'anchor_parameters' in args.config:
            anchor_params = parse_anchor_parameters(args.config)

        # load model
        if args.ensemble:
            model_list = []
            # model_path_list = ['/research/dept8/qdou/mrjiang/Impriving_retina/final_separate_model/private1/vgg19_nl_csv_15.h5',
            model_path_list = ['/research/dept8/qdou/mrjiang/Impriving_retina/federated_results/21_May_siteB_test_fold2/vgg19_nl_csv_15.h5',
                               '/research/dept8/qdou/mrjiang/Impriving_retina/final_separate_model/private2/vgg19_nl_csv_07.h5',
                               '/research/dept8/qdou/mrjiang/Impriving_retina/final_separate_model/private3/vgg19_nl_csv_17.h5']

            for model_path in model_path_list:
                print('Loading {}...'.format(model_path.split('/')[-2]+model_path.split('/')[-1]),flush=True)
                model = models.load_model(model_path, backbone_name=args.backbone)
                if args.convert_model:
                    model = models.convert_model(model, anchor_params=anchor_params)
                model_list.append(model)
        elif args.model is not None:
            print('Loading model, this may take a second...',flush=True)
            model = models.load_model(args.model, backbone_name=args.backbone)
            # optionally convert the model
            if args.convert_model:
                model = models.convert_model(model, anchor_params=anchor_params)
        elif args.weights is not None:
            weights = args.weights

            print('Creating model and Loading weights, this may take a second...',flush=True)
            model, training_model, prediction_model = create_models(
                backbone_retinanet=backbone.retinanet,
                # note : when mapping.csv only contains lesion,0,  generator.num_classes() ==1
                num_classes=1,
                weights=weights,
                multi_gpu=args.multi_gpu,
                freeze_backbone=False,
                config=args.config,
                model_config={}
            )
            # optionally convert the model
            if args.convert_model:
                model = models.convert_model(model, anchor_params=anchor_params)
        else:
            raise ValueError("You have to specify a model")

        # create generator
        # generators = []
        generators = {}

        for i in range(len(client_name)):
            for j in range(len(test_data_list[i])):
                args.annotations = os.path.join(anno_base_dir, data_path[i], test_data_list[i][j])
                print('---client {}---'.format(client_name[i]))
                print('validation csv {}'.format(args.annotations))
                generator = create_generator(args)
                if client_name[i] not in generators:
                    generators[client_name[i]] = []
                    generators[client_name[i]].append(generator)

                else:
                    generators[client_name[i]].append(generator)

        if args.lung_filter:
            print('do lung filter',flush=True)
            logfile.write('do lung filter\n')
        else:
            print('no lung filter',flush=True)

        for i in range(len(generators)):
            print('------client {}-----'.format(client_name[i]))
            for j in range(len(generators[client_name[i]])):
                logfile.write('Writing client {} patient {} prediction results to .npy... \n'.format(client_name[i], patient_names[client_name[i]]))
                print('------patient {}-----'.format(patient_names[client_name[i]][j]))
                generator = generators[client_name[i]][j]
                patient_name = patient_names[client_name[i]][j]
                if args.ensemble:
                    _print_ensemble_detections_to_npy(
                        args,
                        generator,
                        model_list,
                        client_idx=i,
                        client_name=client_name[i],
                        patient_name=patient_name,
                        score_threshold=args.score_threshold,
                        max_detections=args.max_detections,
                        save_path=args.save_path,
                    )
                else:
                    raise ValueError("This is ensemble code")
        logfile.write('Finish writing \n')
        logfile.write('*********************************')
        sys.exit(0)

    # evaluate from npy
    logfile.write('*********************************\n')
    logfile.write('*{}*\n'.format(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())))
    logfile.write('Evaluate ensemble model from .npy\n'.format(args.model))
    logfile.write('thresshold:{}\n'.format(args.score_threshold))

    for i in range(len(client_name)):
        print('client {}'.format(client_name[i]))
        client_tps, client_fps, client_num_annotations, client_num_slices = 0., 0., 0., 0.
        client_precision, client_recall, client_fp_slice = [], [], []
        client_mAP = []
        for j in range(len(test_data_list[i])):
            patient_name = patient_names[client_name[i]][j]
            average_precisions, old, new, num_annotations, num_slices = evaluate_from_npy(
                args,
                client_name=client_name[i],
                patient_name=patient_name,
                iou_threshold=args.iou_threshold,
                score_threshold=args.score_threshold,
                max_detections=args.max_detections,
                save_path=args.save_path,
            )
            patient_tp = new[0]
            patient_fp = new[1]
            client_recall.append(new[2])
            client_precision.append(new[3])
            client_fp_slice.append(new[4])

            client_tps += patient_tp
            client_fps += patient_fp
            client_num_annotations += num_annotations
            client_num_slices += num_slices

            if args.draw_colorful:
                draw_colorful_result(args,
                                     client_name=client_name[i],
                                     patient_name=patient_name,
                                     iou_threshold=args.iou_threshold,
                                     score_threshold=args.score_threshold,
                                     max_detections=args.max_detections,
                                     save_path=args.save_path)

            # print evaluation
            total_instances = []
            precisions = []
            for label, (average_precision, num_annotations) in average_precisions.items():
                # print('    {:.0f} instances of class'.format(num_annotations),
                #       'lesion', 'with average precision: {:.4f}'.format(average_precision))
                total_instances.append(num_annotations)
                precisions.append(average_precision)

            if sum(total_instances) == 0:
                print('No test instances found.')
                return

            if args.weighted_average:
                print('    mAP: {:.4f}'.format(
                    sum([a * b for a, b in zip(total_instances, precisions)]) / sum(total_instances)))
            else:
                print('    mAP: {:.4f}'.format(sum(precisions) / sum(x > 0 for x in total_instances)))

            mAP = sum(precisions) / sum(x > 0 for x in total_instances)
            client_mAP.append(mAP)
            logfile.write('client:{} patient{}\n'.format(client_name[i], patient_name))
            logfile.write('  TP:{} FP:{}\n'.format(patient_tp, patient_fp))
            logfile.write('  FP/slice:{} Sensitivity:{}  Precision:{}\n'.format(patient_fp / num_slices,
                                                                                patient_tp / num_annotations,
                                                                                patient_tp / (patient_tp + patient_fp)))
            logfile.write('  mAP:{}\n'.format(mAP))

        client_total_recall = client_tps / client_num_annotations
        client_total_precision = client_tps / (client_tps + client_fps)
        clint_total_fp_slice = client_fps / client_num_slices

        client_avg_recall = sum(client_recall) / len(test_data_list[i])
        client_avg_precision = sum(client_precision) / len(test_data_list[i])
        client_avg_fp_slice = sum(client_fp_slice) / len(test_data_list[i])
        avg_mAP = sum(client_mAP) / len(test_data_list[i])

        # calc confidence interval
        client_mean_precision = np.mean(np.asanyarray(client_precision))
        client_std_precision = np.std(np.asanyarray(client_precision))
        client_se_precision = client_std_precision / np.sqrt(len(test_data_list))
        client_p_value_precision = 1.96 * client_se_precision

        client_mean_recall = np.mean(np.asanyarray(client_recall))
        client_p_value_recall = 1.96 * (np.std(np.asanyarray(client_recall)) / np.sqrt(len((test_data_list))))

        client_mean_map = np.mean(np.asanyarray(client_mAP))
        client_p_value_map = 1.96 * (np.std(np.asanyarray(client_mAP)) / np.sqrt(len((test_data_list))))

        print('------{}------'.format(client_name[i]))
        print('   total:')
        print('   # TP:{}   FP{}'.format(client_tps, client_fps))
        print('   # FP/slice:{:.4f} Sensitivity:{:.5f}  Precision:{:.5f}'.format(clint_total_fp_slice,
                                                                                client_total_recall,
                                                                                client_total_precision))
        print('   average:')
        print('   # TP:{}   FP{}'.format(client_tps, client_fps))
        print('   # FP/slice:{:.4f} Sensitivity:{:.5f}  Precision:{:.5f}'.format(client_avg_fp_slice, client_avg_recall,
                                                                                client_avg_precision))
        print('   # mAP:{:.5f}'.format(avg_mAP))
        print('   # Sensitivity:[{:.4f}+-{:.4f}]   Precision:[{:.4f}+-{:.4f}] mAP:[{:.4f}+-{:.4f}]'.format(
            client_mean_recall, client_p_value_recall, client_mean_precision, client_p_value_precision, client_mean_map,
            client_p_value_map))

        logfile.write('  total over client:{}\n'.format(client_name[i]))
        logfile.write('  TP:{} FP:{}\n'.format(client_tps, client_fps))
        logfile.write('  FP/slice:{} Sensitivity:{}  Precision:{}\n'.format(clint_total_fp_slice, client_total_recall,
                                                                            client_total_precision))
        logfile.write('  Average over client:{}\n'.format(client_name[i]))
        logfile.write('  TP:{} FP:{}\n'.format(client_tps, client_fps))
        logfile.write('  FP/slice:{} Sensitivity:{}  Precision:{}\n'.format(client_avg_fp_slice, client_avg_recall,
                                                                            client_avg_precision))
        logfile.write('  mAP:{}\n'.format(avg_mAP))
        logfile.write('*********************************')

    logfile.flush()
    logfile.close()

if __name__ == '__main__':
    main()


import math
import torch.nn.functional as F
import collections
import torch
import numpy
from collections import defaultdict
from dataclasses import dataclass
from fairseq import metrics
#
# from fairseq.sequence_generator  import SequenceGenerator
# from sequence_generator import
from fairseq import utils #, bleu
from fairseq.dataclass import FairseqDataclass
from fairseq.criterions import FairseqCriterion, register_criterion
from omegaconf import II
from fairseq.models import FairseqMultiModel
from fairseq.sequence_generator import SequenceGenerator

@register_criterion('v2')
class V2Criterion(FairseqCriterion):

    def __init__(self, task, sentence_avg): #(self, args, task):
        super().__init__(task)
        self.cnt=0

    # def forward(self, model, sample, reduce=True):
    #     """Compute the loss for the given sample.
    #
    #     Returns a tuple with three elements:
    #     1) the loss
    #     2) the sample size, which is used as the denominator for the gradient
    #     3) logging outputs to display while training
    #     """
    #     net_output = model(**sample['net_input'])
    #     lprobs = model.get_normalized_probs(net_output, log_probs=True)
    #     lprobs = lprobs.view(-1, lprobs.size(-1))
    #     target = model.get_targets(sample, net_output).view(-1)
    #     loss = F.nll_loss(lprobs, target, size_average=False,
    #                       ignore_index=self.padding_idx,
    #                       reduce=reduce)
    #     sample_size = sample['target'].size(0) if self.args.sentence_avg else sample['ntokens']
    #     logging_output = {
    #         'loss': utils.item(loss.data) if reduce else loss.data,
    #         'ntokens': sample['ntokens'],
    #         'sample_size': sample_size,
    #     }
    #     return loss, sample_size, logging_output

    def forward(self, model, sample, reduce=True):
        # sample mode
        #print('!!!RL loss.')
        model.eval()
        # src_dict = self.task.source_dictionary
        tgt_dict = self.task.target_dictionary
        eos_idx = self.task.target_dictionary.eos()
        sample_beam = 4 #self.args.sample_beam

        translator = SequenceGenerator([model], tgt_dict=tgt_dict,beam_size=sample_beam)

        translator.cuda()
        ct = 0
        translations = []

        s = utils.move_to_cuda(sample)
        input = s['net_input']
        max_len = 200
        with torch.no_grad():
            hypos = translator.generate([model],s)
        for i, id in enumerate(s['id'].data):
            src = input['src_tokens'].data[i, :]
            # remove padding from ref
            ref = utils.strip_pad(s['target'].data[i, :], tgt_dict.pad()) if s['target'] is not None else None
            translations.append((id, src, ref, hypos[i]))

            ct += 1
        # print("sample batch size:", ct)

        model.train()

        # MLE loss
        mle_net_output = model(**sample['net_input'])
        mle_lprobs = model.get_normalized_probs(mle_net_output, log_probs=True)
        mle_lprobs = mle_lprobs.view(-1, mle_lprobs.size(-1))
        mle_target = model.get_targets(sample, mle_net_output).view(-1)
        mle_loss = F.nll_loss(mle_lprobs, mle_target, size_average=False,
                              ignore_index=self.padding_idx, reduce=reduce)
        mle_tokens = sample['ntokens']
        avg_mle_loss = mle_loss / mle_tokens
        self.cnt  += 1
        if self.cnt % 4 !=0 :
            total_loss = avg_mle_loss
            total_tokens =  sample["ntokens"]
            logging_output = {
                'loss': utils.item(total_loss.data),
                'ntokens': total_tokens,
                'sample_size': total_tokens,
            }
    #         print('total: ',total_loss)
            return total_loss, total_tokens, logging_output

        print('avg_mle_loss:', avg_mle_loss)
        # RL loss
        batch_rl_loss = 0
        batch_tokens = 0
        sample_ind = 0
        for sample_id, src_tokens, tgt_tokens, hypos in translations:
            # calculate bleu
            sample_ind += 1
            rewards = torch.Tensor(sample_beam).float().cuda()
            logprobs = torch.Tensor(sample_beam).float().cuda()
            for i in range(sample_beam):
                hypo = hypos[i]
                trans_tokens = hypo['tokens']
                rewards[i] = self.compute_gleu(tgt_tokens.cpu(), trans_tokens.cpu(), max_order=4, gram=0).cuda()
                # one_sample loss calculation
                tgt_input_tokens = trans_tokens.new(trans_tokens.shape).fill_(0)
                assert trans_tokens[-1] == eos_idx
                tgt_input_tokens[0] = eos_idx
                tgt_input_tokens[1:] = trans_tokens[:-1]
                train_sample = {
                    'net_input': {
                        'src_tokens': src_tokens.view(1, -1),
                        'src_lengths': torch.LongTensor(src_tokens.numel()).view(1, -1),
                        'prev_output_tokens': tgt_input_tokens.view(1, -1),
                    },
                    'target': trans_tokens.view(1, -1)
                }
                train_sample = utils.move_to_cuda(train_sample)
                net_output = model(**train_sample['net_input'])
                lprobs = model.get_normalized_probs(net_output, log_probs=True)
                lprobs = lprobs.view(-1, lprobs.size(-1))
                target = model.get_targets(train_sample, net_output).view(-1, 1)
                non_pad_mask = target.ne(tgt_dict.pad())
                lprob = -lprobs.gather(dim=-1, index=target)[non_pad_mask]
                logprobs[i] = torch.sum(lprob)
                ntokens = len(train_sample['target'])
                batch_tokens += ntokens
            rl_loss = torch.sum(logprobs * (rewards - rewards.mean()))  # one sample loss
            batch_rl_loss += rl_loss

        avg_rl_loss = batch_rl_loss / batch_tokens
        print('avg_rl_loss:', avg_rl_loss)
        if True:
            total_loss = avg_mle_loss +  avg_rl_loss
            total_tokens = batch_tokens + mle_tokens
        else:
            total_loss = avg_rl_loss
            total_tokens = batch_tokens
        logging_output = {
            'loss': utils.item(total_loss.data),
            'ntokens': total_tokens,
            'sample_size': total_tokens,
        }
        print('total: ',total_loss)
        return total_loss, total_tokens, logging_output

    def _get_ngrams(self, segment, max_order):
        ngram_counts = collections.Counter()
        for order in range(1, max_order + 1):
            for i in range(0, len(segment) - order + 1):
                ngram = tuple(segment[i:i + order])
                ngram_counts[ngram] += 1
        return ngram_counts

    def compute_gleu(self, reference_corpus, translation_corpus, max_order=4, gram=0, smooth=False):
        scores = torch.zeros(max_order)
        reference_array = numpy.array(reference_corpus)
        translation_array = numpy.array(translation_corpus)
        matches_by_order = [0] * max_order
        possible_matches_by_order_ref = [0] * max_order
        possible_matches_by_order_trans = [0] * max_order
        reference_length = 0
        translation_length = 0
        reference_length += reference_array.shape[0]
        translation_length += translation_array.shape[0]
        merged_ref_ngram_counts = collections.Counter()
        merged_ref_ngram_counts |= self._get_ngrams(reference_array, max_order)
        translation_ngram_counts = self._get_ngrams(translation_array, max_order)
        overlap = translation_ngram_counts & merged_ref_ngram_counts
        for ngram in overlap:
            matches_by_order[len(ngram)-1] += overlap[ngram]
        for order in range(1, max_order+1):
            possible_matches_trans = translation_length - order + 1
            if possible_matches_trans > 0:
                possible_matches_by_order_trans[order-1] += possible_matches_trans
            possible_matches_ref = reference_length - order + 1
            if possible_matches_ref > 0:
                possible_matches_by_order_ref[order-1] += possible_matches_ref
        precisions = [0] * max_order
        recalls = [0] * max_order

        for i in range(0, max_order):
            if smooth:
                precisions[i] = ((matches_by_order[i] + 1.) / (possible_matches_by_order_trans[i] + 1.))
                recalls[i] = ((matches_by_order[i] + 1.) / (possible_matches_by_order_ref[i] + 1.))
            else:
                if possible_matches_by_order_trans[i] > 0:
                    precisions[i] = (float(matches_by_order[i]) / possible_matches_by_order_trans[i])
                else:
                    precisions[i] = 0.0

                if possible_matches_by_order_ref[i] > 0:
                    recalls[i] = (float(matches_by_order[i]) / possible_matches_by_order_ref[i])
                else:
                    recalls[i] = 0.0
        for i in range(max_order):
            scores[i] = min(precisions[i],recalls[i])

        if False :#self.args.modgleu:
            if reference_length < max_order and translation_length < max_order:
                order = max(reference_length, translation_length)
                scores = scores[0:order]
            else:
                order = max_order
        else:
            order = max_order
        if gram == 0:
            if min(scores) > 0:
                log_scores = torch.log(scores)
                p_log_sum = torch.sum((1. / order) * log_scores)
                geo_mean = torch.exp(p_log_sum)
                return geo_mean
            else:
                return torch.tensor(0.0)
        else:
            if scores[gram] > 0:
                return scores[gram]
            else:
                return torch.tensor(0.0)

    @staticmethod
    def aggregate_logging_outputs(logging_outputs):
        """Aggregate logging outputs from data parallel training."""
        loss_sum = sum(log.get('loss', 0) for log in logging_outputs)
        ntokens = sum(log.get('ntokens', 0) for log in logging_outputs)
        sample_size = sum(log.get('sample_size', 0) for log in logging_outputs)
        agg_output = {
            'loss': loss_sum / sample_size / math.log(2),
            'sample_size': sample_size,
        }
        if sample_size != ntokens:
            agg_output['nll_loss'] = loss_sum / ntokens / math.log(2)
        return agg_output

    @staticmethod
    def reduce_metrics(logging_outputs) -> None:
        """Aggregate logging outputs from data parallel training."""
        loss_sum = sum(log.get("loss", 0) for log in logging_outputs)
        ntokens = sum(log.get("ntokens", 0) for log in logging_outputs)
        sample_size = sum(log.get("sample_size", 0) for log in logging_outputs)

        # we divide by log(2) to convert the loss from base e to base 2
        metrics.log_scalar(
            "loss", loss_sum / sample_size / math.log(2), sample_size, round=3
        )
        if sample_size != ntokens:
            metrics.log_scalar(
                "nll_loss", loss_sum / ntokens / math.log(2), ntokens, round=3
            )
            metrics.log_derived(
                "ppl", lambda meters: utils.get_perplexity(meters["nll_loss"].avg)
            )
        else:
            metrics.log_derived(
                "ppl", lambda meters: utils.get_perplexity(meters["loss"].avg)
            )

    @staticmethod
    def logging_outputs_can_be_summed() -> bool:
        """
        Whether the logging outputs returned by `forward` can be summed
        across workers prior to calling `reduce_metrics`. Setting this
        to True will improves distributed training speed.
        """
        return True


"""
.. _tut_intro_python:

Introduction to Python
======================

`Python <https://www.python.org/>`_ is a modern general-purpose object-oriented
high-level programming language. First make sure you have a working Python
environment and dependencies (see :ref:`install_python_and_mne_python`). If you
are completely new to Python, don't worry, it's just like any other programming
language, only easier. Here are a few great resources to get you started:

* `SciPy lectures <http://scipy-lectures.github.io>`_
* `Learn X in Y minutes: Python <https://learnxinyminutes.com/docs/python/>`_
* `NumPy for MATLAB users <https://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html>`_

We highly recommend watching the SciPy videos and reading through these sites
to get a sense of how scientific computing is done in Python.
"""    # noqa
###############################################################################
# Here are few important points to familiarize yourself with Python. First,
# everything is dynamically typed. There is no need to declare and initialize
# data structures or variables separately.

a = 3
print(type(a))
b = [1, 2.5, 'This is a string']
print(type(b))
c = 'Hello world!'
print(type(c))

###############################################################################
# Second, if you have a MATLAB background remember that indexing in Python
# starts from zero (and is done with square brackets):
a = [1, 2, 3, 4]
print('This is the zeroth value in the list: {}'.format(a[0]))

###############################################################################
# Finally, often there is no need to reinvent the wheel. SciPy and NumPy are
# battle-hardened libraries that offer a vast variety of functions for most
# needs. Consult the documentation and remember that you can always ask the
# IPython interpreter for help with a question mark at the beginning or end of
# a function:
#
#      >>> import numpy as np
#      >>> np.arange?


# Introduction

Classical mechanics is a topic which has been taught intensively over
several centuries. It is, with its many variants and ways of
presenting the educational material, normally the first **real** physics
course many of us meet and it lays the foundation for further physics
studies. Many of the equations and ways of reasoning about the
underlying laws of motion and pertinent forces, shape our approaches and understanding
of the scientific method and discourse, as well as the way we develop our insights
and deeper understanding about physical systems.

There is a wealth of
well-tested (from both a physics point of view and a pedagogical
standpoint) exercises and problems which can be solved
analytically. However, many of these problems represent idealized and
less realistic situations.  The large majority of these problems are
solved by paper and pencil and are traditionally aimed
at what we normally refer to as continuous models from which we may find an analytical solution.  As a consequence,
when teaching mechanics, it implies that we can seldomly venture beyond an idealized case
in order to develop our understandings and insights about the
underlying forces and laws of motion.

On the other hand, numerical algorithms call for approximate discrete
models and much of the development of methods for continuous models
are nowadays being replaced by methods for discrete models in science and
industry, simply because **much larger classes of problems can be addressed** with discrete models, often  by simpler and more
generic methodologies.

As we will see below, when properly scaling the equations at hand,
discrete models open up for more advanced abstractions and the possibility to
study  real life systems, with the added bonus that we can explore and
deepen our basic understanding of various physical systems

Analytical solutions are as important as before. In addition, such
solutions provide us with invaluable benchmarks and tests for our
discrete models. Such benchmarks, as we will see below, allow us
to discuss possible sources of errors and their behaviors.  And
finally, since most of our models are based on various algorithms from
numerical mathematics, we have a unique oppotunity to gain a deeper
understanding of the mathematical approaches we are using.

With computing and data science as important elements in essentially
all aspects of a modern society, we could  then try to define Computing as
**solving scientific problems using all possible tools, including
symbolic computing, computers and numerical algorithms, and analytical
paper and pencil solutions**.
Computing provides us with the tools to develope our own understanding of the scientific method by enhancing algorithmic thinking.

The way we will teach this course reflects
this definition of computing. The course contains both classical paper
and pencil exercises as well as  computational projects and exercises. The
hope is that this will allow you to explore the physics of systems
governed by the degrees of freedom of classical mechanics at a deeper
level, and that these insights about the scientific method will help
you to develop a better understanding of how the underlying forces and
equations of motion and how they impact a given system. Furthermore, by introducing various numerical methods
via computational projects and exercises, we aim at developing your competences and skills about these topics.

These competences will enable you to

* understand how algorithms are used to solve mathematical problems,

* derive, verify, and implement algorithms,

* understand what can go wrong with algorithms,

* use these algorithms to construct reproducible scientific outcomes and to engage in science in ethical ways, and

* think algorithmically for the purposes of gaining deeper insights about scientific problems.

All these elements are central for maturing and gaining a better understanding of the modern scientific process *per se*.

The power of the scientific method lies in identifying a given problem
as a special case of an abstract class of problems, identifying
general solution methods for this class of problems, and applying a
general method to the specific problem (applying means, in the case of
computing, calculations by pen and paper, symbolic computing, or
numerical computing by ready-made and/or self-written software). This
generic view on problems and methods is particularly important for
understanding how to apply available, generic software to solve a
particular problem.

*However, verification of algorithms and understanding their limitations requires much of the classical knowledge about continuous models.*

## A well-known examples to illustrate many of the above concepts

Before we venture into a reminder on Python and mechanics relevant applications, let us briefly outline some of the
abovementioned topics using an example many of you may have seen before in for example CMSE201.
A simple algorithm for integration is the Trapezoidal rule.
Integration of a function $f(x)$ by the Trapezoidal Rule is given by following algorithm for an interval $x \in [a,b]$

$$
\int_a^b(f(x) dx = \frac{1}{2}\left [f(a)+2f(a+h)+\dots+2f(b-h)+f(b)\right] +O(h^2),
$$

where $h$ is the so-called stepsize defined by the number of integration points $N$ as $h=(b-a)/(n)$.
Python offers an  extremely versatile programming  environment, allowing for
the inclusion of analytical studies in a numerical program. Here we show an
example code with the **trapezoidal rule**. We use also **SymPy** to evaluate the exact value of the integral and compute the absolute error
with respect to the numerically evaluated one of the integral
$\int_0^1 dx x^2 = 1/3$.
The following code for  the trapezoidal rule allows you  to plot the relative error by comparing with the exact result. By increasing to $10^8$ points one arrives at a region where numerical errors start to accumulate.

%matplotlib inline

from math import log10
import numpy as np
from sympy import Symbol, integrate
import matplotlib.pyplot as plt
# function for the trapezoidal rule
def Trapez(a,b,f,n):
   h = (b-a)/float(n)
   s = 0
   x = a
   for i in range(1,n,1):
       x = x+h
       s = s+ f(x)
   s = 0.5*(f(a)+f(b)) +s
   return h*s
#  function to compute pi
def function(x):
    return x*x
# define integration limits
a = 0.0;  b = 1.0;
# find result from sympy
# define x as a symbol to be used by sympy
x = Symbol('x')
exact = integrate(function(x), (x, a, b))
# set up the arrays for plotting the relative error
n = np.zeros(9); y = np.zeros(9);
# find the relative error as function of integration points
for i in range(1, 8, 1):
    npts = 10**i
    result = Trapez(a,b,function,npts)
    RelativeError = abs((exact-result)/exact)
    n[i] = log10(npts); y[i] = log10(RelativeError);
plt.plot(n,y, 'ro')
plt.xlabel('n')
plt.ylabel('Relative error')
plt.show()

This example shows the potential of combining numerical algorithms with symbolic calculations, allowing us to

* Validate and verify  their  algorithms.

* Including concepts like unit testing, one has the possibility to test and test several or all parts of the code.

* Validation and verification are then included *naturally* and one can develop a better attitude to what is meant with an ethically sound scientific approach.

* The above example allows the student to also test the mathematical error of the algorithm for the trapezoidal rule by changing the number of integration points. The students get **trained from day one to think error analysis**.

* With a Jupyter notebook you can keep exploring similar examples and turn them in as your own notebooks.

In this process we can easily bake in
1. How to structure a code in terms of functions

2. How to make a module

3. How to read input data flexibly from the command line

4. How to create graphical/web user interfaces

5. How to write unit tests (test functions or doctests)

6. How to refactor code in terms of classes (instead of functions only)

7. How to conduct and automate large-scale numerical experiments

8. How to write scientific reports in various formats (LaTeX, HTML)

The conventions and techniques outlined here will save you a lot of time when you incrementally extend software over time from simpler to more complicated problems. In particular, you will benefit from many good habits:
1. New code is added in a modular fashion to a library (modules)

2. Programs are run through convenient user interfaces

3. It takes one quick command to let all your code undergo heavy testing

4. Tedious manual work with running programs is automated,

5. Your scientific investigations are reproducible, scientific reports with top quality typesetting are produced both for paper and electronic devices.