# Introduction

In my senior year at Miniemeninstituut, to be able to graduate, we had to set up our own company the same way we should in real life. With this "company", we had the opportunity to do an internship in different kind of companies and enterprises. 

Out of 45 students, only **six** could go on international internships; three people could go to The Netherlands and another three people could go to Tenerife (Canary Islands, Spain). 

After some hard work and a lot of presentations and interviews, our "company" called **CodeX Ltd.** was selected for the international internship at Conecta Software in Tenerife, Spain.

# Research assignment

Conecta Software was developing smart cities (in Spain). They were in collaboration with another Spanish company called **Hopu** that is creating Smart POints of Interest (POI). These Smart POIs are interactive areas (1-80 metres) where users can interact via their smartphones thanks to a device known as a Smart Spot that sends “push” notifications with digital content through Bluetooth and Wi-Fi without the need to install native Apps.
 
The data from each individual SmartSport can be monitored in real-time on a dashboard.

# Applications
_Some real life applicatons for SmartSpots:_ 
* **Crowd Monitoring**: To track where the most crowds are based on WiFi signals
* **Air Quality** monitoring (NO2, CO2, O3, ...)
* **Noise pollution** monitoring (30dB-130dB)
* Measure **energy consuption**

Three week internship at Conecta Software & Erasmus in Tenerife, Spain in connection with IoT & smart devices

Conecta Software

This is a portfolio of the activities I did in the three years I studied at PXL University College of Applied Sciences and Arts.

I-Talent Portfolio

# Introduction & Challenge

International variation of the PXL Digital Research Project, in collaboration with Fontys Hogeschool in the Netherlands and Belgium Campus University in South Africa.
Together we are working on the OpenRemote Energy Transition Challenge.

Eco-friendly houses that run on green energy are becoming more popular every day. The challenge with green energy is consistency. The weather changes often and the locations where these differences are adapted are vastly different. As a result, energy production occurs every day, in every place differently. There is therefore a need for a solution to view the energy production and use of off-grid (not connected to the electricity grid) houses (eg through IoT devices).

Our job is to work with homes and/or properties equipped with off-grid (not connected to the powergrid) infrastructure such as solar panels, wind turbines, energy storage batteries and IoT devices that monitor the system for power production, power consumption, etc. The system needs power and an internet connection.

The ultimate goal is to ensure that energy consumption is predicted by a machine learning model. And based on that, decide how the house or building should receive power (via battery or directly via green energy).
The purpose of using green energy is to move away from traditional fossil fuels that cause pollution and deplete our Earth's resources.
The use of green energy is important for two reasons: first, it does not cause pollution or damage to the environment and second, it is an unlimited resource.

# Research Team

## [Jakobus Badenhorst](https://www.linkedin.com/in/jj-badenhorst/) - Belgium Campus
_Front-end_ 
Design, setup and development of ReactJS front-end + API development

## [Stephanus de Beer]() - Belgium Campus
_Front-end_ 
Set up + development of ReactJS front-end

## [Wondimu Woldeaarggiye](https://www.linkedin.com/in/wondimu-gebre-woldeaarggiye-a74a63100/) - Fontys
_Back-end, Security_ 
Setting up Java Springboot back-end + security implementation

## [Matthijs Kotterink](https://www.linkedin.com/in/matthijs-kotterink-8a52051a3/) - Fontys
_Data Management, Security_ 
Setting up MongoDB, back-end API + security implementation

## [Anouk Min](https://www.linkedin.com/in/anouk-min-101543178/) - Fontys
_AI & Machine learning_ 
Data exploration + prediction model implementation

## [Simon Wilmots](https://www.linkedin.com/in/simonwilmots/) - PXL 
_AI & Machine learning_ 
Data exploration, data cleaning + prediction model implementation

# Technology
_Een korte beschrijving van de technologie die gebruikt wordt voor de oplossing_

## Development
During the development of the research project, Microsoft's Visual Studio Code and JetBrains' IntelliJ were used. These IDEs were specifically chosen because many programming languages ​​are supported and can be used for front-end, back-end and machine learning.
Github is used as source control and file management medium. This contains all the code for each part of the project, so that everyone's version is always up-to-date.

## Front end
We decided to use the Javascript library **ReactJS** for the application's front-end or graphical user interface. In this library, the MaterialUI component library is implemented for a more stable and better layout. 

We also implemented our own front-end API.

## Backend
For the back-end or the part that the user does not see (Database, Hosting, APIs, ...), we chose the Java framework **Spring**. A MongoDB is linked to the Java back-end. The data from this database can also be requested by a custom back-end API.

## Machine Learning
The implementation of the machine learning model is mainly done with **Jupyter Notebook**. Jupyter is a free open-source, interactive web tool mainly used to make calculations directly in a notebook. In this notebook, you can combine software code and plain text. This way you have all the documentation in one document.

In these notebooks we have used two libraries in the **Python** programming language, namely: **Pandas** and **FastAI**. The machine learning model will first extract the training data from our API. The data is then cleaned up via Pandas and it is ensured that everything is formatted correctly, so that the model will make the most accurate predictions possible.

## Communication
As a communication medium, we decided to use Discord. In addition, Microsoft Teams and WhatsApp were also used. The Discord server was mainly used between the researchers. Here are several text channels for information exchange, meeting summaries and conversation options.

In Microsoft Teams, each research group had been given its own channel. Our meetings with our coach(es) and OpenRemote were mainly held here. The division of the groups was also done via Teams.
The research project was hardly mentioned in the WhatsApp group. This was intended more as a recreational platform.

# Technical Implementation
_The technical solution with a deeper understanding of machine learning_ 

## Front-end
The front-end consists of a web application made in the Javascript framework ReactJS. The app provides a visual representation of all predictions made in the machine learning model. 

The first API has also been developed in the front-end. Thanks to this API, OpenRemote data is available in every part of the project. The API is set up in an Ubuntu and Nginx docker container on a **VPS(*)** at the cloud computing specialist: **OVHcloud**. The VPS belongs to a member of the group and this person has given full permission to use it during this project. On the nginx, HTTPS is used, i.e. there is secure communication between the sender and receiver of the information.
 
In HTTPS, the communication protocol is encrypted using Transport Layer Security or, previously, Secure Sockets Layer.
The front-end itself also makes calls to a third-party API with weather information to display any possible warnings. 

## Back end
The back-end mainly consists of the Spring framework for the Java platform in combination with a MongoDB database. The core features of the framework can be used by any Java application, but there are extensions for building web backends on top of the Java EE (Enterprise Edition).
 
The back-end also has its own API. With this API, new or previous predictions from the machine learning model can be requested.
 
Secure HTTPS communication has also been used here in combination with API call logging. I.e. that with every API call, information about the requester is noted in a log file. This API also cannot be accessed without authentication. So there are several layers of security in the back-end. This ensures that any intruders cannot get in. This is also known as multifactorial authentication.
 

## Machine Learning - Setup
The machine learning model was developed in **Anaconda's Jupyter Notebook**. This is an open document format based on a JSON file. This makes it very easy to put documentation and code in one file. This notebook can also be found on the OpenRemote Energy Transition Challenge Github. It uses the Python libraries Pandas, pyarrow and some functions of FastAI.
 
Before a model can make predictions, a few things have to happen. The data must be correct and must be cleaned up. This is also known as data cleaning.
 

## Machine learning - Data cleaning
The correct electricity usage data was provided by OpenRemote. But the UV data on which the predictions are based have yet to be retrieved. This is done via the request library in python and the OpenUV API. This API returns UV values ​​at a given date, latitude and longitude.
 

After reading the given data from OpenRemote, we will put in each row of the data the UV value that corresponds to the time of the data.
The columns that are of little or no use will only cause an **overfitting(*)** of the prediction model. Therefore, this and all empty columns are removed from the data. As a final clean-up, the dates in the model are put in the correct time zone. On the right is an example of what the data currently looks like.
 

When the data has been cleaned up, one more step needs to be done before predictions can be made. Namely the pre-processing or pre-processing of the data. 

## Machine learning - Pre-processing
This is an important step in the machine learning process. The phrase **"garbage in, garbage out"** applies especially to machine learning projects. Data collection methods are often loosely checked, resulting in out-of-range values, impossible data combinations and missing values, etc. Analyzing data that has not been carefully screened for such issues can produce misleading results.
 

Often data preprocessing is the most important stage of a machine learning project.

Machine learning models, _and computers in general_, only understand **numerical values** ​​at the end of the day. This means that if data contains categorical elements, they must be coded to numbers before a model can make a prediction. This pre-processing step is called **encoding**. For example; the name of a building, _“Tamar building”_ or _“Humber building”_, is converted to a number (_1_ or _2_). For dates, we don't want an entire date-string (eg _“2021-02-11 04:10:22.926000+01:00”_) to be converted to a number. Therefore, each part of the date is broken down by column. I.e. that the hour, minutes, day, month, and year are put in separate columns.
Now the data is finally ready to use in the prediction model.

 

## Machine learning - Prediction model
We have chosen a “**Random Forest**” model.
As the name suggests, “Random Forest” is an algorithm that contains a number of **decision trees(*)** for different subsets of the given dataset and averages it to improve the predictive accuracy of that dataset.
 
Instead of relying on one decision tree, the random forest takes the prediction from each tree and based on the majority "votes" of the predictions, it predicts the final output.
 
The greater number of trees in the forest leads to higher accuracy and avoids the problem of overfitting.
After implementing, training and testing this model, it is possible to make the most accurate decisions. 

# Conclusion & Reflection
_The ultimate goal is to ensure that energy consumption is predicted on the basis of machine learning. And based on that, decide how the house or building should receive power (via battery or directly via green energy)._ 

The goal that OpenRemote had in mind has certainly been achieved on most points. There is a working, accurate prediction model based on the UV values and the data from OpenRemote. These predictions can be shown on a user-friendly dashboard with comparisons and warnings in the form of graphs and tables. The data is neatly stored in the database and can be retrieved again using the custom APIs. Integration with existing OpenRemote systems was not part of this research project.

 

---

 

_**(*)**_ 
**VPS**: _Een Virtual Private Server, is een virtuele machine die als een service wordt verkocht door een internethostingservice._ 

 

**Overfitting**: _de productie van een analyse die te nauw of precies overeenkomt met een bepaalde reeks gegevens, en daarom mogelijk geen aanvullende gegevens past of toekomstige waarnemingen betrouwbaar voorspelt._

 

**Decision tree**: _is een beslissingsondersteunende tool die gebruik maakt van een boomachtig model van beslissingen en hun mogelijke gevolgen, inclusief toevallige uitkomsten, resourcekosten en het nut._

International variation of the PXL Digital Research Project, in collaboration with Fontys Hogeschool in the Netherlands and Belgium Campus University in South Africa.

Energy Transition Challenge

My bachelor's research internship with the focus on conducting a literature study on existing possibilities within the domain of reinforcement learning and autonomous driving.

Race to the Future

# Unity 3D Engine

# Planets & Star(s)
The planets are made with a sphere 3D object. These have their own material, texture & lighting.

# Orbits
When you press the **O**-key on your keyboard, the orbits of each planet will show. This is made with a linerenderer based on the center (the Sun).

# Velocity
If you use the **scrollwheel**, you can change the velocity at which you travel through the Solar System. The velocity is measured in **C**. 1 C equals the speed of light in a vaccuum (in space).

<img src="/assets/img/project-solar-system.jpg"/>

# Playable Web version
## (Coming Soon)

A selfmade 1:1 simulation of our Solar System using the Unity 3D engine. Inspired by SpaceEngine on Steam.

Solar System Simulation

# A perfect game of Tic Tac Toe
If you play perfectly, every time you play you will either win the game, or you will draw the game. Furthermore if you play against another perfect player, you will always draw the game.
 
There are 3 "end-conditions" in the Tic Tac Toe game:

- You win! You get 10 points, the AI loses 10 points.
- You lose... You lose 10 points, the AI gets 10 points.
- You draw. Nobody gets any points.

# The MiniMax Decision Theory Algorithm

The Minimax algorithm functions based on the logic we just discussed with the point system.
 
The key to the Minimax algorithm is a back and forth between the two players, where the player whose "turn it is" desires to pick the move with the maximum score. In turn, the scores for each of the available moves are determined by the opposing player deciding which of its available moves has the minimum score. And the scores for the opposing players moves are again determined by the turn-taking player trying to maximize its score and so on all the way down the move tree to an end state.

<img src="/assets/img/projects/minimax.png" />

One player will always try to maximize his score, while the other one needs to minimize his score.

# My implementation
I started my implementation with the pseudocode from Wikipedia and turned it into Javascript code.
You can view, clone, fork, ... my full implementation on my [GitHub](https://github.com/Wilmox).

```js
function minimax(node, depth, maximizingPlayer) is
 if depth = 0 or node is a terminal node then
 return the heuristic value of node
 if maximizingPlayer then
 value := −∞
 for each child of node do
 value := max(value, minimax(child, depth − 1, FALSE))
 return value
 else (* minimizing player *)
 value := +∞
 for each child of node do
 value := min(value, minimax(child, depth − 1, TRUE))
 return value


(* Initial call *)
minimax(origin, depth, TRUE)
```


# Play it yourself!
_(Works best on a computer or tablet)_ 
Click [HERE](/project/play-tic-tac-toe-minimax-ai) to play in fullscreen. GLHF!

<iframe src="/project/play-tic-tac-toe-minimax-ai" title="Tic Tac Toe VS Minimax Algorithm"></iframe>

Making an unbeatable Tic-Tac-Toe opponent using the MiniMax decision theory algorithm.

Tic Tac Toe VS Minimax

# Intelligent Virtual Assistants (IVA)
Intelligent Virtual Assistants are AI-powered agents that can produce customised responses by pulling information from different contexts. An IVA uses voice inquiries and natural language processing (NLP) to execute a variety of helpful tasks.

Some popular virtual assistants you might be familiar with:
* Google Assistant
* Siri
* Amazon Alexa
* Cortana
* J.A.R.V.I.S. (Iron Man)
* Skynet (Terminator)
* HAL 9000 (2001: A Space Odyssey)
* ...

# J.A.R.V.I.S. clone
My inspiration to experiment with virtual assistants came, in addition to having a keen interest in AI, from the AI assistants in famous movies like Iron Man and Blade Runner 2049. So I tried to make a version of J.A.R.V.I.S. from Iron Man with a female voice.

# Technologies used

Virtual assistants that may be accessible through online chat in a specialised way are frequently referred to as "chatbots." But this is more than a chatbot. It implements the OpenAI API model called **text-davinci-002**. This is the most capable **GPT-3** model that can understand and generate natural language.

* **[OpenAI API](https://beta.openai.com/docs/api-reference/introduction?lang=python)** + payment plan setup
* **[GPT-3](https://beta.openai.com/docs/models/overview)**: text-davinci-002 (understand and generate natural language)
* **[Speech Recognition](https://github.com/Uberi/speech_recognition)**: Python library that implements the Google Speech Recognition API
* **[Offline Text2Speech pyttsx3](https://github.com/nateshmbhat/pyttsx3)**: male/female Text2Speech voice(s)

# Example outputs

*Some examples of the voice recognition with the OpenAI response* 

Unfortunately, I can only display the input and output of the GPT-3 speech recognition. Because GPT-3 is a paid NLP model, I would like to avoid exploits (and the emptying of my bank account) by implementing a working example on here. **BUT** you can always clone or fork [my version from Github](https://github.com/Wilmox/virtual-assistant-python) and try it out with your own API key (first 3 months you get $18 in free credit).

When provided a prompt, the OpenAI API will answer almost instantaneously with a relevant response.

<img src='/assets/img/projects/virtual-assistant-python/who-are-you.png' /> 

<img src='/assets/img/projects/virtual-assistant-python/meaning-of-life.png' /> 

It can even give you a little bit of history... 

<img src='/assets/img/projects/virtual-assistant-python/philosophy.png' /> 

...or work out a math problem. 
<img src='/assets/img/projects/virtual-assistant-python/math.png' />

Creating a intelligent virtual assistant like J.A.R.V.I.S. using GPT-3 in Python.

Projects_:

Conecta Software

International Internship

I-Talent Portfolio

Portfolio

Energy Transition Challenge

International Research Project

Race to the Future

Bachelor's internship

Solar System Simulation

Side project

Tic Tac Toe VS Minimax

Side project

Intelligent Virtual Assistant

Side project