Reference no: EM133545253

Cloud Computing

Overview and objectives

The goal of this project is to propose a cloud-based application in a proposal and implement the application using learned cloud computing technologies. You could choose one of two types of projects below and find a specific topic suitable for the type. Some project ideas will be demonstrated in the video recordings of the previous student projects that were the winners of the competition last year. A novel project with excellent implementations will be nominated for the student project competition awards this year. In this open project, the understanding of cloud computing's concepts and the ability to use the respective cloud technologies will be assessed. As an individual project, you are required to complete both the proposal and implementation all by yourself.

Proposal

The proposal aims to test your understanding of the concepts, characteristics, and relevant technologies in Cloud Computing. Moreover, your ability to design a cloud-based application with a reasonable budget will be assessed. The project proposal must NOT exceed two A4 pages and it should include the following sections:

Introduction:
o Give background information about this project: What is this project about?
o Explain the motivation of this project: Why is this project important?
o Describe the overall objective and features of the project: What features does this project have?
o Explain the limitations of traditional computing solutions: Why doesn't traditional computing solve the problem well?
o Explain the benefits brought by cloud computing: How does cloud computing fit into this project?

Technical Solutions:
o Describe what cloud technologies you've used in this project (e.g., k8s for the front-end app, NoSQL for the back-end data storage).
o Provide a monthly cost estimation of all the cloud resources used in this project (e.g., costs of VMs, K8s cluster, networking, Load Balancers, etc.).

Architecture Design:
o Depict the workflow or framework of the project in a figure (e.g., micro-service framework).

The teaching team will follow up on your implementation to ensure that the project is feasible.

To broadly accommodate students with different backgrounds, the projects are categorised into two types:
Type I: a scalable and reliable web-based application using Micro-service architecture and related technologies.

• e.g., PHP/JavaScript + MySQL/NoSQL + Docker/K8s + Load Balancing.
• This project type focuses on developing and deploying scalable, reliable, and resilient applications in the cloud.
• The deployed application should be scalable and support rolling updates and rollback features.

Type II: a data-analytic application using computing models for Big Data.
• e.g., Jupyter Notebook + Scala/Python/Java + Spark SQL / Streaming / MLlib
• This project type focuses on solving a complicated real-world problem by analysing a large amount of data.
• The real-world problem should be solved by analysing the large-scale data.

You can only choose one type and find out the specific topic that suits the type according to your background and preference. There are a few technical requirements that must be met when designing your project in the proposal.

Technical requirements:
If your project belongs to Type I,
• For the front-end design, you should have a functional user interface allowing users to interact with the application (e.g., login, search, etc.).
• To support the overall objectives of this project, you should have a back-end design (e.g., database) working together with the front-end UI.
• You should use docker technologies to containerise your applications with multiple containers running in a microservice architecture.
• To make your application scalable, reliable, and resilient, you should use either Docker Swarm or
Kubernetes to orchestrate multiple containers across hosts for your application.

If your project belongs to Type II,

• You need to perform a challenging analytic problem that requires large-scale data processing (e.g., big data queries, classification, regression, clustering, association rule mining over big data). You need to pick up a large dataset in the real world (must be at least 10, 000 records and contain multiple data files/tables). Some publicly big datasets are recommended below. To persist the data, you need to store the data in either a database (relational/non-relational databases, e.g., MySQL/Redis/MongoDB/Cassandra) or a distributed file system (e.g., HDFS).
• To avoid "Garbage in Garbage out", you need data pre-processing (e.g., removing/replacing NaN values, converting data types, imputation, etc.) to ensure the data quality for the downstream analytical task.
• You should run the analytical task on a Spark cluster using Spark programming techniques and the related Spark built-in libraries (i.e., SQL, MLlib, Streaming, and GraphX). You can choose either Python or Scala as the programming language.
• To demonstrate the analysis results, your program should visualise results with some tools (e.g., matplotlib in Python) in Jupyter/Zeppelin Notebook.

No matter which types your project belongs to, the cloud infrastructure costs should be reasonable. You can use the Google Cloud Platform Pricing Calculator when making the budget plan in the proposal.

Implementation
The implementation aims to test your ability to build and deliver a cloud application that is proposed in your project proposal. The implementation should be technically consistent with the proposed features and functionalities in the proposal. You are allowed to use your own code in the previous programming/analytical courses or an open-source web project (with consent). For example, your previous web development project in INFS3202 can be re-used but you need to containerise the application and deploy it on the cloud. If the previous project was a group one, it's your responsibility to have the team members' consent and you need to clearly inform the team members that the workload of the individual project in INFS3208 is to deploy the application in the cloud with the taught technologies. To demonstrate the implementation, you need to present it to your tutor in the tutorial and practical sessions in Week 13. The presentation should include but not be limited to the following parts:
• Background & Motivation.
• Project architecture and applied technologies.
• For Type I projects, database design (for database users) or data storage (for file system users).
• For Type II projects, descriptions of the used data and analytic models (Spark programming).
• Results & Discussion

You MUST back up the implementation code and data regularly (at least every week). It is your responsibility to ensure that you fully understand the technical requirements of this individual project. If you have any concerns or problems, please contact your session's tutor, or ask questions on Ed Discussion.

You should use the GCP credit wisely and avoid all unnecessary expenses on cloud services during the project. Also, you should monitor the balance of the credits during the implementation. Please contact your tutor as soon as possible when the credit is running out (less than $10). Note that the GCP credit is not unlimited. It is strongly recommended to develop, test, and debug locally before deploying the project in the cloud.