System requirement analysis, design and Development

Description of the assessment

The first part is a report of evaluations of different current available big data platforms and implementation of a data warehouse using one of these platforms.

The second part is to do a big data analytics task applying on the big data process technologies and machine learning algorithms.

Assessment Content

Processing Big Data has many challenges with 4Vs.

The assessment 1(CW1) requires you to investigate no less than 3 big data supported cloud platforms with a demo example of data warehouse implementation. The Python-SQL-based data warehouse implementation and data used for the demo will be explained in the practical sessions.You should Report your evaluation results of the platforms with at least 8 criteria following the evaluation guide. After evaluating the cloud infrastructure, you should be able to do data analysis tasks by processing a big dataset using Python (ideally should be PySpark). The dataset will be provided.

The suggested platforms are:
BigQuery (Google)
Azure (Microsoft)
Keboola
Red Hat OpenShift
Deepnote
Deliverables:
Report on evaluation of big data cloud platform and data warehouse demo implementation process. The report structure should follow the structure below

1. Introduction
The purpose and scope of the report
• how many platforms you would like to evaluate?
• the criteria for evaluation
• investigation methodology

2. Platform investigation
• Detailed report of evaluation on each platform according to the defined criteria.
• The comparation result

3. Big Data processingand analysis implementation

• Working on a big dataset to enable applying NoSQL, PySpark or similar techniques to do data analysis on one of the cloud platforms or simulate on your own PC. The analysis should include data EDA, classification and price prediction.
• The explanations and screenshots to support this section. Critical discussion on the reason special algorithms are selected to do the work.
• The dataset will be provided. The dataset is relatively big for assessment purpose, and you can download the data from module blackboard.

4. Conclusion
• Summarisation
• Experience (what you have learnt from the assessment) discussion
• Future work (what can be improved)

Attachment:- BIG DATA ASSIGNMENT.rar