云计算与大数据技术主要内容(云计算与大数据介绍)

The key word of cloud computing is "integration", whether you go through the now mature traditional Virtual machine segmentation technology, or the massive node aggregation technology later used by Google, integrates massive server resources through the network, schedules and distributes them to users, thereby solving the problems caused by insufficient storage and computing resources for users. The problem.

Big data is a new topic brought about by the explosive growth of data, how to store the massive data generated in today's Internet era, how to effectively use and analyze these data, and so on.

You can understand the relationship between the two in this way. Cloud computing technology is a container, and big data is the water stored in this container. Big data relies on cloud computing technology for storage and processing. computational.

Extended information:

Cloud computing is often confused with grid computing, utility computing, and autonomous computing.

Grid computing: a type of distributed computing, a super virtual computer composed of a group of loosely coupled computers, often used to perform large-scale tasks;

Utility computing: IT resources A packaging and billing method, such as measuring costs separately according to calculation and storage, like traditional public facilities such as electricity;

Autonomous computing: a computer system with self-management functions.

In fact, many cloud computing deployments rely on computer clusters (but are quite different from the composition, architecture, purpose, and working methods of grids), and also absorb the characteristics of autonomous computing and utility computing.

The generally accepted characteristics of cloud computing are as follows:

(1) Ultra-large scale

"Cloud" has a considerable scale. Google Cloud Computing already has more than 1 million servers, and the "clouds" of Amazon, IBM, Microsoft, Yahoo, etc. all have hundreds of thousands of servers. Enterprise private clouds generally have hundreds or thousands of servers. "Cloud" can give users unprecedented computing power.

(2) Virtualization

Cloud computing supports users to obtain application services at any location and using various terminals. The requested resources come from the "cloud" rather than a fixed tangible entity. The application runs somewhere in the "cloud", but users don't actually need to know or worry about the specific location where the application is running. With just a laptop or a mobile phone, everything we need can be achieved through network services, even tasks such as supercomputing.

(3) High reliability

"Cloud" uses measures such as multiple copies of data, fault tolerance, and isomorphic and interchangeable computing nodes to ensure high reliability of services. Cloud computing uses More reliable than using your local computer.

(4) Versatility

Cloud computing does not target specific applications. With the support of "cloud", ever-changing applications can be constructed. The same "cloud" can support different applications at the same time. application is running.

(5) High scalability

The scale of "cloud" can be dynamically expanded to meet the needs of application and user scale growth.

(6) On-demand services

The "cloud" is a huge resource pool that you purchase on demand; the cloud can be billed like tap water, electricity, and gas.

Big data characteristics:

1 Volume: The size of the data determines the value and potential information of the data considered;

2 Type (Variety) ): the diversity of data types;

3 Velocity: refers to the speed at which data is obtained;

4 Variability (Variability): hinders the processing and effective management of data process.

5 Veracity: the quality of the data

6 Complexity: the huge amount of data comes from multiple channels

7 Value : Reasonable use of big data to create high value at low cost

If you want to systematically recognize big data, you must decompose it comprehensively and carefully, starting from three levels:

< p>The first level is theory. Theory is the only way of cognition, and it is also the baseline that is widely recognized and disseminated. Here, we will understand the industry’s overall description and characterization of big data from the definition of big data characteristics; we will deeply analyze the preciousness of big data from the discussion of the value of big data; gain insight into the development trend of big data; and start from the special and important issue of big data privacy. Examine the long-term game between people and data from a perspective.

The second level is technology. Technology is the means to embody the value of big data and the cornerstone of progress. Here, the entire process of big data from collection, processing, storage to result formation will be explained from the perspective of the development of cloud computing, distributed processing technology, storage technology and perception technology.

The third level is practice, and practice is the ultimate value manifestation of big data. Here, we will describe the beautiful scene that big data has shown and the blueprint for its upcoming realization from four aspects: Internet big data, government big data, enterprise big data and personal big data.

Reference materials: Baidu Encyclopedia-Big Data Baidu Encyclopedia-Cloud Computing

The big data technology major has statistics, mathematics, and computer science as the three supporting disciplines; biology, medicine, environmental science, economics, sociology, and management are the applied expansion disciplines. In addition, you need to learn data collection, analysis, and processing software, as well as mathematical modeling software and computer programming languages.

1. The courses majored in big data technology mainly include: "Basics of Programming", "Python Programming", "Basics of Data Analysis", "Linux Operating System", "Python Crawler Technology", "Python Data Analysis", "Java Programming", "Hadoop Big Data Framework", "Spark Technology and Application", "HBASE Distributed Database", etc.

2. The big data technology major is an emerging major established in conjunction with the national big data and artificial intelligence industry development strategies. This major is oriented to the field of big data applications and mainly studies big data operation and maintenance, collection, storage, and analysis. , visualization knowledge and technical skills.

3. The research direction of big data technology and application is the "Internet" that combines the cutting-edge technologies of big data analysis and mining and processing, mobile development and architecture, software development, and cloud computing. +"Cutting-edge science and technology major. Graduates of this major can work as big data project implementation engineers, big data platform operation and maintenance engineers, and big data platform development engineers.

4. This major aims to train students to systematically master data management and data mining methods, and become capable of big data analysis and processing, data warehouse management, comprehensive deployment of big data platforms, big data platform application software development and data products Senior professional big data technical talents with visual display and analysis capabilities.

Basic knowledge of big data, popular science, just buy this book personally, there are many introductions to such books in the era of big data of big data.

In addition, big data technologies include data collection, data access, infrastructure, data processing, statistical analysis, data mining, model prediction, and result presentation.

Big data analysis mining and processing, mobile development and architecture, software development, cloud computing and other cutting-edge technologies.

Major courses: object-oriented programming, Hadoop practical technology, data mining, machine learning, data statistical analysis, advanced mathematics, Python programming, JAVA programming, database technology, Web development, Linux operating system, university Data platform construction and operation and maintenance, big data application development, visual design and development, etc.

Aims to train students to systematically master data management and data mining methods, and become capable of big data analysis and processing, data warehouse management, comprehensive deployment of big data platforms, big data platform application software development and visual display and data products. Senior professional big data technical talents with analytical capabilities.

Extended information:

Application fields

Big data technology has been penetrated All aspects of society, including medical and health care, business analysis, national security, food security, financial security, etc. In 2014, from the perspective of big data as an important strategic resource for the country and accelerating the realization of innovative development, a cultural atmosphere and era of "using data to speak, using data to manage, using data to make decisions, and using data to innovate" were formed in the whole society. feature.

Big data science will become the core of computer science, artificial intelligence technology (virtual reality, commercial robots, autonomous driving, all-round natural language processing), digital economy and business, Internet of Things applications, and various humanities and social sciences. core of the development of the field.

Reference: Baidu Encyclopedia-Big Data Technology and Application