Big data common language Scala in 2021: Scala advanced usage sample class
catalogue Sample class Defining sample classes Sample class method Sample object Sample class Sample class is a special class, which can be used to quickly define a class for saving data (similar to Java POJO class), and it will automatically generate apply method, which allows us to quickly crUTF-8...
Posted by zhTonic on Thu, 13 May 2021 02:24:16 +0930
How to make the project take off quickly by using strategy pattern to optimize code practice
1, Background I took over a springboot project. In the module I am responsible for, there is a function of user registration, but in particular, this registration is not re registration, but synchronization of old data from the database of the old system to the database of the new system. BecauUTF-8...
Posted by nbarone on Fri, 14 May 2021 05:14:53 +0930
@ceph Distributed Storage
Distributed storage ceph 1. Introduction to ceph 1.1, What is ceph ceph A unified, distributed storage system designed to provide better performance, reliability, and scalability. "Unified":That means we can just rely on ceph This storage system provides object storage, block storage and file UTF-8...
Posted by PHPBewildered on Fri, 14 May 2021 05:59:49 +0930
Big data technology Flink e-commerce real time data warehouse data acquisition Chapter 5 business database acquisition
five point one Maxwell introduce Maxwell is an open source MySQL real-time crawling software written in Java by Zendesk. Read the MySQL binary log Binlog in real time, and generate messages in JSON format, which are sent to Kafka, Kinesis, RabbitMQ, Redis, Google Cloud Pub/Sub, files or other pUTF-8...
Posted by cronus on Mon, 17 May 2021 06:49:50 +0930
Experiment 5 primary practice of Spark Streaming programming
The purpose of this study was to investigate the effects of different factors on the quality of life Through the experiment, we learn the installation and use of Flume;Master the programming method of using Flume as the data source of Spark Streaming. 1, Installation and configuration of FlumeUTF-8...
Posted by SnakeFox on Mon, 17 May 2021 06:54:41 +0930
[scratch crawler] crawls the review data of all applications in Huawei's application market
preface The project needs to crawl the comment data. Make a record here. The data crawled here is from the web side. You may consider crawling the data from android app in the future. 1, Install and create the Scrapy project Official document of Science: https://docs.scrapy.org/en/latest/intro/UTF-8...
Posted by gauss on Thu, 20 May 2021 04:09:46 +0930
Flink: from getting started to giving up
preface This blog has been written for more than a month. It's the most painstaking blog so far. It's also to summarize the knowledge points and use them as notes in the future. The whole article uses Scala programming. If you don't know Scala, you can see another blog of mine: https://blog.csUTF-8...
Posted by saito on Thu, 20 May 2021 05:39:34 +0930
Hadoop cluster configuration Secret-free login and cluster configuration
Write at the beginning: Is it difficult for De Xuan not to talk to a bosom friend, to talk to a bosom friend, or to waste his tongue on bosom friends? Next time, we finished configuring the basic environment in our previous article. This is just as important as secret-free login, and of course,UTF-8...
Posted by timj on Fri, 21 May 2021 05:59:58 +0930
[3 days to master Spark] - Spark on horn
Spark on YARN Property configuration and service startup It is very important to submit Spark Application to run on YARN cluster. Most enterprises run on YANR cluster file: http://spark.apache.org/docs/2.4.5/running-on-yarn.html When Spark Application is running on yard, you can specify yard asUTF-8...
Posted by kris81 on Sat, 22 May 2021 05:39:35 +0930
[3 days to master Spark] - RDD concept and WordCount case
RDD of Spark RDD concept For a large amount of data, Spark uses a data structure called resilient distributed datasets (RDD) to save the internal calculation. All operations and operations are based on the RDD data structure In the Spark framework, encapsulate the data into the collection: RDD.UTF-8...
Posted by rastem on Sat, 22 May 2021 06:54:50 +0930