Big data technology Flink e-commerce real time data warehouse data acquisition Chapter 5 business database acquisition
five point one Maxwell introduce Maxwell is an open source MySQL real-time crawling software written in Java by Zendesk. Read the MySQL binary log Binlog in real time, and generate messages in JSON format, which are sent to Kafka, Kinesis, RabbitMQ, Redis, Google Cloud Pub/Sub, files or other pUTF-8...
Posted by cronus on Mon, 17 May 2021 06:49:50 +0930
Flink's understanding of watermark
abstract For Flink, Watermark is a hard concept to get around. This article will use the perceptual intuition to help you sort out the concept of Watermark. 1 question It's easy to have a few questions about Watermark What are the common processing requirements / solutions in Flink stream proceUTF-8...
Posted by fabrice on Tue, 18 May 2021 03:34:53 +0930
Flink: from getting started to giving up
preface This blog has been written for more than a month. It's the most painstaking blog so far. It's also to summarize the knowledge points and use them as notes in the future. The whole article uses Scala programming. If you don't know Scala, you can see another blog of mine: https://blog.csUTF-8...
Posted by saito on Thu, 20 May 2021 05:39:34 +0930
flink real combat simulation simple double 11 real time statistics screen
background In the real-time processing of big data, real-time large screen display has become a very important display item, such as the most famous double eleven large screen real-time total sales price display. In addition to this, there are also some applications in other scenarios, such as UTF-8...
Posted by show8bbs on Wed, 02 Jun 2021 07:19:23 +0930
From 0 to 1 Flink: Time and Watermaker
Time and Watermaker event_time 1 Time classification In the flow processing of Flink, different concepts of time are involved, as shown in the following figure: Event time: the time when the event really happened; IngestionTime: the time when the event arrives at Flink; Processing time: the timUTF-8...
Posted by dwfait on Sun, 20 Jun 2021 05:44:16 +0930
The growth of Flink from 0 to 1 - Expansion: Streaming File Sink
Extension: Streaming File Sink streamFile 1 scene description Streaming file sink is a new feature introduced in Flink1.7 to solve the following problems: In big data business scenarios, there is often one scenario: External data is sent to kafka, and Flink is used as middleware to consume kafkUTF-8...
Posted by jamesnkk on Mon, 21 Jun 2021 08:25:10 +0930
Flink+Hologres 100 million users real time UV accurate de duplication best practice
UV and PV computing are usually divided into two scenarios because of different business requirements Offline computing scenario: Based on T+1, calculate historical dataReal time computing scenario: real time computing of daily new data, de duplication of user tags For offline computing scenariUTF-8...
Posted by dutchboy on Tue, 29 Jun 2021 07:09:07 +0930
Parallelism of Source
Part one: Flink's first entry program In flink, a job must have Source (Data Source) and Sink (Data sink), but it may not have Transformations In the real-time computing DataStream API, what is Source used for? Source is used to obtain external data sources. According to the methods of obtaininUTF-8...
Posted by Ark3typ3 on Sat, 03 Jul 2021 03:45:43 +0930
From 0 to 1 Flink's growth path
Order automatic praise demand In the field of e-commerce, there will be such a scenario. If users buy goods, they will not do so for a certain period of time after the order is completed After evaluation, the system automatically gives five-star praise, and uses Flink Timer to realize this funcUTF-8...
Posted by jonsjava on Thu, 22 Jul 2021 05:19:56 +0930
Complete solution of big data Flink e-commerce real-time warehouse actual combat project process (VI) DWM layer business implementation
Project Summary Previously, we have distributed the data to the desired location through dynamic shunting. In order to facilitate the explanation of subsequent contents, we can import the information of the configuration table, and then send the data to the corresponding kafka subject or hbase UTF-8...
Posted by Edwin Okli on Mon, 02 Aug 2021 09:34:35 +0930