Original text link：here Introduction When I talk about Iceberg and open table formats, I have a strong preference to focus on just Iceberg. I like to promote it by talking about what it does well and areas where I think it stands apart. I also dislike pointing out what other projects don’t do or don’t […]
简述 hudi，重度依赖spark做了一套table format的设计和文件管理，解决批流存储统一的问题。所有元数据，如果不依赖metastore都是放在文件中（它自身设计了一个简单的metastore在1.0版本，当前主要与hive metastore结合可以做湖仓，虽然不是必须项，用于其它计算引擎的外部表查询）。 文件众多，会有小文件管理问题，所以建议使用至少需要有spark3环境。 构建 Hudi的meta data使用hfile格式，存储文件信息，避免遍历obs的消耗。问题是源码使用2.4.9版本的hbase，默认基于hadoop2.x版本，如果使用hudi-bundle.jar在hadoop3.x环境运行，会报意向不到的一些异常，例如类文件找不到。 Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.client.HdfsDataInputStream.getReadStatistics()Lorg/apache/hadoop/hdfs/DFSInputStream$ReadStatistics; 解决办法是自己编译hbase2.4.9，指定hadoop.profile=3，再编译hudi。 注意，hbase有一些坑，windows环境很多命令不支持，不安装相应shell命令的情况下，可以注释无用的一些exec-maven-plugin，主要是用来做校验之类的工作。 hudi编译 参考资料： HUDI-META-HBASE ISSUE
Source Link: here I remember learning how to play the recorder in elementary school. I remember square dancing. I remember cursive. I built a model of a Spanish mission out of sugar cubes. Some of this was fun. Some of it wasn’t. Some of it probably contributed, in some indirect way, to my general ability […]
自古以来，帝王多疑，而且亲信身边的人。 兼听则明，偏听则暗。 在企业内，亦是如此。 孰是孰非，难有一个标准。 人的想法，部份来自身边人。 春秋战国，臣子大家养了数千门客，想要多一些能人志士出谋划策。 由上而下，少有真才实干的人，容易出现墙头草的情况。容易改变自己的立场，缺乏独立思考和决断的能力或者被迫改变。一言谈的居多，埋没人才。
Excerpt from official blog: https://blog.twitter.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm Twitter aims to deliver you the best of what’s happening in the world right now. This requires a recommendation algorithm to distill the roughly 500 million Tweets posted daily down to a handful of top Tweets that ultimately show up on your device’s For You timeline. This blog is an introduction to […]
Source Link: https://www.happierhuman.com/live-consciously/ There are generally two groups of people as far as awareness is concerned. Those that are wired to think before acting… and those who inherently act impulsively. Pausing to collect your thoughts, and considering the consequences of your actions, are habits of conscious people. As you may imagine, there’s much more depth to practicing conscious living, and […]
