Short, temporary moments of forgetfulness are happening to more of us more often these days, memory experts say. Grant Shields was teaching a college seminar to 24 students last week when his mind went blank. He’d forgotten the name of his teaching assistant. “I was embarrassed,” says Dr. Shields, who thought he heard students laugh […]
When it comes to the problem of high disk io on yarn cluster, it may lead to many undesirable situations, such as application’s slowing down too much, hdfs operation shows high latency, although the file being operated is very small. What happened? It may be a shuffle problem. For example, spark is our main compute […]
范欣欣的博文讲解深入又很清晰。 如何实现高速读写？ 是否使用offheap？ jdk跳表cslm实现及阿里ccsmap的优化？ 详见：http://hbasefly.com/2019/10/18/hbase-memstore-evolution/
低频用户冷启动需要考验算法模型的泛化能力。建模中容易产生马太效应。 何谓马太效应？摘自wiki media， 天之道，损有余而补不足。人之道，则不然，损不足以奉有余。 凡有的，还要加给他，叫他有余；凡没有的，连他所有的也要夺去。 强者愈强，弱者愈弱。
事件： datax on yarn 3点出现任务失败，错误信息：[Error] DataX receive unexpected signal 15, starts to suicide。jdk还打印了stack信息。 问题分析： signal15为外部kill，jvm打印thread dump也是由于kill产生，尤其是kill -3会thread dump。 是机器内存不足被OS kill? 到/var/log/messages寻找，无相关日志，机器内存监控显示内存充足。 yarn资源抢占，被yarn rm kill了？ 集群使用fair + drf调度策略，并开启了资源抢占。不同的队列之间资源会相互竞争和抢占。定位问题，寻找rm的日志，找到对应container被kill的日志。 添加yarn抢占容器数量监控 Hadoop_ResourceManager_AggregateContainersPreempted{name=”QueueMetrics”,q0=”root”, user=””,instance=”x.x.x.x:7011″} 解决方式： 1、队列划分更精细，资源划分按业务，调整配比 2、关闭资源抢占 原理： FIFO Scheduler、Capacity Scheduler、Fair Scheduler 什么时候发生抢占？ Ø 最小资源抢占, 当前queue的资源无法保障时，而又有apps运行，需要向外抢占。Ø 公平调度抢占, 当前queue的资源为达到max，而又有apps运行，需要向外抢占。 具体详见yarn源码实现。 参考： https://cloud.tencent.com/developer/article/1195056 YARN资源调度策略
The Youth’s Companion, Feb.7 1889, p.73(Vol 62) JUST THE BOY WANTED, II IN THE LAW, by Judge Oliver Wendell Holmes (from Howe, Mark DeWolfe. Research materials relating to life of Oliver Wendell Holmes.) A boy who wants to succeed in the law will probably do so. An encouraging thought, as far as it goes. But […]
