Number of file descriptors used over the limit. Number of client packets sent (responses and notifications). Number of client requests (typically operations) received. This goes up when the server receives more requests than it can process. Note the up and down curves created by the purge task:Īnd here are some Zookeeper alerts configured in Server Density:Īmount of time it takes for the server to respond to a client request (since the server was started). Here is how Server Density graphs disk usage and memory usage. Still, keep an eye on this, especially if your server is shared with other services, say Kafka. Zookeeper disk writes are asynchronous which means they shouldn’t have high IO requirements.
Use purge task to clean up dataDir and dataLogDir. Still we recommend dataLogDir to be on a dedicated partition and watch for disk usage. Zookeeper data is usually ephemeral and small. More nodes result in more writes and higher overall traffic. Zookeeper standards are: 20% writes, 80% reads. Keep an eye on this, especially if you notice any performance degradation. Zookeeper servers can incur a high network usage. Watch for swap usage, as it will degrade performance on Zookeeper and lead to operations timing out (set vm.swappiness = 0). JVM heap size shouldn’t be bigger than your available RAM. Or if you run Zookeeper via supervisord (recommended) you can alert the supervisord resource instead. You can also use the following script to check if the server is running: $INSTALL_PREFIX/zk-server-3/bin/zkServer.sh status When process list contains the regexp /usr/bin/java*$. Is the right binary daemon process running? Well, the Zookeeper list that satisfies this criteria is not that long. Monitoring Zookeeper: Metrics and AlertsĪs per previous articles, our general rule of thumb is “collect all possible/reasonable metrics that can help when troubleshooting, alert only on those that require an action from you”. With that in mind, here is our very own checklist of best practices, including key Zookeeper metrics and alerts we monitor with Server Density. That explains the popularity of higher level API wrappers, like the one created by Netflix team ( Curator).
From a client programming point of view there are plenty of low level and error handling pitfalls. Zookeeper can be quite a tricky service to manage. They store any data inside, and notify watchers on any event pertaining to them. Zookeeper nodes (ZNodes) are like hierarchical file system files (eg. All data replicates across all nodes and the leader performs atomic broadcasts to other servers, therefore guaranteeing strong ordering on changes propagation. It provides a distributed hierarchical file system that helps with the difficulties associated with services working in different machines (discovery, registration, configuration, locking, leader selection, queueing, etc). You know, keep your service-oriented architecture nice and clean. Check out the recorded session at the bottom of this blog post.Īpache Zookeeper works at the zoo-not your usual zoo, but similar-and does what you’d expect. Amongst other topics, we discussed how we use Zookeeper here Server Density. Update: We hosted a live Hangout on Air with some members of Server Density engineering and ops teams. Share on Facebook Share on Twitter Share on Reddit Share on Reddit
0 kommentar(er)
