Showing posts with label performance. Show all posts
Showing posts with label performance. Show all posts

Wednesday, August 9, 2017

Cassandra Performance Tips - Batch inserts, Materialized views, Fallback strategy

Introduction

During a recent project we ran into multiple issues with Cassandra's performance. For example with queries being slow or having timeouts on only a specific environment (though they should have the same setup), inconsistently stored results, and how to optimize batch inserts when using Scala.

This blogpost describes how they were solved or attempted to be solved.


Setup: Cassandra running in a cluster with three nodes.

Performace related lessons learned

  1. On certain environments of the DTAP build-street slow queries (taking seconds) and weird consistency results appeared. Not all 4 environments were the same, though Acceptance and Production where as much the same as possible.

    We found as causes:

    Slow queries and timeouts: Cassandra driver was logging at both OS and driver level
    Inconsistently stored results: The clocks from different clients accessing C* were not the same, some off for minutes. Since the default in v3 of the Datastax/Cassandra driver protocol is clientside generated timestamps, you can get in trouble of course, since then the one with the most recent timestamp just always wins. But implementing serverside also won't be obvious, since different C* coordinators can give a millisec different timestamp.

  2. For Gatling performance tests written in Scala, we first needed inserting 50K records in a Cassandra database, simulating users already registered to the system. Trying to make this perform several options were tried:

    a- Plain string concatenated or prepared statements where taking over 5mins in total
    b- Inserting as a batch (apply batch) has limit of 50KiB in text size. That limit is too low for us: 50K records is almost 5MB. Splitting up was too much of a hassle.
    c- Making the calls async, as done here: https://github.com/afiskon/scala-cassandra-example
    But we were getting:

    17:20:57.275 [ERROR] [pool-1-thread-13] TestSimulation - ERROR while inserting row nr 13007, exception =
    com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /172.20.25.101:9042 (com.datastax.driver.core.exceptions.BusyPoolException: [/172.20.25.101] Pool is busy (no available connection and the queue has reached its max size 256)))
    at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:211)
    at com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:46)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:275)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution$1.onFailure(RequestHandler.java:336)
    at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
    at cle.common.util.concurrent.Futures$ImmediateFuture.addListener(Futures.java:102)
    at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1184)
    at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1120)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.query(RequestHandler.java:295)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:272)
    at com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:115)
    at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:95)
    at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:132)
    at UserDAO.insert(UserDao.scala:58)
    ...


    Turns out it is the driver's local acquisition queue that fills up. You can increase it via poolingOptions.setMaxQueueSize, see: http://docs.datastax.com/en/developer/java-driver/3.1/manual/pooling/#acquisition-queue
    We set it to 50000 to be safe it would just queue all records (50K). For a production environment this might not be a good idea of course, you might need to tune it to your needs.
    And the threads we set to 20 in the Executioncontext (used by the DAO from the example from github above). You can set it as in this example: http://stackoverflow.com/questions/15285284/how-to-configure-a-fine-tuned-thread-pool-for-futures

  3. Increasing CPU from 4 to 8 did seem to improve performance, less CPU saturation.

  4. Each time adding one more materialized view increases insert performance by 10%  (see here)

  5. For consistency and availability when one of the nodes might be gone or unreachable due to network problems, we setup Cassandra write such that first EACH_QUORUM is tried, then if fails, LOCAL_QUORUM as fallback strategy.

Below articles did help to analyse the problems further:

Tuesday, August 8, 2017

Gatling Lessons learned

Introduction

This post describes a couple of best practices when using Gatling, the Scala based performance and load-testing tool.


Also one or two Scala related tips will be shown.

Lessons learned

  1. You can't set a header-field in the header like this in a scenario:

    .header("request-tracing-id", UUID.randomUUID().toString())

    This is because the scenario is only created once, and unless you use session variables it, it is all static (a function).

    To solve this one can use a feeder, like this:

    val feeder = Iterator.continually(Map("traceHeader" -> UUID.randomUUID().toString))

    And then replace the UUID.randomUUID().toString() line with:

    .header("request-trace-id", "${traceHeader}")

  2. A Scala 2.11 example of a ternary (maybe not the best solution Scala-wise but readable :)

    .value(availableBalanceDecimal, if (dto.availableBalanceDecimal.isEmpty) null else dto.availableBalanceDecimal.get)

  3. Connecting correctly to a service with a Gatling test in one environment (Nginx, ingress, Kubernetes) for some reason did not work. But it was able to connect to the service under test correctly in another environment. Apparently it had something to do with a proxy in between because had to add .proxy() and it worked:

    val httpConf = http
    .baseURL("http://172.20.33.101:30666") // Here is the root for all relative URLs
    .header(HttpHeaderNames.ContentType, HttpHeaderValues.ApplicationJson)
    .header(HttpHeaderNames.Accept, HttpHeaderValues.ApplicationJson)
    .proxy(Proxy("performance.project.com", 30666))   // note it is the *same* machine as the baseURL,but specified by name... 


  4. A .check() with in it a .saveAs() will *not* happen when the earlier expression evaluates to false, or the conversion fails.
    Kindof makes sense when evaluates to false, but you might miss this one; or maybe you don't even want this .is() because all it means now only isFinished will be set to FINISHED and else it won't be set in the below example.

    someText is always found in the session, but the other one, isFinished, is not.

    .check(jsonPath("$[0].status").saveAs("someText"))
    .check(jsonPath("$[0].status").is("FINISHED").saveAs("isFinished"))
    ...

    .exec(session => {
       val isFinished = session.get("isFinished").asOption[String]
       logger.debug("Generated isFinished = {}", isFinished.getOrElse("Could not find expected isFinished..."))
       session
    })

    .doIf(session =>
       (!session.get("someText").as[String].equals("FINISHED") ||
       session.get("isFinished").as[Boolean].equals(false)
    ))(
    ...


    When running at DEBUG level the above logs:
       Session(Map(someText -> TIMED_OUT, ...)  /// So .saveAs() ocurred
       13:45:56.983 [DEBUG] SomeSimulation - Generated isFinished = Could not find expected isFinished...  


    So the second.saveAs() did not occur for isFinished at all, since it is not set to true nor false; it is not set at all!

Monday, October 31, 2011

Android UI Development: Tips, Tricks Techniques from Devoxx 2010 Summary

Quick summary of the most important/new tips from the Parleys talk Android UI Development: Tips, Tricks Techniques at Devoxx 2010. Now free for viewing.

13:40: tool for tracking allocations in Android via Debug.setAllocationLimit(int)

25:15: example showing the use of DDMS, Heap Analysis Tool, hprof, adb shell dumpsys meminfo

36:14: don't over-invalidate when redrawing. Only redraw what you have to. Demo shows use of traceview to detect too much being re-drawn.

44:20: (new) hierarchy viewer example. You can use Annotations to export your own values, i.e. the ones you want to check in the viewer.

48:09: you can use the ViewStub to not create your Views too early. Kindof lazy-instantiation. Uses android:inflatedId.

51:15: better way of using an ImageView (e.g icon) with a TextView inside a LinearLayout: android:drawableLeft(Top/Bottom/Right). Much more efficient because not using 3 Views.

51:45: shows use of the static analysis tool 'layoutopt'. It needs an layout XML file and it tells you what can be improved.

53:45: the one you should know by now :) Don't inflate() views in ListView.getView(), re-use. Shows a graph of performance differences if you do vs if you don't.

54:24: tip on View.setDrawingCacheEnabled(true), and explains quicky how Android uses it when you flick a ListView.

Monday, July 4, 2011

Android pro tips for advanced expert Android developers

Short summary of the Google I/O 2011 session "Android Protips: Advanced Topics for Expert Android App Developers". Full presentation can be found here.

6:35: the parallel Activity pattern: based on SDK version, start another "parallel" code line. Easy check to see what your SDK version is so you can distinguish use of more advanced APIs as for example available since HoneyComb.
8:20: use interfaces for backward compatibility
9:09: get real user feedback before launching. Note you can also of course use Analytics to log an exception!
10:38: example of doing A/B testing
11:35: use Android Market for beta testing. It seems there's no special facility for this in the market, you just have to control it yourself... e.g password protect, obscure the name.
12:31: But don't forget to protect your package name before someone else takes it! So upload and save it but don't publish it.
13:43: don't assume the default (natural) orientation on a device is portrait, e.g most tablets start in landscape. The orientation sensor's default is also 90degrees "off"
15:00: Don't track devices to detect unique installations. Give your app a unique id using a UUID, store it in the prefs.
20:00: the app's data has to be always fresh if possible, the user does not have to wait. See also this recent post: http://android-developers.blogspot.com/2011/06/deep-dive-into-location.html
25:51: how to get fresher data
32:00: make your services intelligent like do everything async, let it die asap
33:13: make your app psychic: leverage what the phone already knows, remembers what the user has told it. Like the Account Manager. Preferences should follow the user across devices and resets (which means storing them offline). Give users the ability to delete their preferences, using the Backup Manager to backup shared preferences.
38:21: intercept links using an Intent Receiver
39:25: make your apps adaptive. E.g: behave as expected, optimize for different user experiences. You can specify the IME keyboard behavior by setting the appropriate action (Go, Search, Send, Done etc). Use android:imeOptions for that
43:20: handle volume and playback buttons icw requesting audio focus for playback
47:10: provide translations for your app and title description to have it better found in people's native language
48:00: make your app smooth: make everything async. Loader and CursorLoader are available since HoneyComb.

Sunday, September 5, 2010

Best of this Week Summary 30 August - 05 September 2010

  • A "series of (currently 5) articles introduces Contexts and Dependency Injection for Java EE (CDI), a key part of the Java EE 6 platform. Standardized via JSR 299, CDI is the de-facto API for comprehensive next-generation type-safe dependency injection as well as robust context management for Java EE. Led by Gavin King, JSR 299 aims to synthesize the best-of-breed features from solutions like Seam, Guice and Spring while adding many useful innovations of its own".

  • An extensive tutorial on creating a Spring MVC based mobile web application using MyEclipse. The IDE has already an iPhone skinned version built in, this tutorial shows how to add other mobile handsets like Android and Palm Pre. Basically: providing another style :)


  • How to use Apache ActiveMQ in combination with Spring.

  • Why ACID is hard to scale. and an argument that NoSQL/NoACID is the lazy way around these difficulties. Here's a post considering the performance argument for switching to a NoSQL database.

  • Great interesting series on the progress of converting the native code BBC News iPhone/iPad app to a web app using HTML5.

Sunday, June 13, 2010

Best of this Week Summary 7 June - 13 June 2010

  • When rewriting an app from scratch doesn't make sense, and when it might make sense.

  • Java deadlocks through Cyclic Dependencies: title says it all.

  • Carnegie Mellon University's CERT (Computer Emergency Response Team) has released the Basic Fuzzing Framework (BFF), which is a simplified version of automated dumb fuzzing and includes a Linux virtual machine that has been optimized for fuzz testing and a set of scripts to implement a software test. It helps identify and eliminate security vulnerabilities from software products.

  • A nice (external) analysis of the FIFA World Cup website against performance best practices.

  • "Nokia's set of learning and teaching materials on Java in Mobile Devices (JME/MIDP). In fact, it is Forum Nokia’s most comprehensive set of information ever. Included are five courses, 23 lectures, 25 lab exercises with source code, and more than 1,200 slides with notes. The material has been designed for university instructors and students, developers, and self-learners. Five main topics are covered:

    • Java ME basics

    • Multimedia

    • Networking and messaging

    • Games and graphics

    • Security

    Of course JME is not so sexy as Android these days, but Nokia still has the largest number of mobile phones out there. And most (even quite low end) support JME, so from a market perspective the potential is much higher than building apps for Android phones...

  • Several CSS reset examples, including a few really short ones.

Sunday, June 6, 2010

Best of this Week Summary 30 May - 6 June 2010

  • Google Web Toolkit vs. Smart GWT: Which should you choose as front-end? Start immediately with Smart GWT, or start with GWT and pick SGWT components when needed? Some insights here.
    Related to that, you might want to check Vaadin: "Vaadin is an open source web application framework for rich Internet applications. In contrast to Javascript libraries and browser-plugin based solutions it features a server-side architecture, which means that the majority of the logic runs on the servers. Ajax technology is used at the browser-side to ensure a rich and interactive user experience. On client-side Vaadin is built on top of and can be extended with Google Web Toolkit. Vaadin utilizes Google Web Toolkit for rendering the resulting web page. While Google Web Toolkit operates only client-side (i.e. a browser's JavaScript engine) – which could lead totrust issues – Vaadin adds server-side validation to all actions. This means that if the client data is tampered with, the server notices this and doesn't allow it.
    Historically, Vaadin has been compared to Echo and ZK frameworks that use similar of server-side programming model. The server-side APIs are quite similar providing both events and GUI components, but the client-side (i.e. web browser) interaction differs in the way that Vaadin uses Java programmed GWT widgets, while ZK is jQuery based, and Echo has its own implementation. Currently, the most frequently compared frameworks include Adobe Flex, Google Web Toolkit, Apache Wicket and ICEfaces."

  • A new open spec collaboration has started: OExchange, which is an open protocol for sharing any URL with any service on the web. Bigger parties involved are LinkedIn, Microsoft, Google.

  • A new way of phishing: TabNabbing - phishing by switching background tab content. Discovered by Firefox's creative lead Aza Raskin. Biggest challenge seems to get the malicious Javascript on a site the user goes to.

  • Google announced a partnership with VMWare (and thus SpringSource and thus Spring) at I/O by adding its (GTW) widgets to Spring and deployment to the VMWare cloud. In marketing speak: "This is VMware and Google's view of the power of using Spring along with Google's presentation widgets to get apps started in hours, delivered in days, and deployed in minutes". Below it's shown in a diagram:



  • The iPad isn't without "errors" in its usability (UI) according to Jakob Nielsen. For example: cross-app UI experience is inconsistent, and for some reason almost no app supports scrolling and shows information only per page.

  • "An overview of how to design websites and optimise them for Maemo, iPhone, Android, and a variety of touch and non-touch devices based on S60 on Symbian OS. After reading the document, you will have the basic knowledge you need to start developing mobile web pages that provide cross-browser-compatible content in a user-friendly manner. Furthermore, with the tips and advice contained in the document, you can avoid making design choices that could eventually lead to a dead end or poor design, thus saving time in implementing and debugging features that will not work".

  • Show Slow: an open source tool that helps monitor various website performance metrics over time. It captures the results of YSlow and Page Speed rankings and graphs them, to help you understand how various changes to your site affect its performance

Sunday, May 9, 2010

Best of this Week Summary 03 May - 09 May 2010

Sunday, April 18, 2010

Best of this Week Summary 12 April - 18 April 2010

Sunday, March 21, 2010

Best of this Week Summary 15 March - 21 March 2010

  • Twitter and Digg are moving from MySQL to Cassandra (a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model; a Facebook opensourced project). The reason for Digg for the move "is the increasing difficulty of building a high-performance, write-intensive application on a data set that is growing quickly, with no end in sight. This growth has forced them into horizontal and vertical partitioning strategies that have eliminated most of the value of a relational database, while still incurring all the overhead".
    Twitter has about the same reason: "No single points of failure", "Highly scalable writes (we have highly variable write traffic)", and "A healthy and productive open source community".
    Twitter tried HBase, Voldemort, MongoDB, MemcacheDB, Redis, Cassandra, and HyperTable amongst others before deciding to go with Cassandra. Interesting to read is how they slowly rollout Cassandra to limited sets of users.
    An introduction to Cassandra to get it up and running can be found here.


  • A short post on how to implement Automatic testing Oracle Service Bus using Hudson, Maven and SoapUI.

  • What to expect from HTML5 for webdevelopers.

Sunday, March 14, 2010

Best of this Week Summary 8 March - 14 March 2010

Sunday, January 10, 2010

Best of this Week Summary 28 December - 10 January 2010

  • "An interview with Duane Nickull, Senior Technical Evangelist at Adobe Systems as he revisits the notion of 'Web 2.0', and discusses some of the new architectural and human interaction patterns that are shaping the way in which we build RIA Web applications today. He talks about some of the new Flash authoring tools for the iPhone, the Open Screen Project, as well as the impact HTML 5 will have on Flash adoption." Transcription of the interview below the interview. Summary in four words: "Don't piss off users". He also mentions the Open Screen Project: "It is an industry-wide initiative, led by Adobe with the participation of other industry leaders, to enable the delivery of rich multiscreen experiences built on a consistent runtime environment for open web browsing and standalone applications."

  • Blogpost using Ehcache and Spring 3 for implementing over 100K of RSS feeds, with certain feeds under very high load. A follow-up post describes the use of Ehcache DX Monitor Module (beta).

  • Fun: a ZX-81 chess program in 1K. Actually only 672 bytes were available for the program!

Sunday, December 13, 2009

Best of this Week Summary 07 December - 13 December 2009

Sunday, November 29, 2009

Best of this Week Summary 17 November - 29 November 2009

Sunday, November 1, 2009

Best of this Week Summary 26 October - 01 November 2009

Sunday, October 25, 2009

Best of this Week Summary 19 October - 25 October 2009

Sunday, October 11, 2009

Best of this Week Summary 05 October - 11 October 2009

  • A quite in-depth article on creating highly-scalable components in Java. Described are the ideas used to create the concurrent Java components in the Amino Library Project (a set of concurrent building blocks).

  • Another SOA Design Pattern sample chapter from Thomas Erl' book "SOA Design Patterns": "Service Governance Patterns, comprising a number of 8 patters. Compatible Change and Version Identification deal with service versioning. Termination Notification addresses the final phase of a service, it’s retirement. Service Refactoring explains how to deal with changing service contracts. Service Decomposition, Proxy Capability, and Decomposed Capability include techniques needed to express coarse-grained services through multiple fine-grained ones. Distributed Capability helps increasing service scalability through processing deferral."

  • To ESB or not to ESB? Including a nine points checklist. Note that the article is from the creator of Mule (a lightweight ESB). Check also the comments here.

  • Interesting hack to reduce startup latency for mobile devices with HTML5: put the Javascript code in comments (thus /* */), such that is immediately downloaded but not evaluated. Then to load it when needed, parse the text, strip out the comment-tags & do an eval().

Sunday, October 4, 2009

Best of this Week Summary 28 September - 04 October 2009

  • "Tony Hoare introduced Null references in ALGOL W back in 1965 “simply because it was so easy to implement”, says Mr. Hoare. He talks about that decision considering it “my billion-dollar mistake”. Tony Hoare, is a British computer scientist, probably best known for the development in 1960, at age 26, of Quicksort. He also developed Hoare logic, the formal language Communicating Sequential Processes (CSP), and inspired the Occam programming language."

  • "Peter Thomas has updated his web framework performance benchmark with Grails and Tapestry. Apache Wicket still leads the best of breed frameworks in terms of request times and memory usage." Other observations:

    • Grails was far more productive than Tapestry 5. This was mainly due to the documentation quality of Grails compared to the scattered and not very well organized Tapestry documentation.

    • Ease of writing custom tag-libraries is IMO one of the best things about Grails.

    • Grails still has some way to go in terms of performance. I am told that significant performance optimizations for GSP will make it into 1.2

    • Session usage of the Seam + JSF combination is significantly higher compared to all the rest, around 760 KB per session."


  • "At the JVM Languages Summit, Josh Bloch expressed his concern about what he called the "semantic gap" between the source code we write and its performance at runtime: As performance improves, our ability to predict the performance consequences of our source code degrades." Feedback from people on this subject can also be found here.

  • "Mckoi Distributed Database (MckoiDDB) is a database system used by software developers to create applications that store data over a cluster of machines in a network. It is designed to be used in online environments where there are very large sets of both small and big data items that need to be stored, accessed and indexed efficiently in a network cluster. The focus of the MckoiDDB architecture is to support low latency query performance, provide strong data consistency through snapshot transaction isolation, and provide tools to manage logical data models that may change dramatically in physical network environments that may experience similar dramatic change.
    MckoiDDB is written in 100% Java and runs on any operating system that supports Java 1.6. MckoiDDB is released under the GPLv3 open source license."

Sunday, August 23, 2009

Best of this Week Summary 17 August - 23 August 2009

  • Here's a basic list of software architecture review guidelines. You can definitely also use it as a starting point for your documentation, or to see if your documentation is covering it all.

  • Shame on me: forgot to mention last week the VMWare acquisition of SpringSource! Basically the CTO says "the aim of this acquisition is to provide a comprehensive platform as a service (PaaS)". Here's a view on it from a technical level by Adrian Colver (SpringSource), and here's one from Frank Cohen.

  • Potential cloud "dangers" described during Black Hat USA 2009.

  • And related to the previous one, here's a short summary with interesting results on stress testing the Amazon EC2, Google AppLogic and Microsoft Azure cloud solutions. Bascically, the performance quite varied and response times varied by a factor of 20! And good to know: on Google AppEngine, any task can not last longer than 30 seconds!

  • Need to know "everything" about Java Generics? Then this large PDF Java Generics FAQs - Frequently Asked Questions is for you. Think you know everything about Generics? It still might be for you. For example, do you know what this says: "Enum<E extends Enum<E>>"? Or can you follow this one: "Can I use the type parameter of an outer type as part of the bounds of the type parameter of an inner type or a method?"? :)

Sunday, May 31, 2009

Best of this Week Summary 25 May - 31 May 2009

  • Facebook is now also supporting registration/login via a GMail account and OpenID, see the image below. I especially like that they've implemented it with a lightbox ("popup") so the user doesn't get as much confused anymore, as was the case in the old/standard implementation where the user is completely redirected to Google or the OpenID provider's website. Note that it is actually more "Facebook Connect" like this way! See here another example where OpenID is combined with OAuth to enable a popup login.


  • Understanding how the JVM uses native memory on Windows and Linux. The extensive article explains what native memory is, how the Java runtime uses it, what running out of it looks like (so you're not running out of heap space!), and how to debug a native OutOfMemoryError on Windows and Linux. A companion article covers the same topics for AIX systems.

  • Quite big news was of course Google's announcement of Google Wave at the Google I/O conference. It has been built with GWT. A good description can be found here. It's open source with plugin-like APIs with many integration possibilities. See the 80 minutes video for the full details. It hopes to become the replacement for email... Servers can be run by anybody. Wonder how Google is thinking of making money with it. Ads, just like in GMail? Maybe they are going to charge you for using their Wave server instances (SAAS version)? An interview with Wave's creators can be found here. And six reasons why Wave could be game-changing.

  • Eight generic best practices for scalable high performance systems.

  • Are you any of these two tools with almnost the exact same name? SonarJ is a plug-in for Eclipse that helps you validate your code against a software architecture, using static analysis (free for projects up to 500 classes). And now for the confusion: check also Sonar: enables to collect, analyze and report metrics on source code. It leverages the existing ecosystem of quality open source tools (ex. Checkstyle, PMD, Maven, Cobertura …), to offer a fully integrated solution to development environments and continuous integration tools.