Tech Team Lead News: cassandra

Showing posts with label cassandra. Show all posts

Wednesday, August 9, 2017

Cassandra Performance Tips - Batch inserts, Materialized views, Fallback strategy

Introduction

During a recent project we ran into multiple issues with Cassandra's performance. For example with queries being slow or having timeouts on only a specific environment (though they should have the same setup), inconsistently stored results, and how to optimize batch inserts when using Scala.

This blogpost describes how they were solved or attempted to be solved.

Setup: Cassandra running in a cluster with three nodes.

Performace related lessons learned

On certain environments of the DTAP build-street slow queries (taking seconds) and weird consistency results appeared. Not all 4 environments were the same, though Acceptance and Production where as much the same as possible.

We found as causes:

Slow queries and timeouts: Cassandra driver was logging at both OS and driver level
Inconsistently stored results: The clocks from different clients accessing C* were not the same, some off for minutes. Since the default in v3 of the Datastax/Cassandra driver protocol is clientside generated timestamps, you can get in trouble of course, since then the one with the most recent timestamp just always wins. But implementing serverside also won't be obvious, since different C* coordinators can give a millisec different timestamp.
For Gatling performance tests written in Scala, we first needed inserting 50K records in a Cassandra database, simulating users already registered to the system. Trying to make this perform several options were tried:

a- Plain string concatenated or prepared statements where taking over 5mins in total
b- Inserting as a batch (apply batch) has limit of 50KiB in text size. That limit is too low for us: 50K records is almost 5MB. Splitting up was too much of a hassle.
c- Making the calls async, as done here: https://github.com/afiskon/scala-cassandra-example
But we were getting:

17:20:57.275 [ERROR] [pool-1-thread-13] TestSimulation - ERROR while inserting row nr 13007, exception =
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /172.20.25.101:9042 (com.datastax.driver.core.exceptions.BusyPoolException: [/172.20.25.101] Pool is busy (no available connection and the queue has reached its max size 256)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:211)
at com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:46)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:275)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution$1.onFailure(RequestHandler.java:336)
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at cle.common.util.concurrent.Futures$ImmediateFuture.addListener(Futures.java:102)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1184)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1120)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.query(RequestHandler.java:295)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:272)
at com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:115)
at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:95)
at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:132)
at UserDAO.insert(UserDao.scala:58)
...

Turns out it is the driver's local acquisition queue that fills up. You can increase it via poolingOptions.setMaxQueueSize, see: http://docs.datastax.com/en/developer/java-driver/3.1/manual/pooling/#acquisition-queue
We set it to 50000 to be safe it would just queue all records (50K). For a production environment this might not be a good idea of course, you might need to tune it to your needs.
And the threads we set to 20 in the Executioncontext (used by the DAO from the example from github above). You can set it as in this example: http://stackoverflow.com/questions/15285284/how-to-configure-a-fine-tuned-thread-pool-for-futures
Increasing CPU from 4 to 8 did seem to improve performance, less CPU saturation.
Each time adding one more materialized view increases insert performance by 10% (see here)
For consistency and availability when one of the nodes might be gone or unreachable due to network problems, we setup Cassandra write such that first EACH_QUORUM is tried, then if fails, LOCAL_QUORUM as fallback strategy.

Below articles did help to analyse the problems further:

Wednesday, April 12, 2017

Lessons learned Docker microservices architecture with Spring Boot

Introduction

During my last project consisting of a Docker microservices architecture, built with Spring Boot, using RabbitMQ as communication channel, I learned a bunch of lessons, here's a summary of them.

Architecture

Below is a high level overview of the architecture that was used.

Docker

Run 1 process/service/application per docker container (or put stuff in init.d but that's not intended use of docker)

Starting background processes in the CMD cause container to exit. So either have a script waiting at the end (e.g tail -f /dev/null) or keep the process (i.e the one prefixed with CMD) running in the foreground. Other useful Dockerfile tips you can find here

As far as I can tell Docker checks if Dockerfile has changed, and if so, creates a new image instance (diffs only?)

Basic example to start RabbitMq docker image, as used in the build tool:

$ docker pull 172.18.19.20/project/rabbitmq:latest
$ docker rm -f build-server-rabbitmq
$ # Map the RabbitMQ regular and console ports
$ docker run -d -p 5672:5672 -p 15672:15672 --name build-server-rabbitmq 172.18.19.20/rabbitmq:latest

If there's no docker0 interface (check by running command ifconfig) then probably there are ^M characters in the config file at /etc/default/docker/docker.config. To fix it, perform a dos2unix on that file.

Check for errors at startup of docker in /var/log/upstart/docker.log

If your docker push <image> asks for a login (and you don't expect that) or it returns some weird html like "</html>" then you're probably missing the host in front of the image name, e.g: 172.18.19.20:6000/projectname/some-service:latest

Stuff like /var/log/messages is not visible in a Docker container, but is in its host! So look there for example to find out why a process is not starting/gets killed at startup without any useful logging (like we had with clamd)

How to remove old dangling unused docker images: docker rmi $(docker images --filter "dangling=true" -q --no-trunc)

Spring Boot

Some jackson2 dependencies were missing from the generated Spring Initializr project, noticed when creating unittests. These dependencies were additionally needed in scope test:

<dependency>
  <groupid>com.fasterxml.jackson.core</groupid>
  <artifactid>jackson-databind</artifactid>
  <version>2.5.0</version></dependency>
<dependency>
  <groupid>com.fasterxml.jackson.core</groupid>
  <artifactid>jackson-annotations</artifactid>
  <version>2.5.0</version></dependency>
<dependency>
  <groupid>com.fasterxml.jackson.core</groupid>
  <artifactid>jackson-core</artifactid>
  <version>2.5.0</version></dependency>

Not sure anymore why these then didn't get <scope>test</scope> then... Guess it was needed also in some regular code... :)

In the Spring Boot AMQP Quick Start the last param has name .with(queueName) during binding, but that's the topic key! (which is related to the binding key used at sending), so not the queue name.

Spring Boot Actuator's /health will check all related dependencies! So if you have a dependency in your pom.xml to a project which uses spring-boot-starter-amqp, /health will check now for an AMQP queue being up! So add a for those if you don't want that.

Spring Boot's default AppAplicationIT probably needs a @DirtiesContext for your tests, otherwise the tests might re-use or create more beans than you think (we saw that in our message receiver tests helper class).

@Transactional in Spring: by default only for unchecked exceptions!! It's documented but still a thing to watch out for.

And of course: Spring's @Transactional does not work on private methods (due to proxy stuff it creates)

To see in Spring Boot the transaction logging, put this in application.properties:

logging.level.org.springframework.jdbc=TRACE
logging.level.org.springframework.transaction=TRACE

Note that by default @Transactional just rolls back, it does not log anything, so if you don't log your runtime exceptions, you won't see much in your logs.

mockMvc from spring is not really invoking from "outside", our spring sec context filter (for which you can use @Secured(role)) was allowing calls while no authentication was provided for. RestTemplate seems to work from "the outside".

Scan order can mess up @ControllerAdvice error handler it seems. Had to change the order sometimes:

Setup:
- Controller is in: com.company.request.web.
- General error controller is in com.company.common package.

Had to change
@ComponentScan(value = {"com.company.security", "com.company.common", "com.company.cassandra", "com.company.module", "com.company.request"})

to

@ComponentScan(value = {"com.company.security", "com.company.cassandra", "com.company.module", "com.company.request", "com.company.common"})

Note that the general error controller has now been put in last

Spring Boot footprint seems relatively big especially for microservices. At least 500MB or something is needed, so we have quite big machines for about 20 services. Maybe plain Spring (iso Spring boot) might be more lightweight...

Bamboo build server

When Bamboo gets slow and the CPU seems quite busy and memory availability on its server seems fine, increase the Xmss and Xmsx (or related). Found this out because the java Bamboo process was running out of heap sometimes, increasing heap also fixed performance.

To have Bamboo builds fail on quality gates not met in SonarQube, install in Sonar the build breaker plugin. See the plugin docs and Update Center. This FAQ says so.

Stash

The Stash (now called Bitbucket) API: in /rest/git/1.0/projects/{projectKey}/repos/{repositorySlug}/tags a 'slug' is just a repository name.

Microservices with event based architecture

When you do microservices, IMMEDIATELY take into account during coding + reviews that multiple instances can do concurrent access to database.

This has affect on your queries. Most likely correct implementation for uniqueness check on inserts:
1- add unique constraint
2- run insert
3- catch uniqueness exception --> you know it already exists. Solution with SELECT NOT EXISTS is not guaranteed unique.

Also take deleting of data (e.g user deletes himself) into account from the start. Especially when using events and/or eventual consistency in combination with an account-balance or similar. Because what if one services in the whole chain of things to execute for a delete fails? Has the user still some money left on his/her account then? In short: take care of CRUD.

Multiple services are sending the same event? That can indicate 2 services are doing the same thing --> Not good probably.

Microservices advantages:

- Forces you to better think about where to put stuff in comparison to monolith where you more often can be tempted to "just do a quick fix".
- language independency for service implementation: choose the best language for the job

Disadvantages:
- more time needed for design
- eventual consistency is quite tough to understand & work with, also conceptually
- infrastructure is more complex including all communication between services

More cons can be found here.

Tomcat

Limit the maximum size of what can be posted to a servlet is not as easy as it seems for REST services:

- maxPostSize in Tomcat is enforced only for specific contenttype: Tomcat only enforces that limit if the content type is application/x-www-form-urlencoded

- And the other 3 below XML options are for multipart only:

<multipart-config>
  
  <max-file-size>52428800</max-file-size>
  <max-request-size>52428800</max-request-size>
  <file-size-threshold>0</file-size-threshold></multipart-config>

So that one won't work for uploading just a byte[]. The only solution is in the servlet (e.g Spring @Controller) you'll have to check for the limit you want to allow.

maxthreads seems set to be unlimited by default or something. 50 seems to perform better. (workerthreads)

Security

To securely generate a random number: SecureRandom randomGenerator = SecureRandom.getInstance("NativePRNG");

Good explanation of secure use of a salt to use for hashing can be found here.

Cassandra

Unique constraints are not possible in Cassandra, so there you will even have to implement unique constraints in the business logic (and make it eventually consistent)

CassandraOperations query for one field:

Select select = QueryBuilder.select(MultiplePaymentRequestRequesterEntityKey.ID).from(MultiplePaymentRequestRequesterEntity.TABLE_NAME);
select.setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM);
select.where(QueryBuilder.eq(MultiplePaymentRequestRequesterEntityKey.REQUESTER, requester));
return cassandraTemplate.queryForList(select, UUID.class);

See also here.

Note below two keys don't seem to get picked up by Cassandra in the Spring Data Cassandra version 1.1.4.RELEASE:

<groupid>org.springframework.data</groupid>
<artifactid>spring-data-cassandra</artifactid>

@PrimaryKeyColumn(name = OTHER_USER_ID, ordinal = 0, type = PrimaryKeyType.PARTITIONED)
@CassandraType(type = DataType.Name.UUID)
private UUID meUserId;

@PrimaryKeyColumn(name = ME_USER_ID, ordinal = 1, type = PrimaryKeyType.CLUSTERED)
@CassandraType(type = DataType.Name.UUID)
private UUID meId;
This *does* get picked up: put it into a separate class:

@Data
@AllArgsConstructor
@PrimaryKeyClass
public class HistoryKey implements Serializable {

@PrimaryKeyColumn(name = HistoryEntity.ME_USER_ID, ordinal = 0, type = PrimaryKeyType.PARTITIONED)
@CassandraType(type = DataType.Name.UUID)
private UUID meUserId;

@PrimaryKeyColumn(name = HistoryEntity.OTHER_USER_ID, ordinal = 1, type = PrimaryKeyType.PARTITIONED)
@CassandraType(type = DataType.Name.UUID)
private UUID otherUserId;

@PrimaryKeyColumn(name = HistoryEntity.CREATED, ordinal = 2, type = PrimaryKeyType.CLUSTERED, ordering = Ordering.DESCENDING)
private Date created;

}

Don't use Cassandra for all types of use cases. An RDMS still has its value, e.g for ACID requirements. Cassandra is eventually consistent.

Kubernetes

See my diagram in this post.

Miscellaneous

Use dig for DNS resolving problems

Use pgAdmin III for PostgreSQL GUI

To stop SonarQube complaining about unused private fields when using Lombok @Data annotation: add to each of those classes @SuppressWarnings("PMD.UnusedPrivateField")

Managed to not need transactions nor XA transactions for message publishing, message reading, store db, message sending, by using the confirm + ack mechanism.
And allow message to be read again. DB then sees: oh already stored (or do an upsert).
So,when processing message from the queue:
1- store in db
2- send message on queue
3- only then ack back to queue that read was successful

Performance: instantiate the Jackson2 ObjectMapper once as static, not in each call, so:
```
private static final ObjectMapper mapper = new ObjectMapper();
```
Javascript: when an exception occurs in a callback and it is not handled, processing just ends. Promises have better error handling.

clamd would not start correctly; it would try to start but then show 'Killed' when started via the commandline. Turns out it runs out of memory when starting up. Though we had enough RAM (16G total, 3G free), it turns out clamd needs swap configured!

Linux bash shell script to loop through projects for tagging with projects with spaces in their name:

PROJECTS="
  project1
  project space2
";
IFS=$'\n'
for PROJECT in $PROJECTS
do
  TRIM_LEADING_SPACE_PROJECT="$(echo -e "${PROJECT}" | sed -e 's/^[[:space:]]*//')"
  echo "Cloning '$TRIM_LEADING_SPACE_PROJECT'"
  git clone --depth=1 http://$USER:$GITPASSWD@github.com/projects/$TRIM_LEADING_SPACE_PROJECT.git
done

OpenVPN in Windows 10: Sometimes it hangs on "Connecting..." It doesn't show the popup to enter username/pwd. Go to View logs. Then when you see: Enter management password in the logs: ???? you have to kill the OpenVPN Daemon under Processes tab (windows taskmanager). The service is stopped when exiting the app but that's not enough!

javascript/nodejs log every call that comes in:

app.use(function (req, res, next) {
console.log('Incoming request = ' + new Date(), req.method, req.url);
logger.debug('log at debug level');
next()
}

If ever your mouse is suddenly not working anymore your VirtualBox guest, kill the process in your guest-machine mentioned in comment 5 here. After that the mouse works again in your vbox guest.

Fix Firefox to version 45.0.0 for selenium driver tests:

sudo apt-get install -y firefox=45.0.2+build1-0ubuntu1
sudo apt-mark hold firefox

Setting the cookie attribute Secure (indicating cookie should only be sent over httpS) can be seen when using curl to request the URL(s) that should send that cookie plus the new attribute, even when using HTTP. See also my previous post.

But when using a browser and HTTP, you probably won't see the secure cookie appear in the cookie store. This is (probably) because the browser knows not to store it in that case because it's HTTP being used.

Idempotency within services is key for resilience and be able to resend an event or perform an API call again.

Wednesday, December 9, 2015

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.255.235.17 (Timeout during read)

In a recent project, seemingly randomly, this exception occurred when doing a CQL 'select' statement from a Spring Boot project to Cassandra:

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.255.235.17 (Timeout during read), /10.255.235.16 (Timeout during read))
...

After a lot of research, some people seemed to have reported the same issue. But no clear answer anywhere. Except that some Cassandra driver versions might be the cause of it: they mark (all) the node(s) as down and don't recognize it when it becomes available again.

But, the strange this is we have over 10 (micro) services running, all running at with least 2 instances. But only one of these services had this timeout problem. So it almost couldn't be the driver.... Though it did seem to be related with not using the connection for a while, because often our end-to-end tests just ran fine, time after time. But after a few hours, the tests would just fail. Then we didn't see the pattern yet...

But, as a test, we decided to let nobody use the environment against which the end-to-end tests run for a few hours; especially also because some of the below articles do mention as a solution to set the heartbeat (keep-alive) of the driver.

And indeed, the end-to-end tests started failing again after the grace period. Then we realized it: all our services have a Spring Boot health-check implemented, which is called every X seconds. EXCEPT the service that has the timeouts; it only recently got connected with Cassandra!

After fixing that, the error disappeared! Of course depending on the healthcheck for a connection staying alive is not the ideal solution. A better solution is probably setting the heartbeat interval on the driver on Cluster creation:

var poolingOptions = new PoolingOptions()
.SetCoreConnectionsPerHost(1)
.SetHeartBeatInterval(10000);
var cluster = Cluster
.Builder()
.AddContactPoints(hosts).
.WithPoolingOptions(poolingOptions)
.Build();

In the end it was the firewall which resets all TCP connections every two hours!

References

Tips to analyse the problem:

Log at lower levels: log4j.logger.com.datastax.driver.core=TRACE or DEBUG

Call getErrors(): http://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/exceptions/NoHostAvailableException.html

As reported by somebody else, the results of that call gave us just an empty list.

Tips to see more, like which host is up/down in the Cassandra cluster: https://groups.google.com/a/lists.datastax.com/forum/#!search/nohostavailableexception/java-driver-user/WoqUJrqTm98/KidIFkNdN1MJ

You can potentially see when the driver lost the connection by looking for a log statement like:

2014-12-12 16:36:55,843{UTC} [Reconnection-0] ERROR c.d.driver.core.ControlConnection - [Control connection] Cannot connect to any host, scheduling retry in 600000 milliseconds

Make also sure you are re-using the Session object; it's expensive to create, and maybe you are using up your connection pool due to too many Session objects: http://stackoverflow.com/questions/25145980/datastax-cassandra-java-driver-crashes-with-nohostavailableexception-after-a-few

Pooling options: http://stackoverflow.com/questions/24821966/nohostavailableexception-with-1000-concurrent-request-to-cassandra-with-datastax?rq=1

Connection pool problems, connections stay ESTABLISHED: https://groups.google.com/a/lists.datastax.com/forum/#!searchin/java-driver-user/nohostavailableexception$20timeout/java-driver-user/fbLFX2_uI7w/O8sQSa6XXj4J

Force limiting the query size at driver level when your query just takes too long: http://stackoverflow.com/questions/19528754/nohostavailableexception-with-cassandra-datastax-java-driver-if-large-resultse?rq=1

Setting a very low value for SocketOptions.setReadTimeoutMillis could be a triggering factor for this bug. If you changed this value, make sure it is greater than the server-side timeouts in cassandra.yaml

Try to connect to Cassandra with cqlsh from the host that's giving the timeout. If that's connecting fine, it might be a faulty driver, not correctly detecting the node is up (again).

As mentioned earlier, it might be a bad driver version. Many reports found of people seeing the error occur when upgrading the driver.

Huge batches can cause driver to mark node as down: "It turned out that it was related to the query pattern in that particular client. We found that in certain cases that client would try to write a HUGE batch, which would not complete under driver side default timeout settings. That would cause driver to think that the node is down, and eventually it will end up marking all nodes as down. Setting a limit on the batch size (by chunking) and using UNLOGGED batches seems to have solved the issue."

Operation timed out: Indicates a time out for your request on that host. This error means the host did not complete the query within SocketOptions#getReadTimeoutMillis(). Since you are not explicitly configuring this that means a query is not completing in 12 seconds (the default).

"A host can be made unavailable if a query times out on it, if we receive a 'DOWN' status event from Cassandra, or connection is lost to the Host. Once you understand the cause of your hosts being marked down, the next step is to see if you can mitigate it." Bug report: https://datastax-oss.atlassian.net/browse/JAVA-577

This was pointing in the right direction for us: Too long idle state? Keep alive? Firewall? https://datastax-oss.atlassian.net/browse/JAVA-204

Rolling restart not handled correctly in older driver: https://datastax-oss.atlassian.net/browse/JAVA-250 and https://datastax-oss.atlassian.net/browse/JAVA-367

"Connectiontimeout is by default 5 seconds: You can control the maximum time the driver will try connecting (to each node) through SocketOptions.setConnectTimeoutMillis() (the default is 5 seconds). The timeout above is per host. So if you pass a list of 100 contact points, you could in theory have to wait 500 seconds (by default) before getting the NoHostAvailableException. But there is no real point in providing that many contact points, and in practice, if Cassandra is not running on the node tried, the connection attempt will usually fail right away (you won't wait the timeout)."

SELECT IN is bad, but we don't do that. Though indeed ORDER BY is sub-optimal too: https://groups.google.com/a/lists.datastax.com/forum/#!search/nohostavailableexception/java-driver-user/b076HRgEfoo/s-kuLbhSAzMJ

"Are you doing cross DC writes? When i stopped the cross DC writes , the problem stopped occurring for me. I think the timeout is mainly due to driven lag."

Similar error reports

Sunday, March 21, 2010

Best of this Week Summary 15 March - 21 March 2010

Twitter and Digg are moving from MySQL to Cassandra (a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model; a Facebook opensourced project). The reason for Digg for the move "is the increasing difficulty of building a high-performance, write-intensive application on a data set that is growing quickly, with no end in sight. This growth has forced them into horizontal and vertical partitioning strategies that have eliminated most of the value of a relational database, while still incurring all the overhead".
Twitter has about the same reason: "No single points of failure", "Highly scalable writes (we have highly variable write traffic)", and "A healthy and productive open source community".
Twitter tried HBase, Voldemort, MongoDB, MemcacheDB, Redis, Cassandra, and HyperTable amongst others before deciding to go with Cassandra. Interesting to read is how they slowly rollout Cassandra to limited sets of users.
An introduction to Cassandra to get it up and running can be found here.

A short post on how to implement Automatic testing Oracle Service Bus using Hudson, Maven and SoapUI.

What to expect from HTML5 for webdevelopers.

Tech Team Lead News

Wednesday, August 9, 2017

Cassandra Performance Tips - Batch inserts, Materialized views, Fallback strategy

Introduction

Performace related lessons learned

Wednesday, April 12, 2017

Lessons learned Docker microservices architecture with Spring Boot

Introduction

Architecture

Docker

Spring Boot

Bamboo build server

Stash

Microservices with event based architecture

Tomcat

Security

Cassandra

Kubernetes

Miscellaneous

Wednesday, December 9, 2015

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.255.235.17 (Timeout during read)

References

Similar error reports

Sunday, March 21, 2010

Best of this Week Summary 15 March - 21 March 2010

About Me

Subscribe via RSS

Subscribe via email

Twitter Follow Me

Useful Links

Total Pageviews

Live Traffic Map

Blog Archive