Posts Tagged ‘java’

Attach payload into detached pkcs#7 signature

If you are doing signature generation via a hardware token (for instance 3Skey) then, for large files it is impractical to send the file to the hardware token. Instead you send a hash (SHA256), get a detached PKCS#7 signature and you need to re-attach the payload in java code. For once this was easier to do with plain JCE code instead of my favorite BouncyCastle provider. However for really large files BC does provide the streaming mechanism required.

Of course the best commands to use to help debug the code bellow are:

Verify pkcs#7 signature

#the -noverify means do not verify the certificate chain, this will only verify the signature not the originating certificate
openssl smime -inform DER -verify -noverify -in signature.p7s

(more…)

An “obvious” improvement

It’s been a long time since I felt such satisfaction debuging something so I decided to write about it.

Let’s assume that you need to store (cache) in memory a large object tree during some operations. In practice this happens because some regulatory constraints so you end up having to parse a very large file and store the resulting object tree. Actually you have a single entry cache. You parse your object, store it in memory for search and processing while the current object tree is used.

public ObjectHandler getObjectHandler(Long id) throws Exception{
	if(cachedObjectHandler != null){
		if(cachedObjectHandler.getId().equals(id)){
			return cachedObjectHandler;
		}
	}
	//else
	cachedObjectHandler = parse(...)
	return cachedObjectHandler;
}

The code above is a simplified way to do it, no? Please note that the parse(…) function creates the object tree by parsing a stream and allocates a new object tree. In my particular case the object tree holded a max of 120k objects (~150Mb) and did some large xml parsing using stax.

So what is wrong with the code above? Take a look at what a single change can do:

public ObjectHandler getObjectHandler(Long id) throws Exception{
	if(cachedObjectHandler != null){
		if(cachedObjectHandler.getId().equals(id)){
			return cachedObjectHandler;
		}
	}
	//else
	cachedObjectHandler = null;
	cachedObjectHandler = parse(...)
	return cachedObjectHandler;
}

Did we just reduced the max needed memory by 2? In the first case since java does right to left assignment first a new object tree is allocated by the parse function and only when done it is assigned to the cachedObjectHandler object allowing for the old object tree to be gc-ed. However with the null assignment it can be gc-ed while the new allocation takes place if memory is needed.

As I said, a small change with a big smile.
 

Java SAML2 + simplesamlphp

The use case is as follows: the java application (SP) must use simplesamlphp as an IdP. I tested 2 libraries, these are the required configs.

SimpleSAMLphp

Please note that the default install from ubuntu (16.04.2) of simplesamlphp (14.0) does not work with the php version installed (php7) because of this bug so I ended installing everything from the tar.gz provided (14.14).

Onelogin

This is the first library I tested. To install it:

  • install maven, requires recent version, does not work with 3.0.5
  • export MAVEN_HOME=/usr/local/java/apache-maven-3.5.0
  • export PATH=$MAVEN_HOME/bin:$PATH
  • git clone https://github.com/onelogin/java-saml
  • cd java-saml
  • mvn package
  • download tomcat 7.0.78
  • install java-saml-toolkit-jspsample as a expanded war in this tomcat
  • tweaked the files: onelogin.saml.properties and the simplesamlphp config until it worked. The key is to use the information from the IdP metadata (http://idp-domain/simplesamlphp/saml2/idp/metadata.php?output=xhtml) and transpose it in the properties file.

Pac4j

This is a more complex library. There is also a demo application for j2e.

To clone, compile and run this demo the sequence is straight forward:

  • git clone https://github.com/pac4j/j2e-pac4j-demo
  • cd j2e-pac4j-demo
  • mvn package
  • mvn jetty:run

The test with https://www.testshib.org works.

To configure if for simpleSAMLphp. Modify DemoConfigFactory.java

final SAML2ClientConfiguration cfg = new SAML2ClientConfiguration("resource:samlKeystore.jks",
 "pac4j-demo-passwd",
 "pac4j-demo-passwd",
 "resource:idp-metadata.xml");
 cfg.setMaximumAuthenticationLifetime(3600);
 cfg.setServiceProviderEntityId("test.pac4j");
 cfg.setServiceProviderMetadataPath(new File("sp-metadata.xml").getAbsolutePath());
 final SAML2Client saml2Client = new SAML2Client(cfg);

The idp-metadata.xml file is the file from: http://idp-domain/simplesamlphp/saml2/idp/metadata.php?output=xhtml wrapped in an additional EntitiesDescriptors element:

<md:EntitiesDescriptor xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata" xmlns:ds="http://www.w3.org/2000/09/xmldsig#"> 
 <md:EntityDescriptor entityID="http://idp-domain/simplesamlphp/saml2/idp/metadata.php">

However at this point the application gives a “fatal error”:

org.pac4j.saml.exceptions.SAMLException: Identity provider has no single sign on service available for the selected profileorg.opensaml.saml.saml2.metadata.impl.IDPSSODescriptorImpl@2d6719d3

The error seems to be here https://github.com/pac4j/pac4j/blob/master/pac4j-saml/src/main/java/org/pac4j/saml/context/SAML2MessageContext.java#L104 so I am left with no clue to the problem. The only solution is to change a bit the code to see which is the binding required.

Just cloning the main repository and trying to compile it with maven does not work. The error is:

[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.pac4j:pac4j-couch:[unknown-version]: Could not find artifact org.pac4j:pac4j:pom:2.0.0-RC3-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 5, column 10
@ 
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR] 
[ERROR] The project org.pac4j:pac4j-couch:[unknown-version] (/phantom/java/pac4j/pac4j-couch/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM for org.pac4j:pac4j-couch:[unknown-version]: Could not find artifact org.pac4j:pac4j:pom:2.0.0-RC3-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 5, column 10 -> [Help 2]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

The solution is to checkout the 2.0.0 tag:

  • git clone https://github.com/pac4j/pac4j
  • git tag -l
  • git checkout tags/pac4j-2.0.0

At this point I change the code to give the name of the binding:

org.pac4j.saml.exceptions.SAMLException: Identity provider has no single sign on service available for the selected profileurn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST

The solution is to modify the metadata/saml20-idp-hosted.php file and add:

'SingleSignOnServiceBinding' => array('urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect', 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST'),
'SingleLogoutServiceBinding' => array('urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect', 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST'),

This will generate the

<md:SingleLogoutService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"

in the metadata which generated this error.

At this point the SSO works. Of course the entityID for the SP must be configured in metadata/saml20-sp-remote.php

$metadata['diapason.test.pac4j'] = array(
 'AssertionConsumerService' => 'http://localhost:8080/callback?client_name=SAML2Client',

Read fast or die

I have spend a lot of time today trying to find and fix an issue which ended up to be a fun discovery at the end.

The following java error occurred when loading a pdf file from an url stream:

java.io.IOException: missing CR
at sun.net.www.http.ChunkedInputStream.processRaw(ChunkedInputStream.java:405)
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:572)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:609)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:696)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3066)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3060)

This looked like a java lib error since java version was a bit old so the first idea was to replace the code with some apache httpClient based code to load the URL. This generated the following error, very similar.

java.io.IOException: CRLF expected at end of chunk: 121/79
at org.apache.commons.httpclient.ChunkedInputStream.readCRLF(Unknown Source)
at org.apache.commons.httpclient.ChunkedInputStream.nextChunk(Unknown Source)
at org.apache.commons.httpclient.ChunkedInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at org.apache.commons.httpclient.AutoCloseInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.commons.httpclient.AutoCloseInputStream.read(Unknown Source)

Since this was a windows machine and the requests passed via localhost another try was to use another interface. The result was similar.

After some search I found a nice tool: http://www.netresec.com/?page=RawCap which does not requires any install and can be used even on localhost to generate a pcap compatible file which can then be inspected in wireshark.

The result was strange. Opening the capture in wireshark on my machine showed: [7540 bytes missing in capture file] in the tcp stream. This corresponded to a lot of packets: [TCP ZeroWindow], [TCP ZeroWindowProbe].

Selection_655

Since this was a VMWare install and I previously had some trouble with vmware software switches I assumed this was related to the network card config however it also happened on localhost.

After some more investigation I realized this was only happening when several request where made in parallel. I confirmed this by looking in the code.

The code contained a worker pool. Each worker/thread constructed the url the used then returned an inputStream.

DataSource source = new URLDataSource(reportUrl);
return source.getInputStream();

However all the results were handled in sequence. As such while one url inputStream was read, the server continued to send data on all others request but this data was not read on the client side fast enough. As a result the tcp window got indeed to 0 and the strange error occurred.

Of course the solution was to fully read the data in each worker.

The difficult to find bug

After upgrading from 3.4 to 3.6 my JGroups code stopped working. On a 2 nodes setup when the second node tried to join I got the following errors:

2015-04-22 21:38:12,510 INFO  [ViewHandler,monalisa,tux-7762|fr.mcc.test.TestJGroups] Detected a change in cluster members: [tux-7762, tux-9846]
2015-04-22 21:38:14,460 WARN  [main|org.jgroups.protocols.pbcast.GMS] tux-9846: JOIN(tux-9846) sent to tux-7762 timed out (after 2000 ms), on try 1
2015-04-22 21:38:16,463 WARN  [main|org.jgroups.protocols.pbcast.GMS] tux-9846: JOIN(tux-9846) sent to tux-7762 timed out (after 2000 ms), on try 2
2015-04-22 21:38:18,466 WARN  [main|org.jgroups.protocols.pbcast.GMS] tux-9846: JOIN(tux-9846) sent to tux-7762 timed out (after 2000 ms), on try 3
2015-04-22 21:38:20,468 WARN  [main|org.jgroups.protocols.pbcast.GMS] tux-9846: JOIN(tux-9846) sent to tux-7762 timed out (after 2000 ms), on try 4
2015-04-22 21:38:22,471 WARN  [main|org.jgroups.protocols.pbcast.GMS] tux-9846: JOIN(tux-9846) sent to tux-7762 timed out (after 2000 ms), on try 5
2015-04-22 21:38:24,473 WARN  [main|org.jgroups.protocols.pbcast.GMS] tux-9846: JOIN(tux-9846) sent to tux-7762 timed out (after 2000 ms), on try 6
2015-04-22 21:38:26,475 WARN  [main|org.jgroups.protocols.pbcast.GMS] tux-9846: JOIN(tux-9846) sent to tux-7762 timed out (after 2000 ms), on try 7
2015-04-22 21:38:28,479 WARN  [main|org.jgroups.protocols.pbcast.GMS] tux-9846: JOIN(tux-9846) sent to tux-7762 timed out (after 2000 ms), on try 8
2015-04-22 21:38:30,482 WARN  [main|org.jgroups.protocols.pbcast.GMS] tux-9846: JOIN(tux-9846) sent to tux-7762 timed out (after 2000 ms), on try 9
2015-04-22 21:38:32,485 WARN  [main|org.jgroups.protocols.pbcast.GMS] tux-9846: JOIN(tux-9846) sent to tux-7762 timed out (after 2000 ms), on try 10
2015-04-22 21:38:32,485 WARN  [main|org.jgroups.protocols.pbcast.GMS] tux-9846: too many JOIN attempts (10): becoming singleton
2015-04-22 21:38:32,493 WARN  [Incoming-2,monalisa,tux-9846|org.jgroups.protocols.pbcast.NAKACK2] JGRP000011: tux-9846: dropped message 1 from non-member tux-7762 (view=[tux-9846|0] (1) [tux-9846])

It took some time but finally I managed to write a “simple” (quick and ugly) test to replicate the problem.
(more…)

ojdbc14.jar to ojdbc6.jar migration

Migrating from ojdbc14.jar to ojdbc6.jar is not, as one might think, completely seamless. Here are at least 3 points which required code change:

Changes in class hierarchy

error: OracleTypes is not public in oracle.jdbc.driver; cannot be accessed from outside package
 [javac] private static Integer OracleRefCursorType = new Integer(oracle.jdbc.driver.OracleTypes.CURSOR);

(more…)

Replicated EhCache, the uneasy road

At first replicating EhCache seems a very easy task, just need to configure ehcache.xml with RMI and you are ready. Is it so?

(more…)

Database locking

This is a very simple example demonstrating a method to achieve a reliable lock in a cluster when the cluster shares a database connection. This example uses an Oracle database which has the following table/row:

CREATE TABLE locked(shortname varchar2(100) PRIMARY KEY, info varchar2(100));
INSERT INTO locked VALUES('TestLock', '');
commit;

(more…)

Let’s decrypt

AES encrypt in java and decrypt in java, flex, python, C#.

Encrypt: java

public static void encrypt(InputStream is, OutputStream out, String secret) throws Exception {
	SecretKey secretKey = new SecretKeySpec(Hex.decodeHex(secret.toCharArray()), "AES");
        Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
        cipher.init(Cipher.ENCRYPT_MODE, secretKey);
        out.write(cipher.getParameters().getParameterSpec(IvParameterSpec.class).getIV());
        CipherOutputStream cipherOutputStream = new CipherOutputStream(out, cipher);
        int bufLength = KEY_LENGTH/8 * 100;
        byte buf[] = new byte[bufLength];
        int bytesRead = 0;
	while((bytesRead = is.read(buf)) != -1 ) {
		cipherOutputStream.write(buf, 0, bytesRead);
	}
	cipherOutputStream.close();
}

(more…)

JBoss migration – Quartz

This is a continuation of the previous article regarding some migration points (1, 2) from JBoss 4.2.2-GA to JBoss 7.1.1 and, presumably, Tomcat 7.

3. Quartz

Quartz migration has been the simplest of all, by far.
(more…)