Monday, August 28, 2017

Solaris 10 - fiocompress (UFS file compression) settings

Bernd Schemmer has an interesting post about using fiocompress for file-system level compression of individual files on UFS file systems...

I did some experimentation and found a few more things:

  • Increasing the blocksize from the default of 8192 increases the compression ratio
  • The compression ratio seems to be somewhere between gzip and compress (on a text file)
  • Setting the blocksize to 65536 (64KiB) results in an unreadable and undeletable file (at least with normal tools on test system. This is fixed in the latest recommended patch bundle)
  • Using blocksizes below 8192 also results in unusable files. (I only tested multiples of 2)(This is fixed in the latest recommended patch bundle)
  • fiocompress uses an ioctl call to mark a file as compressed if -m is passed. No method to unmark a marked file exists, even in the filesystem driver. (It is possible to modify the OpenSolaris fiocompress to add an option to just mark (a previously compressed) file as compressed) (Look at ufs_vnops.c for the _FIO_COMPRESSED ioctl implementation)


Test compression code:
ls -lah testfile.txt; du testfile.txt; du -h testfile.txt; for b in 256 512 1024 2048 4096 8192 16384 32768 65536; do fiocompress -b $b -c -m testfile.txt testfile.txt$b; done

Results: (including other common formats)
$ls -lah testfile.txt*
testfile.txt1024: Operation not applicable
testfile.txt2048: Operation not applicable
testfile.txt256: Operation not applicable
testfile.txt4096: Operation not applicable
testfile.txt512: Operation not applicable
testfile.txt65536: Operation not applicable
-rw-r--r--   1 user   group      101M Mar 11 10:26 testfile.txt
-rw-------   1 user   group      4.4M Mar 11 10:29 testfile.txt.7z
-rw-r--r--   1 user   group      5.2M Mar 11 10:26 testfile.txt.bz2
-rw-r--r--   1 user   group      7.2M Mar 11 10:28 testfile.txt.gzip
-rw-r--r--   1 user   group      8.9M Mar 11 10:28 testfile.txt.gzip-1
-rw-r--r--   1 user   group      6.7M Mar 11 10:28 testfile.txt.gzip-9
-rw-r--r--   1 user   group       13M Mar 11 10:27 testfile.txt.Z
-rw-r--r--   1 user   group      7.2M Mar 11 10:32 testfile.txt.zip
-rw-r--r--   1 user   group      101M Mar 11 10:55 testfile.txt16384
-rw-r--r--   1 user   group      101M Mar 11 10:55 testfile.txt32768
-rw-r--r--   1 user   group      101M Mar 11 10:55 testfile.txt8192

$du testfile.txt*
206544  testfile.txt
9040    testfile.txt.7z
10624   testfile.txt.bz2
14848   testfile.txt.gzip
18240   testfile.txt.gzip-1
13840   testfile.txt.gzip-9
27632   testfile.txt.Z
14848   testfile.txt.zip
18784   testfile.txt16384
16480   testfile.txt32768
22656   testfile.txt8192
$du -h testfile.txt*
 101M   testfile.txt
 4.4M   testfile.txt.7z
 5.2M   testfile.txt.bz2
 7.2M   testfile.txt.gzip
 8.9M   testfile.txt.gzip-1
 6.8M   testfile.txt.gzip-9
  13M   testfile.txt.Z
 7.2M   testfile.txt.zip
 9.2M   testfile.txt16384
 8.0M   testfile.txt32768
  11M   testfile.txt8192


eFiling and eHomeAffairs in Chrome

Google bundles Flash with Chorme (making it the only option for some things on GNU/Linux), however they have recently started phasing out Flash. As part of that, Chrome hides the presence of Flash to websites, but gives users an option to enable Flash on the site if the page attempts to use Flash. eHomeAffairs and SARS eFiling gives an error when Flash is not detected and doesn't attempt to load the content anyway, which means that the "Click to run" option does not work.

Recommeneded method: MyBroadband documented one method to get eFiling working.. For eHomeAffairs, the address to add to the list is "https://ehome.dha.gov.za".

Alternative, works on many more sites: Another option is to configure Chrome not to hide Flash from websites. This can be done by visiting "chrome://flags" in the address bar and setting the "Prefer HTML5 over Flash" setting to "Disabled" ("chrome://flags/#prefer-html-over-flash" will take you directly to the setting). You need to restart Chrome for the setting to take effect. The content will then load. (Tested on Chrome 60). On some sites, the Flash content may still be click-to-run, however Chrome seems to currently run it automatically on both eHomeAffairs and SARS eFiling when this flag is set. (Chrome will attempt to detect important Flash content and enable that automatically)

Update: The chrome://flags method stopped working in Chrome 61. Adding the site to content settings as being allowed to run Flash as per the MyBroadband article still works.

Friday, February 24, 2017

OpenSSL cipher suite without forward secrecy

Firstly, you should not use this in normal use.

Sometimes, you might need to debug a problem that occurs behind TLS.

Wireshark can decode TLS traffic, given the session keys, or if forward secrecy ciphers was not used, the private key.

In the case of web traffic, SSLKEYLOGFILE can tell NSS, used by some browsers to log the keys. This is a better method than the one described here, but it is not an option if other clients are used, say in the case of SMTP.

An (OpenSSL) ciphersuite setting that excludes ciphers providing forward secrecy, while keeping strong ciphers is:
HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4:!DHE:!ECDHE:!EDH:!EECDH

This should be avoided in production and should only be used for debugging.

Wednesday, November 9, 2016

Handling messages forwarded as attachment by Outlook with MIME::Parser in Perl

Outlook sends emails that are forwarded as attachment with an .eml extension and the content-type set to application/octet-stream. According to RFC 1341, message/rfc822 should be used. The Perl module, MIME::Parser will automatically parse message/rfc822 attachments, which is useful if you want to do automated processing on an email and its attachments. Outlook's use of application/octet-stream breaks this.

It is possible to fix this. I initially attempted to change the contetn-type and rerun the parser on the file, but that resulted in an empty part. The problem is that according to RFC1341, the Content-Transfer-Encoding field must be 7bit, 8bit or binary for message/rfc822 (Outlook uses base64). Once this is corrected, it works.

A Perl sample: (in this case, the email forwarded as attachment is the second attachment)
#!/usr/bin/perl

use warnings qw(all);
use MIME::Parser;
use strict;

my $fn = '/tmp/input_file.eml';

my $parser = new MIME::Parser;

$parser->output_to_core(1); # Disable the creation of temporary files

my $entity = $parser->parse_open($fn);
$entity->dump_skeleton;   # View initial structure

# Fix the fields
$entity->parts(1)->head->replace('Content-Type','message/rfc822');
$entity->parts(1)->head->replace('Content-Transfer-Encoding','7bit');

# Get encoded message
my $message = $entity->as_string;
#Re-parse
$entity = $parser->parse_data($message);

$entity->dump_skeleton;          # show final structure

\

Here is a general function to handle these. It uses undocumented interfaces, since there does not seem to be a documented method to replace a part with another one.
sub handle_forwarded_messages
{
   my($parser,$entity, undef) = @_;
   return undef unless ($entity && $parser);

   my($part);

   # Recursively process multipart entities, based on number of parts
   if (scalar $entity->parts) # If we have sub-parts
   {
      # Warning, next line uses undocumented interfaces..
      for (my $i = 0; $i <= $#{$entity->{ME_Parts}}; $i++) {
         $part = $entity->{ME_Parts}[$i];
         # Warning, next code line uses undocumented interfaces..
         # Replace part with its expanded version... This seems to be the only way
         $entity->{ME_Parts}[$i] = &handle_forwarded_messages($parser,$part);
      }
   } else { # Once we are at a level that does not have sub-parts...
      # Replace forwarded messages with properly expanded versions...
      if ($entity->effective_type eq 'application/octet-stream' &&
              $entity->head->recommended_filename =~ /\.eml$/) {
          $entity->head->replace('Content-Type','message/rfc822');
          $entity->head->replace('Content-Transfer-Encoding','8bit');
          my $entity_tmp = eval { $parser->parse_data($entity->as_string) };
          $entity = $entity_tmp unless ($@ || $parser->results->errors);
          # And see if they have more levels...
          $entity = &handle_forwarded_messages($parser,$entity);
      }
   }
   # Return the processed result
   return $entity;
}

Wednesday, June 22, 2016

Marking files on UFS as compressed on Solaris

Bernd Schemmer has an interesting post about using fiocompress for file-system level compression of individual files on UFS file systems...

However, it might be useful to mark files as compressed after compressing the file, such as when you forgot the "-m" option when compressing a large file.

The fiocompress utility does this by calling the _FIO_COMPRESSED ioctl on the file.

There seems to be no way to unmark a file that is marked as compressed. (The ioctl sets a cflag on the file called ICOMPRESS, but no operation to clear the cflag seems to exist)

I stripped down fiocompress to a minimal tool to just mark a file as compressed. It is important to ensure that it is a valid file (outfile in the example) generated by "fiocompress -c infile outfile" before running this command. Bad things may happen if this is not the case.

To mark output as compressed, compile the code (compile.sh should do that for you) and run "./markcompressed -m /path/to/outfile" if you are running it from the directory where it was compiled.

Source code can be downloaded from here. (A quick ugly hack, based on fiocompress from OpenSolaris, with some likely bugs)

As always, you should ensure that you have the latest recommended patchset installed.

Older OpenSSH versions - using a different authorized_keys file for a single user

Sometimes, multiple users share a home directory or there are other reasons why a user's authorized_keys file won't pass OpenSSH's permission checks. While those can be disabled, it is generally a bad idea.

In recent OpenSSH versions, using a different authorized_keys file for a single user is easy:

Match User username
AuthorizedKeysFile /etc/ssh/authorized_keys/%u

In older versions however,AuthorizedKeysFile is not supported under a Match block.

In order to get around that, at least on some version, you can use AuthorizedKeysCommand instead, which is supported under a Match block (at least on an up-to-date RHEL5): (It can also be used outside a Match block to enable the option for all users)

Match User username
AuthorizedKeysCommand /etc/ssh/etc_ssh_authorized_keys

You can then create the following script to use as the command and put keys into /etc/ssh/authorized_keys/username #!/bin/sh
/bin/cat "/etc/ssh/authorized_keys/${1}"

To prepare the script and directory: (Users can view each other's public keys in this example, but that for a small number of users is a smaller risk than disabling the permission checks, which might allow other users to edit the user's authorized_keys file)

echo -e '#!/bin/sh\n/bin/cat "/etc/ssh/authorized_keys/${1}"' > /etc/ssh/etc_ssh_authorized_keys
mkdir -m 755 /etc/ssh/authorized_keys
chmod 755 /etc/ssh/etc_ssh_authorized_keys
chmod 644 /etc/ssh/authorized_keys/* # there is probably nothing there yet
chown -R root:root /etc/ssh/etc_ssh_authorized_keys /etc/ssh/authorized_keys

Wednesday, May 4, 2016

Comparing FreeBSD pkgng package options between repositories

When switching FreeBSD repositories, it can be tricky to find which options changed in the package configuration.
This set of bash functions might be useful:
# This will list the options for the package matching the first parameter in the repo given by the second parameter
options ()
{
    sqlite3 /var/db/pkg/repo-$2.sqlite "SELECT p.name,o.option,po.value FROM option o, packages p, pkg_option po WHERE po.package_id = p.id and po.option_id = o.option_id and p.name like '$1' ORDER BY name,option"
}

# This will show the difference between the myrepo repo and the FreeBSD repo for the package fiven as a parameter. This can be changed for any repos or repos from parameters
options_diff ()
{
    diff -u <(options "$1" myrepo) <(options "$1" FreeBSD) | grep '^[+-]'
}